Jump to content
Alonzo

Best ways to keep dserver actually working properly

Recommended Posts

Combat Box has been running for a couple of months now, and we regularly fill up. People like the maps. We've had a few issues where having too much AI (where "too much" is more than ~10 ships active at once, or more than ~30 armored vehicles) will cause a server hang, but we have ironed most of those out. But I'm still left with dserver sometimes deciding it doesn't want to accept player connections, or boots everyone off.

 

Those folks who are running successful servers, do you have any tips? I am running dserver with a time limit per map, and a fixed rotation of maps (about 6 maps, but a list of them repeated to about 20, so it 'seems' random to the players). Anything I can do to improve server stability? I am thinking maybe one of:

  • Daily operating system reboot to clear everything out
  • Apply scripting on top of dsever: only run one map in my dserver SDS file, wait for server to complete (and forcibly kill if it goes longer than my map timer + 2 minutes), force kill any remaining dserver processes, pick a new map and put it into the SDS file, start dserver again.

Or possibly both. Would these help? What do other server operators do to improve reliability?

Share this post


Link to post
Share on other sites

We have an application that sniffs if the dserver stops producing logs (i.e. hangs), and in such cases kills and restarts the dserver and the campaign system.

  • Thanks 1

Share this post


Link to post
Share on other sites
7 hours ago, LLv34_Temuri said:

We have an application that sniffs if the dserver stops producing logs (i.e. hangs), and in such cases kills and restarts the dserver and the campaign system.

 

This is very interesting. Which logs do you use? The little text logs that get produced about once every minute? Or the binary logs, or something else? Thanks!

Share this post


Link to post
Share on other sites

The text logs. IIRC, a log file is generated every 30 seconds (or something like that). When the campaign app sees that a new log file hasn’t been generated in a given amount of time, it does the restart.

Share this post


Link to post
Share on other sites

Alonzo, what is the hardware of your server, because, 30 armored vehicles is very low. I can't say for the ship, because i don't think having a mission with 10 ships.

Share this post


Link to post
Share on other sites
Posted (edited)
7 hours ago, Habu said:

Alonzo, what is the hardware of your server, because, 30 armored vehicles is very low. I can't say for the ship, because i don't think having a mission with 10 ships.

 

It's an 8600K @ 4.8ghz with AVX-2 and 3200mhz RAM. I don't think it's a CPU limitation because the tick times are fine (below 10ms usually). There is something strange when you add too much AI. The server process somehow 'hangs' and doesn't allow players to connect. We have seen this on a new map -- players can play all the way through the map, no problem, and then the server loads the next map. Now no-one can connect until I kill the server ('x' to close the window, then find it in Task Manager and kill it). Seems like something to do with the AI on the maps.

Edited by Alonzo_LW

Share this post


Link to post
Share on other sites

Here is the hardware of my server :

i7 4790k @ 4ghz

32 GO ram

SSD for the system

SSD for the game server

Windows 2012 R2

 

Bandwidth 500 Mbps

 

The server is in a data center.

 

I host some mission with around 100 ground units in 10/20 km square without any problem. We played one of my mission with 40 or 50 human players.

 

What is the bandwidth of the server ?

It could be the i5 which has fewer threads than the I7.

 

I join the last version of the mission. From the original version, i add  AI in the air, and give the possibility to the player to activate some Ai flight. We played it with the Ai in the air with 10 friends without any problem.

 

 

PX_Multi_Anapa_Battle_v3.1.zip

Share this post


Link to post
Share on other sites

The problem doesn't seem to be server performance -- it would give all those "overload" warnings to players if it were server load. There's some kind of reliability problem that gets tickled in different ways by different maps. I'm going to do more forced scripting of the server starting this weekend to see if I can narrow down the problem.

Share this post


Link to post
Share on other sites

No, the message server overload is only linked to the tick delay. When the tick delay is over 18 (if i remember), you have the message which is displayed. But you can have the message and your CPU can be à 50%. I never see the CPU of my server at 100%.

 

If you want, put your mission here and i can host it on my server. and we'll see if we have the same problem. (i can't do it now because i need all the resource for beta testing 😉 )

 

On 6/6/2019 at 8:24 PM, [DBS]Tx_Tip said:

What cha' got there Habu?

Downloading. Whoop! :)

Thx.

The goal is to play it even if we are are a small group of player, and the player can choose to activate or not some AI flight. They can use the administrator airfield, or choose planes with a non generic name on some airfield. For example, IA-HS129 will spawn a group of HS 129, and you as a player you can choose to escort them, or follow them in a HS 129. There are many random for fighters. I add too the possibility to relaunch the mission, even if i'm not present. Airfield are close for 3 or 4 minutes at the begining (i don't remember) to avoid player which spawn and fly when there are player which  are connecting to the game.

Share this post


Link to post
Share on other sites

I've observed this kind of problems before. In addition to players not being able to join, the "end mission" button would never turn to green, and AIs and mission logic would stop to work. Other than that, player-controlled planes would continue to fly normally. I'm not sure but I think that ground targets would not take damage.

But that was fixed a long time ago.

The "end mission" button stuck in gray came back, but was fixed again.

My way to deal with this kind of issue is to restart DServer every 3 missions (which for my missions means 9 hours). There's an option in SDS files to kill the server every N missions (I think it's called shutdownloads, IIRC). Then all you need is to start DServer from a bat file in a loop to bring it back every times it goes down.

I have personally never observed the bug where DServer does not produce log entries, which is weird considering my campaign can generate rather heavy missions with tons of AIs and AA.

 

Maybe you could post the mission that's giving you trouble? Maybe there's a specific AI you use that Habu did not that is the culprit?

Share this post


Link to post
Share on other sites
On 6/8/2019 at 12:05 PM, coconut said:

My way to deal with this kind of issue is to restart DServer every 3 missions (which for my missions means 9 hours). There's an option in SDS files to kill the server every N missions (I think it's called shutdownloads, IIRC)

I didn't notice that option. You learn me something.

 

As you said, even if i use a bunch of AI vehicles, it could be one type i don't use.

Share this post


Link to post
Share on other sites

Hey all, sorry I didn’t reply before now. I ended up using the “quit after 1 map” option in dserver along with a custom map picking and restart script. This has fixed the problems for now. The down side is players are kicked and need to rejoin the server, but I think it’s worth the trade off for extra reliability.

Share this post


Link to post
Share on other sites
On 6/14/2019 at 9:48 AM, Habu said:

I'm interested in your batch to re-run the dserver.

 

Here is the script. WARNING WARNING WARNING -- it includes an initial "kill any running dserver" function. You might not want that.

 

Back up your files before using this script. No warranty. Use at your own risk, etc.

 

This is a very dumb script. It does some brute-force things, and some Combat Box-specific things:

  1. Kills any running dserver process, then enters a loop:
    1. Loads SDS config file, looking for round time and current map.
    2. Replaces map in SDS with a randomly chosen one (not same as previous).
    3. Checks for round time and remembers it.
    4. Writes a "current map" text file so that il2 stats Django web page can show current map and time remaining.
    5. Copies map JSON over static file in il2 stats so that il2 mission planner integration works.
    6. Executes dserver for the map, waiting for 
      1. dserver not running any more -- crashed or exited cleanly via ShutdownLoads = 1
      2. Maximum round time + 2 minutes is exceeded -- server probably got "stuck"
      3. A new "zero" [0] log appears -- map has for some reason rolled
    7. Map is done, so kill any remaining dserver processes

As you can see this is brute force because something about one or more of the Combat Box maps leaves the server "hung". You might not need all this logic on your server. 

 

While it's dumb, it's also quite successful. Combat Box has been running for several days now with only one undetected server failure, and I've adjusted the script to pick that up too.

 

dserver-controller-0.2.pl.txt

  • Upvote 1

Share this post


Link to post
Share on other sites

Thank you Alonzo.

 

I don't need the stat part, only the check of the server. Maybe i'll try to run it. 

 

What do you call maps is the mission, if i understand ?

Share this post


Link to post
Share on other sites
43 minutes ago, Habu said:

What do you call maps is the mission, if i understand ?

 

Yes, sorry I use "map" and "mission" interchangeably, but I guess "map" should mean "Moscow" or whatever. To me it usually means mission.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...