To devs and admins. DServer.exe instability.

[=LG/F=Kathon]

I would like to talk about DServer.exe instability. We have encountered many crashes of DServer (even several times per day) in the last few months. Maybe it's due to latest changes in the game or increased number of objects in the mission. Probably DServer is not free of bugs. It's very irritating for players and for us. 

 

So I have a few simple question to the IL-2 Sturmovik Developers Team:

 

1. Do you have plans to fix this problem in a short or long period of time?

2. Is there any way we (servers admins) can help you in this? By providing log files, errors codes etc?

 

 

 

I have also questions to admins of the other servers:

 

1. Do you encounter crashes of DServer.exe?

2. Do you think this problem has increased lately (in the last few months)? 

3. Do you use console interface to send/receive data?

4. Do you have any idea what cause DServer crash? 

 

 

Thank you for your answers.

[LLv34_Temuri]

I would like to talk about DServer.exe instability. We have encountered many crashes of DServer (even several times per day) in the last few months. Maybe it's due to latest changes in the game or increased number of objects in the mission. Probably DServer is not free of bugs. It's very irritating for players and for us. 

 

So I have a few simple question to the IL-2 Sturmovik Developers Team:

 

1. Do you have plans to fix this problem in a short or long period of time?

2. Is there any way we (servers admins) can help you in this? By providing log files, errors codes etc?

 

 

 

I have also questions to admins of the other servers:

 

1. Do you encounter crashes of DServer.exe?

2. Do you think this problem has increased lately (in the last few months)? 

3. Do you use console interface to send/receive data?

4. Do you have any idea what cause DServer crash? 

 

 

Thank you for your answers.

1. Yes.

2. We have an extensible record of the crashes happening. Sometimes it's better, sometimes worse.

3. Yes. Rcon commands to trigger mission logic.

4. We suspect it's the rcon. If it is, it's a serious issue for multiplayer, as rcon is the only way to interact with the mission that is running, because the game itself has zero scripting support. I wonder if it has something to do with syncing things with the master server, as sometimes the server can crash even three times within half an hour, and it doesn't seem to be tied to the number of sent commands.

Edited January 29, 2018 by LLv34_Temuri

[FTC_DerSheriff]

Good luck guys. Thats really a major topic and I have the feeling that the devs aren't fully aware how bad the situation is. Especially since the game gets more and more popular. Servers get fuller and they act more stressed.

Edited January 29, 2018 by DerSheriff

[LLv34_Temuri]

Just checked quickly: our server has jammed (stopped producing logs) or crashed over 60 times during this January alone.

[SAS_Storebror]

1. Twice so far in two months operation on new hardware (dedicated server). Every other day before (virtual server).

2. No. 

3. Yes.

4. Not at all, but from our experience if your hardware can't handle the load caused by dserver.exe, it's more likely to crash than just to show decreasing performance. Would be nice if dserver.exe would have a better scalability with multi-core systems.

 

Cheers!

Mike

[=IL2AU=chappyj]

IL2AU has had a few disconnects from master server perhaps but not really crashes. 

I am running 3 relatively basic missions on it with some AI Air and ground units that do move and engage/fire.

So far i've been stable for about 3 days and like i said only issues have been connectivity which is probably my local situation. 

I dont use console or commands at all.

 

I'm using an old laptop quad core @3-3.4ghz and 8gb ram and an SSD. so well above virtual server specs. 

 

what sort of machine is TAW on?

 

LLv34 and LG are running dynamic campaigns... these are a lot more complex with scripting and creating missions through scripted .msbn and mission files yes?

Edited January 29, 2018 by =IL2AUS=chappyja

[=IL2AU=ToknMurican]

I'm not an admin or a dev but with the limited number of populated servers available to us excluding Berloga,  WoL and TAW crash regularly during the hours I play. 

 

On both servers - suddenly kills stop being recorded.  On WOL with its map settings, the GPS beacon will become frozen in place despite the fact that you're still flying.

 

I don't think there's been a night of the current TAW campaign I've been flying and the server hasn't crashed. 

 

I haven't been on WoL since it crashed and reset 3 times within like an hour a month or two ago. 

 

I don't mean to sound whiney.  But I do agree with Sheriff that perhaps the devs aren't aware how much this is occurring.

 

I have no idea if the fault lies within DServer or within the scripting necessary for those servers.   I do know I'd probably be flying and enjoying the sim a lot more of if this wasn't a regular occurrence. 

[=LG/F=Kathon]

The TAW is running on i7-4790K 4GHz. The generated mission is quite simple: some objects, check_zone triggers etc. The complex script calculates and analyzes events after end the mission so it doesn't influence the DServer.

[curiousGamblerr]

It seems related to population to me.

 

TAW and WOL are pretty much the only servers that crash this often, and are also the only servers that fill up with 80+ people on a regular basis.

 

Berloga will only get to like 40ish people and I've never seen it crash (but I also spend less time there). Also, when TAW was less popular, and less often full, it seemed to crash less.

 

These are just my impressions but there seems to be a correlation with server population. I also get the impression hardware isn't the bottleneck and more hardware isn't a solution, which indicates the software is implemented with inefficiencies and more likely bugs, things like memory bugs etc. that accumulate damage over time, exacerbated by high population, and eventually start wreaking havoc.

 

Hopefully with the new in-game server option they've given the server code some love.

[SAS_Storebror]

We'd need to know the Server specs of e.g. WOL and TAW to put this into dimension.

For instance, our Server is an i7-4770 @3.4GHz, 32 GB RAM, IL-2 being on a Raid-1 with 2x2TB HDD, Windows Server 2016 Standard.

No clue whether WOL and TAW are in the same league.

 

Cheers!

Mike

[=IL2AU=ToknMurican]

It seems related to population to me.

 

TAW and WOL are pretty much the only servers that crash this often, and are also the only servers that fill up with 80+ people on a regular basis.

 

Berloga will only get to like 40ish people and I've never seen it crash (but I also spend less time there). Also, when TAW was less popular, and less often full, it seemed to crash less.

 

These are just my impressions but there seems to be a correlation with server population. I also get the impression hardware isn't the bottleneck and more hardware isn't a solution, which indicates the software is implemented with inefficiencies and more likely bugs, things like memory bugs etc. that accumulate damage over time, exacerbated by high population, and eventually start wreaking havoc.

 

Hopefully with the new in-game server option they've given the server code some love.

 

I fly odd hours for being in the U.S. My wingmates are aussies.    Crashes have occurred with less than 40 people online most certainly.  I too haven't seen Berloga crash, but also don't spend much time on it.  If anything - Berloga's mission design is kind of simplified, using only a small portion of the map compared to what we get on TAW and WoL.

 

I don't want to speculate too much with my limited knowledge on this topic and will leave this thread to the server admins and hopefully dev team.  The hours I tend to fly are some of the least populated hours of the IL-2 community.  If anything the crashes are a little more devastating at this time period as usually no one's around to get things running again for many hours. 

Edited January 29, 2018 by =TBAS=headwarp

[=LG/F=Kathon]

It seems related to population to me.

 

Not necessarily, TAW crashes very often during the European night when low number of players are on the server.

Check this: http://www.thewetbandits.org/smf/index.php?topic=285.0 (horizontal lines are periods where server was not working)  

 

We'd need to know the Server specs of e.g. WOL and TAW to put this into dimension.

For instance, our Server is an i7-4770 @3.4GHz, 32 GB RAM, IL-2 being on a Raid-1 with 2x2TB HDD, Windows Server 2016 Standard.

No clue whether WOL and TAW are in the same league.

 

Cheers!

Mike

The TAW is running on i7-4790K 4GHz. The CPU is hardly above 50%-60% with max players. The problem with DServer is that when it crash players still fly but it stops generating log files.

 

Some server uses remote console to send commands to the server. And this may be a problem.

Edited January 29, 2018 by =LG=Kathon

[1PL-Husar-1Esk]

On last TS event Q&A to Jason there was a guy (don't remember name now -sorry ) who asked about similar things you are trying now.

[SAS_Storebror]

The TAW is running on i7-4790K 4GHz. The CPU is hardly above 50%-60% with max players. The problem with DServer is that when it crash players still fly but it stops generating log files.

Okay, so that should not be the cause of the issue.

Maybe it'd make sense to have some debugging switch for dserver.exe to help Devs with more detailed logs.

Our Servers definitely could take the additional log load.

 

Cheers!

Mike

[LLv34_Temuri]

It seems related to population to me.

No. We've had  an empty dserver crash about two minutes into a mission.

[LLv34_Temuri]

Okay, so that should not be the cause of the issue.

Maybe it'd make sense to have some debugging switch for dserver.exe to help Devs with more detailed logs.

Our Servers definitely could take the additional log load.

Yes. As we all know, the dev team has its hands full, so if there was a way to produce test logs or somesuch, we'd be happy to do it.

[Fidelity]

Kathon, have you investigated Event Viewer around the times of the crashes? I wonder if you might be able to find some correlations there.

 

It seems related to population to me.

I've seen WOL and TAW crash with less than 20 people connected. I'm pretty sure it has to do with complexity.

 

I'm really hoping that Dserver can be fixed and expanded to allow greater logic and make better use of system resources. I have some great ideas I'd like to build into a COOP server, but dserver is in no condition to realize those ideas.

[curiousGamblerr]

A few folks have shot down the idea that it's related to population with useful counter examples. However, note that a memory leak that is exacerbated by high traffic might not rear its ugly head until hours after the fact, when the server might be much less full.

 

I just find the fact that this problem only affects high traffic servers to be significant. Of course, it's very possible it does affect those low traffic servers and we just don't notice because we don't fly on them. Who knows.

 

I'm not a server admin so I guess I should stop speculating now

[LLv34_Temuri]

A few folks have shot down the idea that it's related to population with useful counter examples. However, note that a memory leak that is exacerbated by high traffic might not rear its ugly head until hours after the fact, when the server might be much less full.

What I meant with "two minutes into a mission" was that I'd just started the dserver and it crashed

[Fidelity]

I'm not a server admin so I guess I should stop speculating now

Nothing wrong with hazarding a guess! You're just trying to help, after all. 

[ACG_HotLunch]

Devs have stated Single Player is where their focus lays. I wouldn't get my hopes up about a fix any time soon.

[coconut]

I have also questions to admins of the other servers:

1. Do you encounter crashes of DServer.exe?

2. Do you think this problem has increased lately (in the last few months)?

3. Do you use console interface to send/receive data?

4. Do you have any idea what cause DServer crash?

1. Seldom. I had one crash last weekend, I don't remember last time I had one before that.

2. No

3. Yes

4. Not really, but rcon could be a cause. I've had crashes when running my "ground commander" system.

 

Notes:

I kill and restart DServer.exe every 4th mission. I don't let it run much longer than 10h.

My dynamic campaigns do not rely on rcon interaction for the mission logic, beside triggering a mission end. Most rcon interaction is to retrieve the list of players and send messages to teams when someone takes off or lands.

I keep the player limit low (max 32 players)

Edited January 29, 2018 by coconut

[ACG_pezman]

[...]

 

I keep the player limit low (max 32 players)

 

And this Coconut, is the biggest problem with your magnificent server.  You should look into changing this

[=TBAS=Sshadow14]

Does dserver.exe even use all those specs.??

Running it on a 2ghz intel Core 2 Duo (single core) with 3gb of ram and running off a Sata SSHD on Windows 7 32bit.
should give the same results as
Running it on a i7 7700L @ 4.5ghz with 16gb of ram running of a raid array on 64bit system using latest Windows 10

Aren't the dserver.exe specs stated to only support upto.(but i cant find the link to devs post)
2ghz (1 core only)
2gb ram 32bit os.
??

 

Edited January 29, 2018 by =TBAS=OccludedLight14

[Fidelity]

Yes, you are correct. This is why we wish the application would be expanded for scalability. Think how much more would be possible if we could run this on a 4+ core and 16GB+ server.

Edited January 29, 2018 by Fidelity

[curiousGamblerr]

Yes, you are correct. This is why we wish the application would be expanded for scalability. Think how much more would be possible if we could run this on a 4+ core and 16GB+ server.

 

Soon DServer will be web scale!

Edited January 29, 2018 by 19//curiousGamblerr

[1PL-Husar-1Esk]

LOL about /dev/null

[=IL2AU=chappyj]

It seems related to population to me.

 

TAW and WOL are pretty much the only servers that crash this often, and are also the only servers that fill up with 80+ people on a regular basis.

 

Berloga will only get to like 40ish people and I've never seen it crash (but I also spend less time there). Also, when TAW was less popular, and less often full, it seemed to crash less.

 

These are just my impressions but there seems to be a correlation with server population. I also get the impression hardware isn't the bottleneck and more hardware isn't a solution, which indicates the software is implemented with inefficiencies and more likely bugs, things like memory bugs etc. that accumulate damage over time, exacerbated by high population, and eventually start wreaking havoc.

 

Hopefully with the new in-game server option they've given the server code some love.

Not sure it's population either

several taw crashes have occurred when it's in low population of 30 or less

 

It's also not just after long up times (in taws case) it can happen in 2 missions in a row for example

 

Coconuts method of deliberate dserver restarts though could be worth looking at after every say 2nd mission just for ticking this box

 

 

Is it worth exploring the mission itself as a cause of crashes

How many triggers and complex triggers are in use . How many msec between trigger fires are there any points in a mission where too many triggers have the potential to fire etc

 

 

Some logging ability in dserver w0uld be very helpful too

Edited January 29, 2018 by =IL2AUS=chappyja

[LLv34_Temuri]

Is it worth exploring the mission itself as a cause of crashes

This is actually something I've thought about. Is it possible to create such a mission logic that will cause a crash, or does the dserver have some kind of handling for stupid mission design?

 

We've brought the mission size down from about 10 MB to about 800 KB. The noticeable changes have been that the loading time was reduced and perhaps the lagging eased somewhat.

[=EXPEND=CG_Justin]

Soon DServer will be web scale!

 

I'm in tears watching this!!! :lol: :rofl:

[[DBS]Tx_Tip]

 

 

I have also questions to admins of the other servers:

 

1. Do you encounter crashes of DServer.exe?

2. Do you think this problem has increased lately (in the last few months)? 

3. Do you use console interface to send/receive data?

4. Do you have any idea what cause DServer crash? 

 

 

Thank you for your answers.

 

1. Not often.   

2. No. DServer.exe if anything has become more stable since the latest updates. I believe this has something to do with the active linked entities, (planes, trains, artillery, vehicles, tanks) themselves and their CPU demand.

3. No

4. Have never experienced a DServer crash when running FNBF which crams as much moving and active entities along with buildings, bridges etc... as the DServer software and our hardware allows for 44 players. 

 

*However I do test and let the mission run for at least 30 minutes before hand with as much as possible active to check .sps and ticks. While that may seem anecdotal after coming up on 2 1/2 years of running these I can attest that there is a noticeable difference for the better within the DServer .sps and ticks performance after test running a full out mission the first time then shutting the DServer down and restarting the same mission.

 

Not that every server has that luxury. But perhaps this is one of the causes of the DServer crashes when a new mission is created from rcon/scripting and must download then upload to a number of players that is eventually creating those crashes.  

[Herne]

Dserver imo needs some urgent attention, instability is more than an inconvenience when you only have a short window to play in the evenings, decide to do a long bomb run which takes most your available time, and the server crashes before you release your bombs.

 

its really disheartening.

[Ribbon]

This is actually something I've thought about. Is it possible to create such a mission logic that will cause a crash, or does the dserver have some kind of handling for stupid mission design?

 

We've brought the mission size down from about 10 MB to about 800 KB. The noticeable changes have been that the loading time was reduced and perhaps the lagging eased somewhat.

As you said population has nothing to do with it, it happened many times with only few ppl online.

I noticed that crashes often occur when trigger is activated/ or should be activated but it does not activate.

For example;

-enemy tankbase is destroyed and friendly tanks should attack but they dont(server block/crash, no trigger activation idicated)

-friendly tanks or airfield should be captured but trigger blocks and nothing happen, shortly after server crash fully.

 

Ignore me if im sayin obvious things that you already know, i'm just trying to help.

 

Btw if i had to choose between flying dynamic war on Finnish with frequent crashes and some simple missions without crashes i would still choose Finnish so don't give up guys.