DServer unstable/crashing frequently in 3.005

[DeadMeat0383]

5 minutes ago, Thad said:

Most frustrating for server providers. This should be considered a serious problem facing IL2 multiplay. ?

 

Perhaps this is why the developers don't run an official server themselves. 

It is indeed frustrating, especially since we cant definitively narrow down the cause of the issue despite our best attempts, hopefully the il2 devs can narrow it down.

 

I doubt that this is why they don't run an official server, its more likely that its just too expensive to run and maintain a multiplayer server for the team.
It's not cheap to run a server unless you are lucky to be in a situation where hardware and internet connection are available that are up to the task.

 

Now that the devs are returning from their holidays I am hopeful that we will get some progress on this one.

[sniperton]

I didn't want to trivialize the problem. Maybe I put it in the wrong way. I only wanted to say a heartfelt thank to server-providers and assure them that online play is overwhelmingly fun despite the eventual crashes. I too hope that the bug can be tracked down and fixed soon.

[=AD=uumembwa]

Sometime i think about master-server... Couse server crushes at evening, and work at night and morning. And crushes not depend from players, sometime server crushes without players.

Before 3.005 Action server works fine. But in some versions was that unreasonable crushes.

[DeadMeat0383]

So thats 2-3 crashes per day for the last 3 days in a rown now, not a good run.

Latest crash was at 3:18pm GMT+10

[sniperton]

On 8/20/2018 at 12:15 PM, Mr_Pickles said:

Sometime i think about master-server... Couse server crushes at evening, and work at night and morning. And crushes not depend from players, sometime server crushes without players.

Perhaps you and DeadMeat should put your crashes into a similar chart to test whether it's a viable hypothesis.

[=AD=uumembwa]

Usually it happens between 14 and 23 moscow time. 11 - 20 utc

Edited August 23, 2018 by Mr_Pickles

[II./JG1_Kadin]

Just so you know guys, I still do not have any crashes whatsoever!  I am not sure where to attribute that.  Maybe there is much less traffic on mine, maybe the missions are setup a certain way or even the customized server parameters have something to do with it.  Heck, even my server location and type of connection could be at play...

[=AD=uumembwa]

2 hours ago, II./JG1_Kadin said:

Just so you know guys, I still do not have any crashes whatsoever!  I am not sure where to attribute that.  Maybe there is much less traffic on mine, maybe the missions are setup a certain way or even the customized server parameters have something to do with it.  Heck, even my server location and type of connection could be at play...

Well, i join you server and find two difference with mine:

1. Playable tanks

2. Allow mouse joy

 

and i use mission resaver

[sniperton]

10 hours ago, Mr_Pickles said:

Well, i join you server and find two difference with mine:

1. Playable tanks

2. Allow mouse joy

The FAC server has them both from the beginning, and as you can see from the chart, the FAC server has improved a lot in the past month regarding crashes, so I would seek the problem elsewhere. Any of the factors listed by Kadin can have a role, but in the absence of meaningful logs only luck and chance can deliver a solution.

[II./JG1_Kadin]

Mr_Pickles, you mean "NOT allow mouse joy", right?  Sometimes "not"s escape during typing.  It happens to me frequently

 

I could post one my Missions if anyone wants to try that variable?

[=AD=uumembwa]

Just note differences, on my server allowed mouse joy and tanks. But i think problem not in server settings. Before 3.005 server was stable.

[II./JG1_Kadin]

Apologies, I mistakenly thought your were referring to my setting, because Drivable tanks are there in my missions too.  But the later models are in only on 6 of the 18 mission rotation.  Probably you logged on a mission that those tanks are not in.

Edited August 25, 2018 by II./JG1_Kadin

[stupor-mundi]

I was just on the Unprofessionals server. I finished a mission, picked a different tankfield, tried to spawn, and got "Already spawned error".

So I left the server and tried to get back in, but couldn't. Retried a few times and gathered that the server had probably crashed? That was about 14:05 UK time.

I mention this because in the recent, server-crashy times, I had seen that error once before. But in the time before the server crashes, had never got that.

 

[DeadMeat0383]

2 minutes ago, stupor-mundi said:

I was just on the Unprofessionals server. I finished a mission, picked a different tankfield, tried to spawn, and got "Already spawned error".

So I left the server and tried to get back in, but couldn't. Retried a few times and gathered that the server had probably crashed? That was about 14:05 UK time.

I mention this because in the recent, server-crashy times, I had seen that error once before. But in the time before the server crashes, had never got that.

 

Something went strange with the server, I have rebooted the server (operating system) and it should be back up momentarily.

@Jason_Williams can I please get at least an acknowledgement of this? It is getting very frustrating trying to run a multiplayer server and having it crash randomly, and now its doing strange things like now allowing people to connect (just throws them back to the menu with no error) and also spawning issues.

I don't expect a eta on the fix or even that its on the list of things to fix all im asking for is an acknowledgement that you are aware of it, is that too much to ask for?

[DeadMeat0383]

What do I need to do to get acknowledgement on these issues?

I have spent large amounts of money on this game (all expansions in deluxe except BoS which I later bought the two collector planes for, all collector planes), and my support tickets are ignored, this thread appears to have no acknowledgment from staff at all, and its getting very frustrating trying to run a server for this game.

 

I tried to hold a flyout on Friday night on my server, I lost count of the amount of times I had to restart the server because it either became unresponsive or crashed. As a result half of the people who joined to try fly out with us were unable to fly and gave up. I know the guys are a small team and its a busy and disrupted time for them but I don't think its unreasonable to expect an acknowledgement of the issue in this thread, do you?

 

Please find the below screenshot of the most recent crash (this morning at 8am by the look of it) and im happy to supply whatever information is required to investigate this issue.

[SAS_Storebror]

Can't help but to sign what @DeadMeat0383 said.

This is getting frustrating.

Server Operators spend real life money, time and efforts for the best of this game, yet we're sort of ignored and left alone with the issues caused by recent updates.

Not exactly the type of customer support you'd wish for, sorry to say.

 

This game is impressive, no need to argue about that fact, but take care devs: Lack of communication killed many other good games before.

 

Mike

[SAS_Storebror]

FWIW, I just noticed that your mission @DeadMeat0383 seems to run at roughly the same tick delay like our training mission (Tick Delay ~10ms), your DServer window shows way less grey spikes than ours. Wondering where this comes from... maybe the massive bunch of AI crowd we're running?

Did anyone ever see an explanation of what the grey lines behind the yellow "Tick Delay" line on the DServer window are actually meant to represent?

 

Mike

[DeadMeat0383]

Yep we found that our patrols use around 1-1.5 tickrate for fighters and 2ish for bombers/attackers/multicrew aircraft.

[SAS_Storebror]

It's a pity.

On the same server we run an IL-2 1946 dedicated server with Ultrapack 3 + Patch Pack (probably the heaviest server version you can have in 1946).

We have missions on it where way more than a hundred AI bombers are prowling the skies - the server load on a single core is less than 20% in that case.

In BoS it would be more than 1000% (no typo), with nothing else on the map/mission.

 

Mike

[=AD=uumembwa]

@Jason_Williams

Players donate me money for server upgrade, i paid every month for priveleged 100mbit up/down, and on the end that we get?

40 days after release 3.005 and all that days dserver crush and crush and crush.

I don't want donate anymore for this game. 

I think is better now close server and forget about that nightmare.

Edited August 28, 2018 by Mr_Pickles

[SAS_Storebror]

That'd be the worst case outcome of this mess.

Stay with us for a few more time please @Mr_Pickles.

 

Mike

[sniperton]

@Mr_Pickles: As Mike said, stay with us, please.

 

@SAS_Storebror: You know I'm an ignorant outsider, but do I understand correctly: is one AI in BoX more resource-heavy than 20 in the old game? If so, the AI requires more CPU power to perform as bad as I do when I'm half asleep, half drunk. Very bad news for the future, as the AI seems to struggle with the FM instead of making reasonable combat decisions.

[SAS_Storebror]

Actually AI planes are about 100 times as resource heavy in BoS as they were in IL-2 1946 (UP3 that is, which already is twice as resource heavy as stock 1946 at least).

 

I've just checked: When I have 120 bombers flying at the same time on a mission in UP3 (B-17 and B-24 plus a handful Mosquitos) plus a couple of AI fighters, the CPU usage on the single core UP3 is using is 20%.

That means I could theoretically have 600 bombers on that mission - theoretically. Practically that would cause other issues.

 

In contrast, with 3 AI bombers on each side flying at the same time in BoS on Stalingrad map (6 AI bombers in total, 3 He-111 H-6 and 3 Pe-2 Ser.87) plus a few AI ground units, the Tick Delay reaches 16ms with spikes up to 18ms. One more AI bomber and the Server will issue overload warnings.

 

That's 6 AI bombers on BoS vs. 600 on 1946.

And in 1946 I can spawn them in formation, which still isn't possible in BoS, just saying...

 

Mike

[sniperton]

[I don't want to hijack this thread, but what you say seems to have far-reaching consequences for the whole AI issue discussed in other threads. It's either a very serious code optimization issue (much more cycles are processed than actually needed), or the current concept of letting the AI fly the same complex FM is fundamentally wrong and unsustainable in the long run. In either case a complete code overhaul is needed if we want to see any significant improvement to the AI (better individual AI and more AI actors at the same time). And an AI code overhaul is still easier than native multi-threading support, I guess.]

[DeadMeat0383]

Guys please try not to derail this thread with off topic chatter, please keep on topic.

[SAS_Storebror]

Roger that 

 

Mike

[=AD=uumembwa]

3 hours ago, DeadMeat0383 said:

Guys please try not to derail this thread with off topic chatter, please keep on topic.

AI is about dserver couse maybe for me is not necessary 600 AI bombers, even 6 AI bombers, but AAA and PT-guns - is AI too.

And now AAA and cannons want more resources of dserver than players. And it ruined all possibilites of mission builder, i can use in mission only 20 AAA and cannons to keep tick delay <5ms.

Ju-52 can be used in missions but then AI Ju-52 drop paratroopers - it crush server(was in 3.004 now i remove that plane from server)

[sniperton]

Earlier it was stated by Mike that crashes cannot be related to user activity.

Earlier it was stated by DeadMeat that crashes cannot be related to a system overload.

Can it also be stated that AI activity can be ruled out?

If so, what remains is

(1) the master server, or

(2) connection to the master server, or

(3) a mere coincidence of various factors that cannot be tracked down easily due to the incomplete logs, or,

what is the same in effect, (4) a single piece of defective DServer code that only activates under very rare (I mean, statistically rare) circumstances.

Edited August 29, 2018 by sniperton

[II./JG1_Kadin]

I have still not had a crash and the server is running 24/7...just updating.

[=AD=uumembwa]

Maybe disable mouse joy? I see no another differences...?

[II./JG1_Kadin]

It could be in the mission design, server config file, location, connection type and speed, amount of traffic, what other variables could we have different?  Obviously, there is something wrong in the code.  We are just trying to narrow it down.

[=AD=uumembwa]

After crushes start i remove all new missions, and use only Lapino, mission design is simple and have stable 0 tick delay at empty server.

Connection 100mbit.

25 minutes ago, II./JG1_Kadin said:

server config file

This is i talking about in my config turned on mouse joy.

[II./JG1_Kadin]

Yes but there are other settings that are hidden from clients

[=AD=uumembwa]

17 minutes ago, II./JG1_Kadin said:

Yes but there are other settings that are hidden from clients

Developers hide from clients settings for stable server usability? Nice.

[II./JG1_Kadin]

These settings are available to anyone that operates a dedicated server.  E.g, difficulty settings, Connection speed, ports, Max ping, Tackview setting, Mission timers, other limiting parameters like locking skins, weapons, penalties and so on.  Some of these settings MAY be involved in the current issues. It is not a certainty that they are causing a problem.

Edited August 29, 2018 by II./JG1_Kadin

[=AD=uumembwa]

I know, this game like timers)) So i set 300sec to death penalty and 600sec to change coalition, 1200 to delete away players, 10sec for end flight and 10 to start. 60 to mission end.

I give for dserver all pauses between events that it need. In mission also all events with AI AAA and cannons have timers, to minimalize events in one moment in time.

I don't no what else need that damn dserver?

[SAS_Storebror]

Just for the record:

With the same settings as always, on the same server, with 1GBit up/down unclocked connection, and with mousejoy enabled, and with missions that have massively gained in complexity (we've been at about 1.5MB mission file size in the beginning of the crash series, now we're at about 3MB), the very same thing crashed within a second before (two times at least, the server didn't run for long enough to even count it) - and now it's running without a single crash for almost two weeks.

 

Tell me what you want, this is nothing on our end.

It's not in the mission, it's not in the settings, it's not in the server hardware or anything of that kind.

It might indeed be something in code that gets triggered only once in a full moon, or it's some network thing on the master server end.

Concerning the latter, it might even make sense: In the beginning of 3.005c, many users had to update their game. Heavy load on the "Master" Server(s), many crashes.

Later it became better, but there were still "peak times" for crashes - maybe at the same time when lots of "latecomer" users updated their game.

Now the update wave has passed, load on "Master" Server(s) decreased and crash times are over.

Just wild guessing.

Customer Service unfortunately leaves us alone with this.

 

Mike

[=AD=uumembwa]

7 hours ago, SAS_Storebror said:

It's not in the mission, it's not in the settings, it's not in the server hardware or anything of that kind.

I agree.

18 hours uptime, crushes stop without any reason.

 

[sniperton]

12 hours ago, SAS_Storebror said:

Now the update wave has passed, load on "Master" Server(s) decreased and crash times are over.

Just wild guessing.

 

5 hours ago, Mr_Pickles said:

18 hours uptime, crushes stop without any reason.

Time will show who guessed right. It has been either fixed, or we'll have to face with a new wave of crashes when the next update is out

[DeadMeat0383]

Times and dates of the recent crashes