DServer unstable/crashing frequently in 3.005

[SAS_Storebror]

That much to my theory.

If it was about updating, crashes would have stopped for @DeadMeat0383 too.

 

Mike

[DeadMeat0383]

My thought is that there is a programming bug in dserver that causes the issue and I'm sure there is a trigger for it... but [edited] if I know what the trigger/cause is

Edited September 5, 2018 by SYN_Haashashin
lenguage

[sniperton]

The FAC mission files grew in size, but possibly something was also removed from them in the process that used to trigger the crash.

Edited August 31, 2018 by sniperton

[SAS_Storebror]

While we can never rule that out, fact is that our current training mission has one significant feature that would mark it as "crash prone": Whatever kind of object there is on the mission editor, you'll find one instance of it on the mission at least.

 

Mike

[DeadMeat0383]

And here is yet another update on the frequency of crashes....

[=AD=uumembwa]

my crashes are stable: after 16.00 local time server begin crush every ~2 hours?

[SAS_Storebror]

Now we only need to know the timezone you're in, in order to judge this statement.

 

Mike

[sniperton]

On 8/23/2018 at 6:50 PM, Mr_Pickles said:

Usually it happens between 14 and 23 moscow time. 11 - 20 utc

16 Moscow time should be 13 UTC.

How do crash intervals relate to mission rotation? Is there any pattern?

[=AD=uumembwa]

it happens in any mission

[DeadMeat0383]

Updated list of crashes, seems to have stabilized for now, will be interesting to see how long before the next crash.

[SAS_Storebror]

It's odd to see how server crashes come and go for apparently no reason.

At the risk to court disaster, let me state that our last DServer crash was on August 17.

The only major thing that changed since then is this: We have enabled mods.

Yes, you read correctly.

Maybe that's because we're now the only one's on "our" server list?

God knows. Or hopefully devs too. I'm not going to state that they could let us know. Too obvious 

 

Mike

 

 

[=AD=uumembwa]

On 9/6/2018 at 8:41 PM, SAS_Storebror said:

It's odd to see how server crashes come and go for apparently no reason.

At the risk to court disaster, let me state that our last DServer crash was on August 17.

The only major thing that changed since then is this: We have enabled mods.

Yes, you read correctly.

Maybe that's because we're now the only one's on "our" server list?

God knows. Or hopefully devs too. I'm not going to state that they could let us know. Too obvious 

 

Mike

In wartunder dev testing 1.81, players go to there on master-server reduce load?

[stupor-mundi]

== Falling between the polygons ==

Ok here's another crazy hypothesis:

When tanking, one encounters two types of annoying terrain discontinuities. The first, frequent one, that you bonk into, and the second, much rarer one, that you fall into.

This is not something new. Many times, when driving along the shore at Kerch, did I fall between polygons and drowned. Before 3.005 this only resulted mission end (for the player) by drowning.

This happened to me twice Today and a few days ago at Gelendzhik: Right at the moment I fell in, the server crashed. Right at the moment of another crash, another player observed a tank driving into the ground (at an AF, not the beach).

In this context I remembered how some days earlier, on a different server, a different map, I had shot a tiger on a bridge, which then collapsed, and the server crashed in that moment.

I should add that the frequency of crashes today on the Gelendzhik map was much higher than recently 'normal', and that the landscape there is much more crumpled than on most maps.

 

[=AD=uumembwa]

Yeah this map crashes every ~30min.

And yes i drive on this map, and found terrain errors.

[sniperton]

2 hours ago, Mr_Pickles said:

Yeah this map crashes every ~30min.

And yes i drive on this map, and found terrain errors.

@Mr_Pickles, I don't want to be rude, but document, document, document, report, report, report, if you think you've found the culprit.

[stupor-mundi]

Victory! I just drove along the sea again, Gelendzhik, fell in, and crashed the server. It's thus reproducible.

I'm not saying this is THE cause of crashes since 3.005, but it certainly does crash the server.

 

 

 

 

 

[SAS_Storebror]

That actually makes sense.

One of the changes we've made to our training mission set between the times when the server used to crash and nowadays where it doesn't do so anymore, was to redo all ground vehicle movement and, most notably, move the whole tank battle area some 40km up north on the map.

Maybe we thereby got the tanks out of a "critical" area where they crashed the server before, and have them in a "safe" place now where there's no hole in the map.

 

Mike

[sniperton]

21 hours ago, stupor-mundi said:

Victory! I just drove along the sea again, Gelendzhik, fell in, and crashed the server. It's thus reproducible.

I'm not saying this is THE cause of crashes since 3.005, but it certainly does crash the server.

Could you reproduce it in SP as well?

[stupor-mundi]

On 9/11/2018 at 4:44 PM, sniperton said:

Could you reproduce it in SP as well?

 

After unsuccessfully trying to guess for a while, I have to admit ignorance about what SP means. A server?

 

There is no particular knack to reproducing this I think. Although initially, when I crashed the server, each time I was driving along the beach, and drowned when there was a terrain error, when I tried it later I just drove along the beach, diagonally into the water. I'm pretty sure it would work just as well driving straight into the water. And I would guess a river would be just as good as the sea.

[sniperton]

SP = single-player. If you can reproduce it in single-player, you don't have to make your server-operator unhappy about the extra crashes your testing leads to.

If online crashes have to do with maps (due to e.g. corrupted elevation map data), then those 'holes' should be there on offline maps as well, right? But if you can't always reproduce it on the very same location, it may be related to the vehicle rather than the place. Just some wild guessing.

[stupor-mundi]

10 minutes ago, sniperton said:

SP = single-player. If you can reproduce it in single-player, you don't have to make your server-operator unhappy about the extra crashes your testing leads to.

If online crashes have to do with maps (due to e.g. corrupted elevation map data), then those 'holes' should be there on offline maps as well, right? But if you can't always reproduce it on the very same location, it may be related to the vehicle rather than the place. Just some wild guessing.

 

While the 'holes' are in the terrain, I don't think the holes are necessarily crucial here, because I found I could just drive into the water and crash the server.

The correct, or pre-3.005 normal outcome is of course that the tank sinks and the mission ends. This happens in single player, offline, I've done it many times.

But since in single player there is no server that can be crashed, what knowledge could be gained?

 

[sniperton]

9 hours ago, stupor-mundi said:

But since in single player there is no server that can be crashed, what knowledge could be gained?

The exact location of the 'holes' or 'hotspots' without crashing a server. Holes on the map are bugs, after all.

From what you describe it seems that server crashes (all crashes or some crashes, we do not know yet) have something to do with how vehicles interact with the terrain. Maybe a sinking vehicle suffices, and you can sink a vehicle either by falling through a hole into the void or by driving it into the see. If you can report exact coordinates where this frequently happens, you give the devs a chance not to neglect it.

[stupor-mundi]

The bay of Gelendzhik is a little bit SE of Novorossiysk.

For comparison, I just drove a tank straight into a river on the Taman peninsula, which didn't crash the server, and then drove another tank diagonally along the beach at Golubitskaya, which did crash the server.

I don't know if the difference has to do with the angle, straight vs diagonal, or with river vs sea (altitude).

All the beaches / riverfronts I've encountered in-game have this property:

You drive along them in a manner where IRL you would just get wet feet/ wet tracks, i.e. you're basically just touching the water, but on quasi solid ground. Then you fall between two polygons and drown.

I think the ball is now in the court of the server admins to try this out. I can only try this at the risk of attracting their wrath, they can try it as much as they like.

 

[sniperton]

3 hours ago, stupor-mundi said:

You drive along them in a manner where IRL you would just get wet feet/ wet tracks, i.e. you're basically just touching the water, but on quasi solid ground. Then you fall between two polygons and drown.

That's probably because water type terrain has no solid bed, land ends where water begins, and you simply fall into a hole with no bottom, and when you reach a negative altitude, the game crashes. Maybe rivers have a higher elevation than sea level with zero altitude, and you drawn before you could crash the server. Anyway, many thanks for your efforts, unfortunately I'm away from my rig for two weeks, so i can't test it right now. Just one more question: have you always driven the same type of tank (if yes, which one), or did all this happen with various tank types?

 

Edit: Has anyone ever tried what happens if you or the AI ditches an airplane on the sea in multiplayer? I know parachutes die, but does the plane with a living driver inside also crash the server as it sinks?

Edited September 13, 2018 by sniperton

[stupor-mundi]

I've done this in T34s and Tigers.

[SAS_Storebror]

Alright, bug confirmed.

Attached you can find the test results of driving into shallow waters at the coast of Gelendzhik bay.

Including mission file, all logs, tracks and DServer crashdump.

 

Steps to reproduce:

1. Load the mission in DServer.
2. Connect to the server.
3. Spawn on russian base, pick T-34, all default.
4. Start engine.
5. Drive to the lagoon about 30 degrees off to your right.
6. As soon as your tracks touch water, your tank will dive into the sea.

The outcome depends on whether you host the mission on DServer or locally on an IL-2 Client game.

Hosted on DServer, the server will crash a few seconds after you start diving into the water.

Hosted locally on a Client game, you will die and further dive down to the deepest point roughly in the middle of Gelendzhik bay, after which you keep lying there, dead. The client game will not crash.

 

Mike

Tank Drowning Test 01.zip

Edited September 14, 2018 by SAS_Storebror

[DeadMeat0383]

@SAS_Storebror latest dev diary meantions improvement in multiplayer tanks stability... wonder if they found the root cause?

[SAS_Storebror]

Yep, sounds like a precise match.

[stupor-mundi]

On 9/14/2018 at 12:03 PM, SAS_Storebror said:

Alright, bug confirmed.

Attached you can find the test results of driving into shallow waters at the coast of Gelendzhik bay. 

Including mission file, all logs, tracks and DServer crashdump. 

 

I think if the bottom line of this bug is: vehicles falling below 0 altitude, we can be sure that this caused only a part of the crashes since 3.005. Plenty of missions far away from any sea have crashed regularly.

 

[SAS_Storebror]

But you've reported holes in the terrain here:

 

These holes might cause the same issue, and in that case, it can happen to any mission.

This would fit to what we've observed on our server.

At the times when our server was crashing frequently, we've been having AI ground units travelling along the road between Morozovskiy and Volotsky (Stanlingrad map), along the road heading south from Morozovskiy, and in the area of 1407.1.

Furthermore we've had player tank spawn points at the road between Morozovskiy and Volotsky.

At that time, our server crashed frequently, most of the times on the Summer/Autumn maps, rarely on the Winter one.

Even that seasonal difference makes sense because the terrain holes aren't necessarily the same across the seasons.

 

Nowadays, the whole ground action has been shifted away north to the 1107 area, and ever since then we've been free of server crashes altogether.

Could be that the new area doesn't have terrain holes, whereas the old one did.

 

Mike

[SAS_Storebror]

Not the slightest DServer glitch with 3.006 anymore, running for two days now.

I think our issues are solved.

 

Mike

[stupor-mundi]

I get the impression that the crashes are, at least mostly, fixed now, by the new update.

A new effect that can be seen when tanking, is that planes and tanks undergo a sudden yank, apparently a correction of their position (I saw a lot of that on the Efront server). I'm guessing a multiplayer/latency related position descrepancy between client and server is being corrected there. I've not seen that yet when flying, probably the larger scale makes it hard to perceive.

I can't help but wonder whether there is some connection between, fixing the crashes, and this new behaviour. On its own, the new behaviour could hardly be regarded as plus, since it obviously breaks immersion. Maybe before, there was some positional smoothing applied which now has been removed?

Since the update, I've pretty much spent only time on one server, so maybe this observation is not representative of the sim in general.