dserver transmits extremely long range flak data to pilots

[Alonzo]

Hello. I've been working on reports of poor performance when pilots fly on the Combat Box server. For some pilots, when they enable track recording, they see a large drop in FPS and their wingmates say they start to 'warp' around the sky. If they switch off the in-game recording feature, everything goes back to normal.

 

The Combat Box maps do have a lot of stuff going on, but I have been optimizing the maps, and server-side performance looks good. I asked the players to send me a TacView file, so I could look at some of the events being transmitted to their game clients. I wanted to see if I had too many active units on the map.

 

Here are some example screenshots from the TacView analysis of the mission. In this first picture, a squad has just taken off from Eindhoven. The squad only sees aircraft in the immediate vicinity of the aerodrome, but the TacView shows flak firing 80km to the west, near Antwerp (there is heavy flak on this mission and several mission objectives in that area of the map). So while they are 80km away, well beyond the render range of the flak, the server is sending "shots fired" data to these pilots' game clients:

 

 

Here's a second example. In this shot, the red P51 has just become visible in TacView. There are lots of planes on screen, and it looks like dserver has decided to transmit the position of the enemy plane at a range of about 10km. This is consistent with optimizing data being sent to clients. But look in the distance -- flak puffs (yellow) are exploding 80km away on the other side of the map. So the server has chosen to only tell the game client about the P51 at a range of 10km, but is communicating about flak puffs at a distance of 80km.

 

 

My suspicion is that this is what is causing the poor performance for some pilots -- their game client is processing lots of long range invisible flak bursts that are not relevant to the pilot's experience but do cause CPU usage on the client. When they enable track recording, the extra CPU load causes them to see bad performance, stuttering, etc. My theory is that since the new longer range visibility patch (which is great! thank you for this!) the server is sometimes sending more data than it should to the clients.

[Alonzo]

Also, mad props to =gRiJ= for fielding a giant squadron and putting the fear of god into the opposition! Nice!

[-DED-Rapidus]

Сontact the server holders. they can theoretically set up complex triggers or check zones for the appearance of an enemy, which will first reduce the load on the server

[Nocke]

Rapidus, with all respect - I believe this deserves to be read again with a little more time and concentration.

[Talon_]

3 hours ago, -DED-Rapidus said:

Сontact the server holders. they can theoretically set up complex triggers or check zones for the appearance of an enemy, which will first reduce the load on the server

 

You just responded to one of the administrators. I'm also an administrator ?

[1PL-Husar-1Esk]

This distance flak is activated by distant players which are not recorded in the track. If I understand correctly why it is waste of net  resources. BTW for replay recordings you can choose not to record ground activity if I remember correctly , I wonder if this can help with stuttering. But imho why even have this data at 80 km away flak in net packed anyway ? 

[-DED-Rapidus]

2 hours ago, 216th_Nocke said:

Rapidus, with all respect - I believe this deserves to be read again with a little more time and concentration.

I carefully read this topic looking for a bug, but it all came down to thinking why anti-aircraft guns were in the output of information (tacview)

[FTC_DerSheriff]

14 minutes ago, -DED-Rapidus said:

I carefully read this topic looking for a bug, but it all came down to thinking why anti-aircraft guns were in the output of information (tacview)

Summing it up:

Most stuff isnt renderinig for a client beyond. 10km

Flak however is. Regardless of his distance to the flak.

An admin can programm that a flak isnt spawned in when when nobody is near the flak unit.
But  player's #1 client is still processing flak bursts even when they are targeting player #2 80km away. That flak is not visible for player #1.
But is still taking processing power away.
That is beyond the admins control, since the flak is spawned in to shoot at player #2.

thats your bug 

Edited February 12, 2020 by DerSheriff

[Alonzo]

3 hours ago, -DED-Rapidus said:

I carefully read this topic looking for a bug, but it all came down to thinking why anti-aircraft guns were in the output of information (tacview)

 

This is a complex mission with 80 players flying on the server. We have check zones that deactivate guns when players are far away. In this case, though, there are players near the flak guns. Maybe a diagram will help.

 

 

Point 1: Location of player who is recording the track.

Point 2: Flak guns, with check zone, and other players on the multiplayer server. We cannot see the other players in the track, but they are there.

 

The players and guns at "point 2" are far away from the player who is recording the track. There is no reason for the server to tell the recording player, at point 1, about the players far away from them. There is never a reason for the server to tell game clients about flak that is more than ~15km away, because it doesn't render for the client. The server is transmitting data about flak puffs 80km from the player, which wastes server CPU, server network, client network and client CPU.

 

If you are able to show this picture to a network programmer, they will realize there is a bug. The technique is called network traffic culling and dserver is definitely doing it, just there is a small bug in the implementation for heavy flak guns.

Edited February 12, 2020 by Alonzo
Image size

[-DED-Rapidus]

8 hours ago, -DED-Rapidus said:

set up complex triggers or check zones for the appearance of an enemy, which will first reduce the load on the server

"I believe this dserves to be read again with a little more time and concentration."

[Alonzo]

1 minute ago, -DED-Rapidus said:

"I believe this dserves to be read again with a little more time and concentration."

 

Here's the mission. If I have made a mistake with the check zones, maybe you could let me know the mistake.

 

 

 

Battle_of_the_Scheldt_Sep_1944.zip

[-DED-Rapidus]

I'm sorry, but my first question is, where is this trigger? need a simple example to understand the problem, thank you.

[Talon_]

Forget about the check zone - why is the server sending data from events 80km away at all?? It should cull anything outside 20km...

[Alonzo]

@-DED-Rapidus Here's an explanation:

• In the recorded track, the player had just taken off from Eindhoven airfield (grid 1514).
• The flak is firing somewhere north of Antwerp, I believe it is at the objective "Kampfgruppe Chill" (grid 1606).
• The flak is controlled by a group called "Axis AAA Destructible Medium". You can find this in the object browser by expanding Battle of the Scheldt > Axis Kampfgruppe Chill > Axis AAA Destructible Medium.
• Flak switches on using the CLOSER check zone, editor ID 12428, within the flak group.
• This is spawn/delete style flak which was introduced as a workaround when "activated" flak was invisible (bug both introduced and fixed some time in 2019).

Please understand, I am not some random player who has no idea what he is doing. I've been building maps for a year and I run the Combat Box multiplayer server, which is one of the "top 3" servers. I really do believe that this is a bug in the network culling code. I think it's a small bug that seems to have quite an outsize performance impact for some players. I know it's your job to shield the developers from things that would waste their time, and I respect that, but I believe this is definitely a bug.

 

Thank you for your time and attention.

[-DED-Rapidus]

36 minutes ago, Talon_ said:

Forget about the check zone - why is the server sending data from events 80km away at all?? It should cull anything outside 20km...

because the anti-aircraft gun is activated and works, another question is, every time this happens with so view or randomness

35 minutes ago, Alonzo said:

@-DED-Rapidus Here's an explanation:

• In the recorded track, the player had just taken off from Eindhoven airfield (grid 1514).
• The flak is firing somewhere north of Antwerp, I believe it is at the objective "Kampfgruppe Chill" (grid 1606).
• The flak is controlled by a group called "Axis AAA Destructible Medium". You can find this in the object browser by expanding Battle of the Scheldt > Axis Kampfgruppe Chill > Axis AAA Destructible Medium.
• Flak switches on using the CLOSER check zone, editor ID 12428, within the flak group.
• This is spawn/delete style flak which was introduced as a workaround when "activated" flak was invisible (bug both introduced and fixed some time in 2019).

Please understand, I am not some random player who has no idea what he is doing. I've been building maps for a year and I run the Combat Box multiplayer server, which is one of the "top 3" servers. I really do believe that this is a bug in the network culling code. I think it's a small bug that seems to have quite an outsize performance impact for some players. I know it's your job to shield the developers from things that would waste their time, and I respect that, but I believe this is definitely a bug.

 

Thank you for your time and attention.

thank you for the detailed answer, I will ask you to track with tacview files

[Alonzo]

Here are steps to reproduce, with a mission to reproduce the problem, and a track showing the problem.

1. Enable track recording including TacView recording on server and client.
2. Load mission into dserver (real dserver, not game client hosted deathmatch).
3. Connect to dserver and spawn a plane from the airfield near Eindhoven.
4. Record track for several seconds.
5. Observe that aircraft near Antwerp are not visible in the recording, but flak puffs are visible in TacView.

Expected behavior is that server 'culls' the flak and does not send it to the client, because player plane and flak are very far apart.

 

The AA guns are not attached to a check zone. It would make no difference. The guns should be active because there are P51 enemy planes nearby.

 

 

dserver-long-range-flak-dogfight.2020-02-12_10-21-34_00.zip dserver-flak-long-range-bug-alonzo.zip

Edited February 12, 2020 by Alonzo

[Alonzo]

Update: It looks like the issue affects not just "puffy" heavy flak, but all flak. Here's a video showing tracer fire at a player airfield ~100km away. No planes are visible, but my game client is being sent the data and is processing it.

 

https://www.youtube.com/watch?v=D9X6ZUE7JOk

[Alonzo]

And also parachutes from long distance:

 

[DD_Arthur]

Here you go Alonzo.

 

This is a bug and a half you've found! 

[Alonzo]

Hello. This issue still seems to be present. @-DED-Rapidus is there anything I can do to add information? Do you or the developers agree it might be a bug, or it's intended that all clients receive flak/AA shots everywhere on the map?

[RedKestrel]

Does this still exist in 4.006? I ask because it seems like after the DM a lot of the frustration around certain weapons online may be traced back to the server dropping packets because of too much info being sent/received. On a map with a lot of AA going on around airfields and ground targets its easy to see how this would quickly overwhelm the server, and probably contribute to issues with people seeing 'hits' on enemy aircraft that are in fact not registered.

[Alonzo]

@-DED-Rapidus 

 

Привет. Это все еще проблема. Я думаю, что это вызывает проблемы с производительностью сервера.

 

У меня есть записанный трек, показывающий огонь наземного трассера примерно в 50 км от моего самолета. Я считаю, что ошибка в том, что каждый выстрел из наземного оружия передается каждому клиенту игры, независимо от их расстояния от оружия. Это вызывает проблемы на сервере и для игровых клиентов.

 

Не могли бы вы дать мне знать, как я могу помочь объяснить проблему лучше? Я верю, что ваш программист по сетевому коду довольно быстро распознает проблему, если увидит ее в действии.

 

Вот прикрепленный трек. Откройте его в TacView. Заметьте, что мой самолет (Алонзо) находится в сетке 1706. Мой трек показывает зенитную артиллерию в 1401 году в 60 км. Это также показывает зенитную артиллерию в 1611 году, в 50 км. Я считаю, что это ошибка.

 

Hello. This is still an issue. I think it is causing server performance problems.

 

I have a recorded track showing ground tracer fire approximately 50km away from my aircraft. I believe the bug is that every ground gun shot is transmitted to every game client, regardless of their distance from the gun. This is causing problems on the server and for the game clients.

 

Could you please let me know how I can help explain the problem better? I believe your netcode programmer will recognize the problem quite quickly, if they can see it in action.

 

Here is an attached track. Open it in TacView. Observe that my aircraft (Alonzo) is in grid 1706. My track shows flak in 1401, 60km away. It also shows anti-aircraft artillery in 1611, 50km away. I believe this is a bug.

 

 

@-DED-Rapidus Here is a video showing the problem. dserver is transmitting anti-aircraft artillery shots to game clients even if the player is more than 50km away. I believe this is a bug.

 

Вот видео, показывающее проблему. dserver передает выстрелы зенитной артиллерии игрокам, даже если игрок находится на расстоянии более 50 км. Я считаю, что это ошибка.

 

 

[AbortedMan]

On 2/16/2020 at 4:41 PM, DD_Arthur said:

Here you go Alonzo.

 

This is a bug and a half you've found! 

The bug here is that the track player shouldn't be displaying parachutes and flak so far away outside of the "display bubble". People observing this are mistakenly attributing it to a "network culling bug" when there is no network culling feature in IL2 multiplayer. All of the relevant data from every entity is being sent to each player during gameplay whether it's visible on the track or not. 

 

The issue is too many AI entities in the mission and/or dserver can't handle so many AI/player/network operations on one thread. Whoever owns that particular issue is up for debate based on the intention for what parameters the game limitations are designed.

 

Are missions with 50+ active AI AA with 80+ players really feasible?

[Alonzo]

You’re not helping.

 

Player planes are culled. Previously using a strict 10km radius, now with a more dynamic radius. Flak shots also should be culled. Saving the CPU cycles to package up this data to send to 84 clients would help both client and server performance.

[LLv34_Temuri]

This seems to be due to the "we sync everything" netcode the game engine has.

 

It's been said earlier, and I say it again: the netcode needs to cheat where it can. The game should not send data about things from other side of the map to a client, when that client has no way to be able to even see the action that originates the data.

[Alonzo]

2 hours ago, LLv34_Temuri said:

The game should not send data about things from other side of the map to a client, when that client has no way to be able to even see the action that originates the data.

 

I agree, and this is why I think this is a bug, rather than some kind of core design flaw with the netcode: the distant AA in that TacView video is being triggered by aircraft near the AA. But the aircraft are not in the track. dserver is (correctly) not telling my game client about aircraft that are a long way distant. But the server tells the game client about distant AA fire. Seems like a small code bug that could have a significant effect on server and client performance.

[Mitthrawnuruodo]

12 minutes ago, Alonzo said:

I agree, and this is why I think this is a bug, rather than some kind of core design flaw with the netcode: the distant AA in that TacView video is being triggered by aircraft near the AA. But the aircraft are not in the track.

 

The only thing that I can question is the assumption that the culling seen in TacView is representative of the culling of network traffic. As far as I know, it's possible that some other layer of culling exists between the network and the TacView output.

 

Other than that, the analysis seems flawless.

 

Has anyone actually looked at the packets that are being sent/received by the client? Perhaps some statistics from that test mission could be revealing.

[-DED-Rapidus]

As soon as I have any information on this issue I will respond.

[AbortedMan]

On 7/5/2020 at 6:25 AM, Alonzo said:

You’re not helping.

 

Player planes are culled.

I'm trying to help you understand. Player planes are culled from the recording track, not clients in game. Track recordings are not a representation of network traffic. The absence of a method of network transmission (network culling, in this case) is not a bug as if it can be fixed with a quick review of code. Network culling in the context of this thread would have to be added to the game first for it to be a bug.

[Alonzo]

2 hours ago, AbortedMan said:

I'm trying to help you understand. Player planes are culled from the recording track, not clients in game. Track recordings are not a representation of network traffic. The absence of a method of network transmission (network culling, in this case) is not a bug as if it can be fixed with a quick review of code. Network culling in the context of this thread would have to be added to the game first for it to be a bug.

 

My interpretation of the things I can observe:

• Network code on the server culls events "too far" from the player for them to see/act on them, in order to optimize server, client, and network performance.
• Track recording is a simple 'save' of network events received by the client. Whatever the game client receives, it writes to disk.
• TacView displays everything that is in the track.
• The server does a good job of culling aircraft outside of the view of the client, but has a bug where it fails to cull AA shots.

Your version:

• IL2 has a networking model that sends all events in a mission to all game clients all the time.
• Track recording is a filtered 'save' of network events received by the client. It doesn't save planes that are distant, but it does save AA shots. OR
• TacView filters distant planes in a track, but not AA shots.
• The server does no culling.

I understand what you are saying, I just like my interpretation better. My version credits the developers with using a simple technique that is at least a decade old now, that optimizes traffic for multiplayer games. My version doesn't require the mental gymnastics that tracks are somehow filtered views of network data. And my version leaves room for this being a small netcode bug that can be fixed, and that will massively help game client performance.

 

In support of my theory, when the 10km bubble was removed the release notes made it clear that server performance would likely change with that patch. If game state is always synced to all clients all the time, why would removing the 10km bubble do anything to affect server performance?

[AbortedMan]

1 hour ago, Alonzo said:

 

My interpretation of the things I can observe:

• Network code on the server culls events "too far" from the player for them to see/act on them, in order to optimize server, client, and network performance.
• Track recording is a simple 'save' of network events received by the client. Whatever the game client receives, it writes to disk.
• TacView displays everything that is in the track.
• The server does a good job of culling aircraft outside of the view of the client, but has a bug where it fails to cull AA shots.

Your version:

• IL2 has a networking model that sends all events in a mission to all game clients all the time.
• Track recording is a filtered 'save' of network events received by the client. It doesn't save planes that are distant, but it does save AA shots. OR
• TacView filters distant planes in a track, but not AA shots.
• The server does no culling.

I understand what you are saying, I just like my interpretation better. My version credits the developers with using a simple technique that is at least a decade old now, that optimizes traffic for multiplayer games. My version doesn't require the mental gymnastics that tracks are somehow filtered views of network data. And my version leaves room for this being a small netcode bug that can be fixed, and that will massively help game client performance.

 

In support of my theory, when the 10km bubble was removed the release notes made it clear that server performance would likely change with that patch. If game state is always synced to all clients all the time, why would removing the 10km bubble do anything to affect server performance?

A simple way to prove your interpretation is to take metrics of network output/input on a client machine while AAA is firing once in your assumed culling range and outside of your assumed culling range and see if there is a difference in output/input. Also measure this on the server at the same times. You can do this with Windows Task Manager or any network traffic app.

 

This type of data is something the devs will actually look at and reply to.

Also, this is still ignoring the fact that Dserver can't control all of those AI units at the player load this server populates, whether flak firing 80km away is affecting it or not. If fixing a culling bug is the proposed solution, that's a band-aid on a bullet wound. The real issue is offloading AI cycles somewhere else, using multi-threading, or whatever.