Multiplayer stutters cause identified

[RossMarBow]

CB in orange 
Finnish in green

 

It seems to be cause by spawn points
as CB training has the most spawn points and the biggest spikes

followed by Berloga

then Finnish

 

WOL and EU official have no spikes and a similar number of spawn points to CB

 

Is it possible that the devs tweak how spawn points update so they are staggered and not all updating at once?

Edited November 15, 2022 by RossMarBow

[Drum]

Those last two charts are identical.

[RossMarBow]

6 hours ago, Drum said:

Those last two charts are identical.

Yea good point.

 

 

I have over 30 frame time plots now and so I don't know which is which anymore.

I was viewing them as I made them, and taking screen caps as I went, so didn't change the file names.

CB training had the biggest spikes by far. Which has the most spawn points.

Berloga and Finnish looked the same, maybe if I knew which was which and looked hard they might have different sized spikes.

WOL/EU offical/CB had no spikes.

Berloga might be this one.

 

 

 

This is finnish with the game uncapped running at 282 fps when looking straight up at the blue sky for the last 7s
You can see how even with an insane frame rate the spikes are even more insane.
Thats a drop from 282 to 55 fps.
My average fps is about 100-120fps spawning in in the ground.
But as you can see when fps drops down closer to 60fps 
The spikes are always on top of the average.
So someone getting 60fps is gonna get spikes down to god knows how low fps.

This must be terrible for people who play at low frame rates because in VR your going to be dropping below reprojection.
And if you have a gsync/VRR monitor its going to be dropping out of range resulting in the screen dimming.

For example, playing at 143 capped fps as I do and get flying (lower closer to the ground) I get a drop to 55fps.
The VRR range on my monitor is 45-144. So if my spikes drop me below 45 fps which happens sometimes my entire monitor dims for that/those frames. If I turned graphics up so my average fps was lower this would result in an almost strobe light effect with a dimming every 5-6s.

Edited November 15, 2022 by RossMarBow

[IckyATLAS]

One way to avoid spawn spikes is not to spawn whenever possible. I mean that is to have deactivated elements that you activate when needed. 

It is feasible when the maximum number of elements to spawn can be known in advance or at least be planned. Then you can create a "background store" of deactivated elements that get activated when needed. Activation of a deactivated element is probably much faster than spawning. On the other hand, the mission is more "loaded" from start. It is a matter of compromise here. The advantage of activating and deactivating is that you can do it for complete flights (leader and wingmen) which you cannot do with spawning. Activation/deactivation preserves links but spawning/delete not.

Edited November 15, 2022 by IckyATLAS

[RossMarBow]

1 minute ago, IckyATLAS said:

One way to avoid spawn spikes is not to spawn whenever possible. I mean that is to have deactivated elements that you activate when needed. 

It is feasible when the maximum number of elements to spawn can be known in advance or at least be planned. Then you can create a "background store" of deactivated elements that get activated when needed. Activation of a deactivated element is probably much faster than spawning. On the other hand, the mission is more "loaded" from start. It is a matter of compromise here. The advantage of activating and deactivating is that you can do it for complete flights (leader and wingmen) which you cannot do with spawning. Activation/deactivation preserves links but spawning/delete not.

 

Sounds good but is a bit beyond me.

All I know is I started playing this game at the end of 2020.
I first reported this problem in early 2021.
Finnish has been the most popular server by far.
It is also very new player friendly.
And is currently the only popular server that has tanks in multiplayer.
Every single person playing on that server has been plagued by this problem.
I'm not a VR person but dropping into reprojection every 5s sounds like a big issue.

At first I thought it was a problem universal to IL2 multiplayer, as everyone else could replicate the same problem.
It wasn't until I went though all the servers did I realise it was a specific problem.
 

[SYN_Vander]

8 hours ago, RossMarBow said:

CB in orange 
Finnish in green

 

It seems to be cause by spawn points
as CB training has the most spawn points and the biggest spikes

followed by Berloga

then Finnish

 

WOL and EU official have no spikes and a similar number of spawn points to CB

 

Is it possible that the devs tweak how spawn points update so they are staggered and not all updating at once?

 

Not sure I understand completely... What do you mean by spawn points? Airfields that player can spawn from?

And what is happening every 5 seconds? ('updating')

[AEthelraedUnraed]

10 hours ago, RossMarBow said:

It seems to be cause by spawn points
as CB training has the most spawn points and the biggest spikes

I miss the causal connection here. Remember that correlation does not imply causality!

 

10 hours ago, RossMarBow said:

WOL and EU official have no spikes and a similar number of spawn points to CB

Isn't this a rather good argument *against* spawn points being the problem?

[MisterSmith]

This doesn't belong in general. I can't decide between Multiplayer, complaints, or technical issues. I'll let you decide but it is getting moved by tonight.

 

Smith

[MisterSmith]

I re-reviewed the criteria for Multiplayer and it doesn't fit. Nor does it fit the criteria for a bug report. Complaints it is.

 

Addendum:

 

The forums are for exchanging ideas and information. It is not reasonable to expect members not to post in threads as they wish, so long as they abide by the rules. Even possibly, and perhaps specifically, opposing viewpoints.

 

Smith

[SYN_Vander]

14 hours ago, RossMarBow said:

Yes, spawn points are locations on multiplayer servers that the player can click on and be taken to a window where they are offered a selection of tanks or planes to spawn from. Spawn points also show how many other players have spawned there. And how many enemy planes and tanks are within a 10 km circle. You should check out multiplayer IL-2 sometime it is a lot of fun.

 

 

I know multiplayer, in fact I host a multiplayer server. The term 'spawn point' was not clear to me as it could also mean a spawn object in the mission that is used to spawn AI aircraft. I'm still interested in those spikes every 5 seconds. How is this related to the airfields with spawn locations? Is this a mechanism in the mission being used on that particular server or a generic effect from airfield objects because I have not seen it before.

[RossMarBow]

On 11/17/2022 at 5:04 AM, SYN_Vander said:

 

I know multiplayer, in fact I host a multiplayer server. The term 'spawn point' was not clear to me as it could also mean a spawn object in the mission that is used to spawn AI aircraft. I'm still interested in those spikes every 5 seconds. How is this related to the airfields with spawn locations? Is this a mechanism in the mission being used on that particular server or a generic effect from airfield objects because I have not seen it before.

I have been doing further testing.

 

CB main server was giving out overloaded server messages with a lot of AI bombers in the air.
Only 6 spawn points present, but huge fps spikes.
So I'm thinking its purely a function of server performance /load level now.
And maybe spawn points put a lot of load on the server too.

 

I asked around and someone came up with this explanation. 
"any real-time multiplayer engine that has synchronization clock between involved players will slow time, fps, call whatever to keep sync"

 

I guess the issue is now to make sure people running servers know they can cause client side fps drops/stutters if the servers running poorly.

 

But spawn points might benefit from some optimisation too. 

Without a  server I can measure further understanding of this is limited. 

 

Edit: the next map on combat box had a lot going on including big flights of single engine AI squadrons doing ground attack and no ping spikes. 

 

 

Edit:
Here is an example of everything working ok. On a multiplayer server (combat box).

 

Edit just a smaller update and comparision between two maps on combat box.
This time making use of spectator mode which spawns the camera on the edge of the map so is the most raw data form.

Battle_of_the_Scheldt with 4 players 

this is the same map - not sure on the player count - but spawned in an aircraft at least
 

you can see the spikes always exist - so only severity is the issue

 

and just to be clear the spikes are constant in timing and height for the most part the random big spikes in these two graphs are not related to the issue 
i highlighted the offenders here

 

 

based on an average FPS of 143
(if your average fps is lower you are going to suffer more, as seen in the graph showing 7s of 283 fps)
considering both flat screen and VR uses 
spikes above 22 ms = 45 fps
which is below the reprojection speed of the G2 headset which is the most common headset
not a VR user myself but I believe this can cause nausea 
dropping below 90 fps on the G2 results in a screen tear

However, for flat screens drops below 50 fps (or spikes above 20 ms) cause variable refresh rate (VRR) gsync or freesync monitors to drop out of their range which causes dimming for that frame. Constant spikes above 20ms therefore has a slow strobe like effect. Which results in user discomfort. 
 

 

To put this issue as simply as possible:
high server loads making spawn points a real issue - that has physical effects on client user hardware

Edited November 25, 2022 by RossMarBow

[dgiatr]

Hello something possibly relevant?

 

 

[RossMarBow]

On 11/17/2022 at 5:04 AM, SYN_Vander said:

 

I know multiplayer, in fact I host a multiplayer server. The term 'spawn point' was not clear to me as it could also mean a spawn object in the mission that is used to spawn AI aircraft. I'm still interested in those spikes every 5 seconds. How is this related to the airfields with spawn locations? Is this a mechanism in the mission being used on that particular server or a generic effect from airfield objects because I have not seen it before.

 

The player selectable spawn points.
With the theory being that they all update the enemy plane count within 10km at the same time.

I don't think this is anything to do with AI spawn points as Berloga shouldn't have any, yet at this problem.

[stupor-mundi]

I want to chime in, since I'm convinced that this 'spawn point' issue, which for many people causes a stutter issue, for a subset of players, such as myself, causes what I call slow-motion-syndrome, basically, a handicapped vehicle. I experience this with tanks, but in theory it should happen to planes as well. I have an AMD Ryzen laptop, so I suspect the slo-motion issue is experienced by people with rel. low single thread cpu speed and/or amd.

Stutters are quite annoying, but the slow-motion effect makes multiplayer tanking (and maybe flying) actually unfair, so I would call upon the devs to take some interest in the topic.

Going by my experience in the AAS tank missions, the ones with the least slo-mo effect are those where the mission logic doesn't SWITCH spawn points based on the taking of flags, for instance Tessy and Etretat. Those have multiple spawn points fairly close to each other, but they are active from the beginning and don't change. Among those missions with switching spawn points, the one with least slo-mo for me is Prelestnoye, which I believe has spawn points relatively widely spaced.

The issue with spikes doesn't seem impossibly hard to solve, and, after all, it was introduced with an update at some point. I well remember a long period of tanking before this issue arose. I first experienced it when AAS intruced the Engelskirchen mission.

So why are the devs uninterested? This is a massive bug.

[=THORN=Sphere-]

@RossMarBow

 

Been following your post for a couple of days but have the same issue with VR. (Quest 2 and quest 3 with open composite to skip steam VR).  

I've had the issue for years with some servers and maps having an almost constant micro stutter sometimes every second. One thing that eliminates the micro stutters sometimes is turning there HUD off.

  The worst server for me is EU with the labels, especially when it gets busy it becomes unplayable sometimes. 

 

I run at 72hz on my head set and turn ASW off (frame generation).

 

I'll get some frame times from my headset the next time I'm on but I'm pretty sure they show the same results as you've proven above but in the 40/50 range due to latency. 

 

For reference my PC config is 

Rtx 3080

Ryzen 5800x3D

Win 10

 

Cheers for continuous testing, hopefully some one will pick it up and look into it with a possible resolve 

[=MERCS=JenkemJunkie]

The stutter and performance loss from icons and other things related to the GUI is from a different cause of stutter, described in this thread. You'll never get a stutter free experience in IL2, the best you can do is reduce it. They're changing the GUI for the next project, so don't expect any fix until then.