Measuring rig performance: Kuban test track

[chiliwili69]

As many of you know, Balapan provided a track record test flight to measure the performance in VR. Many people have run it to compare their performance with their rig peers and use it as a measuring tool for graphics settings and software/hardware tweaking.

 

This is the thread for Balapan test flight:

https://forum.il2sturmovik.com/topic/29322-measuring-rig-performance-common-baseline/

 

The "problem" with that track is that the top rigs were achieving scores very close to 90 average fps. In addition, the plane used in the Balapan track has not mirrors (they were not introduced at that time), so it was not possible to measure the impact of mirrors.

 

Some people has been commenting about the performance of the Kuban map, with mountains, sea water, etc.

 

I have created a new test flight with the Kuban map trying to capture most of the items that affects performance: smoke and shooting, clouds at 1000m, sunlight and reflections, sea water, dense city, spitfire plane with mirrors, etc. But it was difficult to cover all these items in just 1 minute, so this test flight takes 170 seconds, so almost 3 minutes.

 

You can download the test track from this link:

https://drive.google.com/open?id=1gFeanZBpc6myubq31Y_jgrLyDo_z6xad

 

To perform the test just follow the same instructions and settings than in the Balapan thread indicated above (don´t need to be repeated here) but with just three exceptions:

 

1.- Configure Fraps to record 170 seconds:

 

2.- In the graphics Settings use:

    - Screen resolution 1600x900 (it is easier to read and it doesn´t affect performance at all)

    - Mirrors: Medium

 

3.- Adjust the view of the VR device to match this view. The reference is to just see a little of the rudder pedals:

 

I have also created a new tab called "Kuban2018" to record all test flights with this track in the same link:

https://docs.google.com/spreadsheets/d/1gJmnz_nVxI6_dG_UYNCCpZVK2-f8NBy-y1gia77Hu_k

 

If you run this Kuban test, please report it in this thread.

Edited February 4, 2018 by chiliwili69

[chiliwili69]

I have run the test with i7-4790K@4.9, RAM at 2933 and low 1070 GPU overclock and the results are:

 

Frames: 7685, Min:38, Max:65, Avg:45.206

 

I don´t know which of the items of this track (mirrors, sunlight, clouds, mountains, etc) is the major responsible for this low fps. But it would be good to know it taking into account that this year is the Kuban year!

[SCG_motoadve]

What SS setting in Oculus?

[dburne]

I am 50 missions into a Spit PWCG campaign over Kuban.

 

The map is definitely more demanding, it is gorgeous but performance in VR at least for me is a little less than over Stalingrad ( flying PWCG campaign).

Still certainly doable, but little less. Not so bad as to induce stutters for me, still very smooth flying with ASW off.

 

Pretty sure it is just the map.  

[chiliwili69]

 

 

What SS setting in Oculus?

 

Just same settings than Balapan test with the three exceptions indicated above

[Madmatt]

I'll be running this latest test this week and reporting my findings. I do wonder if the various attempts to address the Spectre and Meltdown CPU exploits will affect the CPU scores though. I did some tests a few weeks ago on my 7700K which is now overclocked to 5.0Ghz and I noticed a decrease in some CPU Mark scores when compare to the time before the patches for Meltdown and Spectre were released. Of interest was that the results were not negligible as Intel, Microsoft and others would want you to believe. 

 

I won't guess as to how this may affect this new test or performance in IL-2 in general (I have not noticed any dramatic decreases in FPS while playing over the last month or so) but for statistical sake it may be interesting to also rerun the old Balapan test to see if we can detect a downward trend between tests done last year before the patches started getting being pushed out and now.

 

Madmatt

[katdog5]

I have run the test with i7-4790K@4.9, RAM at 2933 and low 1070 GPU overclock and the results are:

 

Frames: 7685, Min:38, Max:65, Avg:45.206

 

I don´t know which of the items of this track (mirrors, sunlight, clouds, mountains, etc) is the major responsible for this low fps. But it would be good to know it taking into account that this year is the Kuban year!

 

 

Ive come to the conclusion that its the clouds and sun rays. 

 

I'd recommend to add another test run if you have the time by sliding your Dynamic Resolution factor to .8 to see the results after you've done this frame crusher so you can see the comparison. I bet you go to close to +30fps ave with this one tweak.  i also predict that will be the main conclusion of this testing. brilliant piece of programming. 

Edited February 5, 2018 by katdog5

[dburne]

Ive come to the conclusion that its the clouds and sun rays. 

 

 

 

Are the clouds and sun rays different for Kuban map, than Stalingrad or Moscow?

 

I was under the impression the clouds and sun were the same for all the maps? I certainly could be wrong in that.

[katdog5]

I think theyre the same...just majorly responsible. typcial drops to -25fps when DRF set at 1. mountains too add another level. I have to go .7 DRF to fly in the cloudy mountains using 1.5 ss, its great though ...

[Madmatt]

I just had time to run a few tests of this tonight but then realized I still have the 3DMigoto mod installed. Because that mod does incur some performance impact (although latest versions are better than ever) I recorded results with it installed and uninstalled both. Perhaps these test results will benefit those that are evaluating what sort of overhead 3dMigoto demands. I will also get my monitor benchmarks later this week and post that as well.

 

My system speeds are slightly different than what's listed on the spreadsheet for me, as noted below. 

 

IN VR w/:3DMigoto installed

2018-02-07 20:10:12 - Il-2

Frames: 7634 - Time: 170000ms - Avg: 44.906 - Min: 39 - Max: 69 

This score updated in my post below

 

IN VR: no mod

2018-02-07 20:49:17 - Il-2

Frames: 11702 - Time: 170000ms - Avg: 68.835 - Min: 44 - Max: 91

 

 

CPU 5.0Ghz (no AVX offset)

CPU Mark: 14439, CPU Single Thread: 2936 (measured with Performance test v9.0)

GPU Clock: 2050 MHz (it will go as high as 2063 MHz in some instances, but this track never makes it go higher than 2050 MHz)

Mem Clock: 11922 MHz

Drivers: 390.77

 

 

Madmatt

Edited February 8, 2018 by Madmatt

[HunDread]

This looks like a huge impact. I thought the latest version had minimal impact on fps.

[Madmatt]

This looks like a huge impact. I thought the latest version had minimal impact on fps.

 

It does, but I will retest with the mod re-installed to verify that score. I'll be sure to run the test after a clean reboot etc...

 

I may also test with various settings enabled like the prop turned on/off, label occlusion on/off, smaller gunsight on/off... Just a matter of time as I currently don't have much free until later this weekend.

 

Madmatt

Edited February 8, 2018 by Madmatt

[OrLoK]

I cant go back to not using the 3DMigoto mod now. It feels fine. rarely any  gfx anomalies. I *might* be losing frames but the ASW makes up for it and 3DMigoto fixes all the things I didnt like about ASW.

 

Prop off/smaller sight/enhanced IPD at at default settings.

[Madmatt]

Okay, I had a hunch that a clean reboot and reinstall of 3DMigoto would have a positive effect and i was right, here are some new results. I will remove the test result from above as it could be misleading.

 

 

3DMigoto installed but all settings at default (Prop On, Gun Sight Stock)

2018-02-08 14:31:37 - Il-2

Frames: 9628 - Time: 170000ms - Avg: 56.635 - Min: 43 - Max: 91

 

3DMigoto installed (Prop Off, Gun Sight Turned Off)

2018-02-08 15:44:39 - Il-2

Frames: 9658 - Time: 170000ms - Avg: 56.812 - Min: 42 - Max: 91

 

3DMigoto installed (Prop On, Small Gun Sight)

2018-02-08 15:27:28 - Il-2

Frames: 9494 - Time: 170000ms - Avg: 55.847 - Min: 38 - Max: 91

 

3DMigoto installed (Prop Off, Small Gun Sight)

2018-02-08 14:35:22 - Il-2

Frames: 9614 - Time: 170000ms - Avg: 56.553 - Min: 34 - Max: 91

 

Synopsis:

 

#1 The smaller modded gun sight DOES half a slight impact on the minimum frame rate but not on the average.

 

#2 The mod DOES have an overall impact on average FPS performance as compared to running stock with mods uninstalled, regardless if any of the new effects are enabled or not. 

 

For comparison, here is my No Mod score as reported above:

 

IN VR: no mod installed

2018-02-07 20:49:17 - Il-2

Frames: 11702 - Time: 170000ms - Avg: 68.835 - Min: 44 - Max: 91

 

Madmatt

Edited February 12, 2018 by Madmatt

[HunDread]

Thanks for sharing this Madmatt.

[chiliwili69]

Thank you Madmatt for being the first to run this test. You are now in the top of the Kuban list!

Thank you also for the mod comparative. (personally I have never used mods)

 

You achieve a nice avg fps with your top rig, but still far from 90. The curious thing of this track is that it uses HIGH settings and medium settings for most of the options. I believe that if we max out everything the results could be worse.

 

I will do more testing this weekend with my new 1080Ti and other graphics settings and analyze the influence of mirrors.

[TG-55Panthercules]

#2 The mod itself does have an overall impact on average FPS performance as compared to running stock with mods uninstalled, regardless if any effects are enabled or not. 

 

For comparison, here is my No Mod score as reported above:

 

IN VR: no mod installed

2018-02-07 20:49:17 - Il-2

Frames: 11702 - Time: 170000ms - Avg: 68.835 - Min: 44 - Max: 91

 

 

Thanks for providing all this info, but I'm confused about your statement above.

 

How/why can you say that "The mod itself does have an overall impact on average FPS performance" when your results without the mod installed are reported as 68.835 (AVG) and your results with the mod installed all hover around 56 (AVG), which seems to me to indicate that the mod is exacting about a 12 FPS (about 18%) performance drop?

 

Am I misinterpreting your results somehow?

[TUS_Samuel]

How/why can you say that "The mod itself does have an overall impact on average FPS performance"

 

My brain inserts "not" into this sentence as well.

[TG-55Panthercules]

My brain inserts "not" into this sentence as well.

 

LOL - so weird.  Sorry about that 

[Madmatt]

Yeah, I guess I could have written that better. Bottom line is that the mod affects average frame rates. Not surprising, you can't get something for nothing.

 

 

 

I will do more testing this weekend with my new 1080Ti and other graphics settings and analyze the influence of mirrors.

 

I wanted to retest without mirrors too. Something tells me those are eating up frames. I will try and do that this week as well. 

 

One last thing I am noticing, the position of your head (or the HMD I guess) can have a HUGE impact on reported FPS with these tests. I did one test where I think my head was just a few degrees off from your example position and the framerates were different by nearly 10%. I wish there was a way we could ensure that all tests are done with the exact same head position. What i did was place my HMD on a small table and then propped it up and covered the proximity sensor (to keep it activated) until i perfectly matched your baseline image. Maybe a better way to do these tests is to set the HMD on  stable surface and simply "recenter" the view before launching. That would keep the view consistent throughout the test and everyone that is reporting results would have the same test environment.

 

Just an idea, but all the test results are tainted a bit by small differences in position we introduce when wearing the HMD ourselves.

 

Madmatt

Edited February 12, 2018 by Madmatt

[chiliwili69]

 

 

One last thing I am noticing, the position of your head (or the HMD I guess) can have a HUGE impact on reported FPS with these tests. I did one test where I think my head was just a few degrees off from your example position and the framerates were different by nearly 10%. I wish there was a way we could ensure that all tests are done with the exact same head position. What i did was place my HMD on a small table and then propped it up and covered the proximity sensor (to keep it activated) until i perfectly matched your baseline image. Maybe a better way to do these tests is to set the HMD on  stable surface and simply "recenter" the view before launching. That would keep the view consistent throughout the test and everyone that is reporting results would have the same test environment.

 

Yeah! that is important.

When I wrote the initial benchmark (Balapan) I included a picture of how I put my Rift in the top of the chair fixed with a rope (and with a hanky inside to activate sensor).

The test should not be done using the head as support of the Rift. (The image you see in the screen is different from the one shown at the monitor).

The reference picture was taking from the monitor, so the procedure should be something like:

 

1. Fix the Rift to a support (table, chair, etc) and put a hanky inside. Try to put it as centered as possible, positional and rotational.

2. Launch Fraps, IL-2 , ASW off and launch track

3. Adjust the vertical rotation to match the image in the monitor

4. Press H and then P.

[wheeliemonsta]

Hi guys,

 

Has there been any word on whether there will be any VR optimizations/changes/improvements in 3.001?

Just wondering if anyone has read anything in any of the announcements/comments from devs, etc.?

 

Cheers

[SCG_motoadve]

I actually read that no VR optimizations will come in the next patch. 

[C6_lefuneste]

Thanks for your track and your methodology.

I play it on my less powerfull rig (I2500K @ 4,5 GHz and GTX1070 at standard clock speed), with my Pimax 4K caped to 60 hz and maybe more agressive graphic settings, and here is what I have:

IN VR: no mod installed : 

Frames, Time (ms), Min, Max, Avg
  6504,    170000,  26,  46, 38.259

 

IN VR: mod installed

Frames, Time (ms), Min, Max, Avg
  6149,    170000,  25,  44, 36.171

 

So, I loose only 5% with the mod...I do not know why you loose about 20% and not me...But your min fps is higher than my max !

I did not manage to get even one time the max fps of the Pimax, what are your graphics settings ?

 

 

Edited March 8, 2018 by lefuneste

[C6_lefuneste]

I didn't notice that by default Piplay + steamVR gave a resolution of 2014x2486 per eye !

I lower SS to 0.7 in steamVR and with 1249x1541 per eye I have the following results:

 

IN VR: no mod installed : 

Frames, Time (ms), Min, Max, Avg
  8036,    170000,  33,  61, 47.271

 

IN VR: mod installed

Frames, Time (ms), Min, Max, Avg
  7653,    170000,  33,  59, 45.018

 

so still the same gap introduced by the mod...

Edited March 9, 2018 by lefuneste

[BeastyBaiter]

There is absolutely no point in reporting results with greatly reduced resolution. Whenever you set DSR, pixel density or super sampling to anything below 1.0, you are running BELOW the native resolution of the device. It's the same as reporting 1440p results by setting your 1440p monitor to run at 1080p.

[C6_lefuneste]

Did you know a VR Helmet with a native resolution of 2014x2486 per eye ? The input resolution of Pimax 4K is 1280*1440 upscaled to 1920*2160 by the hardware. But their Piplay software is giving to steamVR and so IL2 a resolution of 1249x1541...