What would be a good benchmark flight for VR

[Dutch2]

It seems I have now get FCAT VR working, as it shows the red bar in my VR HS screen, so it does regonize IL2 and the O+. 

To compare settings on my pc I need an mission to run this benchmark, so I would like to know what would be the best mission to do the testing on. 

Starting on the ground or airstart at ????meters

lonely flight or lots of AI planes enemy or friendly 

lots of ground activity or empty terrain 

Above Stalingrad, snow terrain or full of trees

Clear sky or max clouds, rain? 

max Ground smoke/fire

Bombers or fighters

flying a prefab mission on autopilot or using a recorded flight as ChiliW use it. 

 

I do feel the best methode would be if using the:

prefab mission, it has to be with as less as possible activities and very basic, like a clear sky, empty snow terrain, single mission, airstat etc. All to try to keep the mission much the same as possible. 

 recorded flight, I think it would be a combination from high activities, lots of planes alternating with a lonely part. Clear sky alterneted with a sky clouded with rain. I would say full CPU and low GPU load alternated with low CPU load and full CPU load. 

 

What are you ideas about this. 

 

 

 

 

[SharpeXB]

The trouble is that live gameplay is more demanding than a recorded track. Or last time I checked it was and probably that hasn’t changed. Certainly running active FM and AI contributes to this. 

If you want to test your CPU just do a runway start on the ground with lots of other planes especially bombers. Like a whole flight of He111s

To test your GPU do a free flight. 

Make the missions simple like just fly straight or sit on the runway turning your head back and forth 20 times. 

The first time you look around the frames will drop a bit so start your benchmark after that point. 

Doing complex missions like dogfights, you can’t easily replicate them and it will vary what you’re testing. I usually want to know the worst case. 

[blitze]

Stalingrad City over flight with clouds and bombers, their escorts and 2 flights intercepting at 2000m ish.

 

Enjoy. ?

[dburne]

My best benchmark flight:

Any flight I get assigned in either my Career or in my PWCG campaign.

 

That is all I care about, as long as I get a good experience in those I am quite happy - and that is what I tune my system for.

Then I turn the fps counter off, and have more fun than is hardly imaginable!

[Dutch2]

After doing a couple of try outs, the whole problem is you have to compare and that can only if you using two equal missions. To get that I do use an single mission, snow, clear sky, no ground objects. I’m using the Camel on autopilot as testplane and I’m looking after the recordings on the details of the wing wires, wings edges and a quick look at the terrain. Recordings are done by FCAT VR software, which is actually working great now.  

 

Think SharpeXB has it right you need an mission and not a recorded flight. 

 

 

 

 

[JonRedcorn]

Wasn't chilwilis test done with a recording? Worked pretty well. Test would have to be either flying over a large city or some other hard hitting area, getting a reading while in free flight with no other objects or action going on gives you a very positive outlook on how your framerate will hold up under scrutiny. This is my main issue with VR right now, the performance is all over the place, just looking in certain directions can drop the frame rate in half. It's extremely weird, believe its the cause of most stuttering issues, even in 2d it does the same.

Edited March 15, 2019 by JonRedcorn

[SharpeXB]

Back during early access for BoS we were asked to do benchmark scenarios to help figure out the graphic presets. We posted descriptions of missions and hardware etc. 

I imagine if it had been possible for 1CGS to furnish a benchmark that they would have done that but I think the way this game engine works that’s not possible. 

The live play FM and AI take a good chunk of resources and those will be different very time. And recorded tracks don’t give you the same result as live. 

For myself I always want to know which hardware is limiting me. So I’ll purposly select what I know will test either and do some very simple runs like mentioned above. 

3 hours ago, JonRedcorn said:

Wasn't chilwilis test done with a recording?   

If it was done with a recorded track then the results won’t equal what live gameplay would be. I suppose it’s at least consistent in some way but not representative of what you get actually running the game. Particularly with regard to the CPU load (FM and AI) which is the bottleneck for VR. 

I haven’t run those tests in a while myself but I don’t imagine that much has changed. And I’m just speculating on why there’s a difference but it seems logical from what I know about IL-2

[dburne]

I still say the best benchmark for VR is just playing the game.

Get acceptable performance (an individual taste), then turn fps counter off and have fun.

[Dutch2]

1 hour ago, dburne said:

I still say the best benchmark for VR is just playing the game.

Get acceptable performance (an individual taste), then turn fps counter off and have fun.

 

 

how do you compare two adjustments, the FPS, is in my case, a mid spec system, does fluctuating a lot and you still need two of the same missions. I have fpsVR payware from Steam and while this is so fluctuating, without logging it is impossible to check the difference. 

BTW as watching Nvidia’s Tom Petersen he explains the that tools like FRAPS and I guess the same for IL2 FPS ingame display is not the right tool to measure the VR performance. 

 

 

[dburne]

22 minutes ago, Dutch2 said:

 

 

how do you compare two adjustments, the FPS, is in my case, a mid spec system, does fluctuating a lot and you still need two of the same missions. I have fpsVR payware from Steam and while this is so fluctuating, without logging it is impossible to check the difference. 

BTW as watching Nvidia’s Tom Petersen he explains the that tools like FRAPS and I guess the same for IL2 FPS ingame display is not the right tool to measure the VR performance. 

 

 

 

I just mainly go for smoothness, I only fly single player so if I get good smoothness in the Career Mode and in PWCG I am happy.

[Dutch2]

On 3/15/2019 at 7:06 PM, dburne said:

 

I just mainly go for smoothness, I only fly single player so if I get good smoothness in the Career Mode and in PWCG I am happy.

Think your system is far better then my i7-7700k/GTX1080 so onlike my system, your system can handle the high setting much better, it has much more performance room, while mine is constantly trying to avoid being drowned. So in my case, tweaking is much more important and necessary to get the optimal adjustments. 

 

I did have the same when running RoF on an AMD 6400+ dualcore/HD4890 that was much more sensitive then my i5-2500k/HD7970 

 

On the other hand I think tweaking is also the part of the fun I have. 

 

Edit: smoothness in VR is all about a straight 11 sec Frametime and that is what I measure using FCAT VR. The FPS ingame is not suited for that mainly because you can not log it to compare, but also sinds 2012 we know it is all about a flat line in logged frame times and not FPS.  

Edited March 17, 2019 by Dutch2

[Alonzo]

The point of any benchmark is to have reproducible results. To say "a recording doesn't perform the same as the game, so I'll just fly a bunch of different missions" will not produce consistent results. If you do a bunch of runs and change nothing, are you getting consistent results? How consistent? How often do you get significant deviation? That's the baseline you need before you start tuning stuff. Chili's benchmark track isn't actually intended to be representative of in-game gameplay, it's a benchmarking and tuning tool. Even with that benchmark track, where you take off the HMD and leave it on the desk and try very hard to reduce variations, I *still* get significant variation on some runs.

 

I'm going to argue that even with FCAT what you are looking for are the objectively measurable differences that come from changing settings, doing overclocks, that kind of thing. If you want to measure it, you need a consistent baseline. Otherwise all your figures are junk.

[TUS_Samuel]

On 3/12/2019 at 2:58 AM, Dutch2 said:

as less as possible activities

You can just look at FPS while track is paused. This readings are very consistent and don't require external measuring tools. I run this stall test before actual flying because FPS sometimes drops for no reason between game launches.

[Dutch2]

19 hours ago, Alonzo said:

The point of any benchmark is to have reproducible results. To say "a recording doesn't perform the same as the game, so I'll just fly a bunch of different missions" will not produce consistent results. If you do a bunch of runs and change nothing, are you getting consistent results? How consistent? How often do you get significant deviation? That's the baseline you need before you start tuning stuff. Chili's benchmark track isn't actually intended to be representative of in-game gameplay, it's a benchmarking and tuning tool. Even with that benchmark track, where you take off the HMD and leave it on the desk and try very hard to reduce variations, I *still* get significant variation on some runs.

 

I'm going to argue that even with FCAT what you are looking for are the objectively measurable differences that come from changing settings, doing overclocks, that kind of thing. If you want to measure it, you need a consistent baseline. Otherwise all your figures are junk.

If you have anything better then let me know, I’m still trying to figure that out.  But I did a couple of the same flights and did notice they were all the same. Only you have always fly in the same direction and use a very lonely mission, clear sky, disabled all groundactivities etc.  

About FCAT VR I think it is very clear that since this tool has been released, the most decent PC sites like, HWInfo, Tweakers, GAMERSNEXUS and Tomshardware are using it for testing, or did because not much activity on VR testing on these sites. 

 

BTW Here is what Gamernexus is writing down: If the tester is performing a manual benchmark, it is critical to the benchmark's reproduction that the same map, location on the map, and game events are experienced in each test pass. By introducing complex combat elements and dynamic events, the tester runs the risk of invalidating results.

 

[chiliwili69]

I think developer should put some attention to this. People is in general lost about what settings to use for their own particular machine.

 

There is poll here: (about 75% thinks it is a good idea!)

 

With a good common benchmark produced by the game we will be able to make multiple runs and calculate the hit of every single step in the graphics settings, software settings, hardware settings, etc, etc. Ideally we could build a matrix with every single element of settings (Game, software, hardware) in rows and in the columns the setting itself (low, med, high, etc).

 

People will be also able to run the test just to diagnose their PCs, just to check if they achieve same performance than their peers.

 

Probably we will need just a track, or free demo track (of 1 to 2 minutes), created by them (IL-2 dev team).  I understand they have a lot on the table right now, but this will be an important thing for the users to get the most of the next BoBP and the new VR headsets. Probably the IL-2 team is already using some sort of track internally to measure performance.

 

Then, having a track (either recorded flight or fixed mission) we can then analyze what tool we could use to measure fps, frametimes, etc (FRAPS, FCATVR, fpsvr, etc)

 

 

On 3/12/2019 at 3:30 AM, SharpeXB said:

The trouble is that live gameplay is more demanding than a recorded track. Or last time I checked it was and probably that hasn’t changed. Certainly running active FM and AI contributes to this.

 

I did´t test this myself. We don´t what is really disabled in a recorded flight (FM, AI, etc). We could simply ask this to Devs. I will do (and I will wait... seated).

Can anyone record with Fraps or FCAT a live game flight (about 2-3 min) and at the same time record it as flight track record?

Then replay the track with Fraps and compare results.

On 3/11/2019 at 8:58 PM, Dutch2 said:

prefab mission, it has to be with as less as possible activities and very basic, like a clear sky, empty snow terrain, single mission, airstat etc. All to try to keep the mission much the same as possible. 

 

Does anyone know if our mission builders can build a mission which always start everything at the same conditions (same plane, same location, same enemies, same actions taken by others,etc) so we press the Autopilot button and let the mission go for 2 minutes.

Ideally the mission should be complex enough to deliver fps around 60-70 fps in a good PC (let say  CPU OC to 4.8 GHz and 1080), so this way we are able to measure increments or decrements with every change in settings (game, software, hardware)

Edited March 22, 2019 by chiliwili69

[SharpeXB]

9 hours ago, chiliwili69 said:

 

I did´t test this myself. We don´t what is really disabled in a recorded flight (FM, AI, etc). We could simply ask this to Devs. I will do (and I will wait... seated).

Can anyone record with Fraps or FCAT a live game flight (about 2-3 min) and at the same time record it as flight track record?

Then replay the track with Fraps and compare results.

 

Back during EA we were asked to submit performance results on the forum here in order for 1CGS to develop the graphic presets. I imagine if it had been possible for them to create an in-game benchmark track then they would have done so back then. I wonder if that isn’t possible because of how the AI are. It’s not possible to have a canned mission that replays identically every time. Even your own autopilot AI is I think just a repeat of the non player AI and so will not just follow a script. 

[Alonzo]

16 hours ago, chiliwili69 said:

Probably we will need just a track, or free demo track (of 1 to 2 minutes), created by them (IL-2 dev team).  I understand they have a lot on the table right now, but this will be an important thing for the users to get the most of the next BoBP and the new VR headsets. Probably the IL-2 team is already using some sort of track internally to measure performance.

 

I really wish they were, but some of the steps backward for performance make me think, maybe not.

 

16 hours ago, chiliwili69 said:

Does anyone know if our mission builders can build a mission which always start everything at the same conditions (same plane, same location, same enemies, same actions taken by others,etc) so we press the Autopilot button and let the mission go for 2 minutes.

 

I've built a mission. See below. Unzip and drop into your IL-2 Sturmovik Battle of Stalingrad\data\Missions folder. This has the same initial conditions and where a lot of action happens, including ground and air AI, and an invulnerable player plane that can auto-pilot through it all. Unfortunately the actions taken by the AI do appear to be somewhat randomized, so the tracks are not 100% the same every time. The autopilot is also AI, so it takes different actions each time also.

 

That might not be the end of the world. It might be that if you did (say) 3 runs on autopilot you could get an average that was fairly consistent. I ran out of time this evening but if anyone wants to try it, you might get consistent timings from a 1 or 2 minute run. The map itself shuts down after 3 minutes automatically.

 

To benchmark, use Chili's usual benchmarking setup. Load the mission using Normal difficulty. Hit P to unpause and start the benchmark with Fraps, then immediately press A to engage the autopilot. Make sure your HUD and FPS counter are off, as in Chili's benchmark method.

 

I tried to make the scenario quite pretty. All planes are invulnerable and will shoot at each other for the full 3 minutes. Enjoy. ? 

 

 

Alonzo VR Benchmark 1.0.zip

[chiliwili69]

On 3/23/2019 at 3:51 AM, Alonzo said:

It might be that if you did (say) 3 runs on autopilot you could get an average that was fairly consistent.

 

Thanks Alonzo for trying this. Last weekend I downloaded your mission and gave it a try.

 

The first run on autopilot my plane crashed with another enemy plane and foll into the ground but since it is invulnerable it was keep running through the ground. It was not a valid run.

I tried more times with no crashing but I saw that the AI introduce a lot of variability in the flight (go up, go down, rolls, etc) and you will need to run several runs (10 or more) to get a constant average value. I didn´t measure that with Fraps, but just looking what I saw there is a lot of variability. 

 

Maybe a mission with enemy planes very far away (50Km) could produce a fixed pattern flight, but it would be just a straight line solo flight. Maybe with some clouds, sun, cities and mountains it could produce a benchmark with  constant 70-80 fps.

 

Otherwise the only options is that dev team create a fix demo flight experience for the game.

[Dutch2]

I see you  have the same problem as I have with that mission. For benchmarking you do need equally flights which in this mission is not. I did fly a couple on auto pilot and did notice this. 

An alternative could be the first mission in the fire and ice mission campaign, that is in the first period equal and it rather stresses your CPU a lot because of the AI it is generating.

But I do stay to my point, the whole benchmarking goal, is not about stressing your components. 

 

Edited April 5, 2019 by Dutch2

[whitav8]

Alonzo,

   I would like to use your benchmark - but I don't see the new mission after unzipping and placing the folder in 

"D:\SteamLibrary\steamapps\common\IL-2 Sturmovik Battle of Stalingrad\data\Missions\Alonzo VR Benchmark 1.0"

 

 I tried having the files in the separate folder as above  (Alonzo...Benchmark...) and also just placing them in the main mission folder but it just doesn't show it as a selectable mission. I only have BoS and not anything else. When I leave the files in the separate folder beneath ....Missions, I see your folder but when I click on it, nothing happens. The files that are the  default files have names like _gen.eng, _gen.Mission, _gen.msnbin - do I need to rename your files like "Alonzo VR Benchmark 1.0.Mission" to something else?

 

Thanks

 

Dave W.

[HunDread]

14 hours ago, whitav8 said:

Alonzo,

   I would like to use your benchmark - but I don't see the new mission after unzipping and placing the folder in 

"D:\SteamLibrary\steamapps\common\IL-2 Sturmovik Battle of Stalingrad\data\Missions\Alonzo VR Benchmark 1.0"

 

 I tried having the files in the separate folder as above  (Alonzo...Benchmark...) and also just placing them in the main mission folder but it just doesn't show it as a selectable mission. I only have BoS and not anything else. When I leave the files in the separate folder beneath ....Missions, I see your folder but when I click on it, nothing happens. The files that are the  default files have names like _gen.eng, _gen.Mission, _gen.msnbin - do I need to rename your files like "Alonzo VR Benchmark 1.0.Mission" to something else?

 

Thanks

 

Dave W.

 

This might help (copied from the IL-2_Sturmovik_Mission_Editor_and_Multiplayer_Server_Manual)

 

[whitav8]

@HunDread,

       Sorry, but that (removing the cache files - upon startup it did create new files )didn't work. I just got IL-2 BoS recently - is there a possibility that the benchmark files are somehow not the correct version for the latest version??

Thanks

 

Dave

[chiliwili69]

The mission works OK with current version, just checked now.

 

This is how the mission looks like:

 

the folder is this: