Potential for LLMs in-game?

March 24, 2024

I'm wondering if there would be any benefit on having a local LLM or generative AI model that can perform certain tasks in-game instead of hard-coding some logic.

I know some of the challenges can be: inference computation power or inaccurate outputs due to hallucinations.

This is just me brainstorming some potential uses of an LLM in a game like IL2 in the future:

- Generate Pilot Biographies

- Generate briefings

- Act like the traffic control for airfields and radars, leverage text-to-speech to generate radio commands.

- similarly, generate wingman radio chatter based on context.

- generate dynamic missions and campaigns (maybe not an LLM task).

- generate pilot pictures.

I know it's a bit of a naive take, but just thinking in how new tech can improve simming experiences in the future.

March 25, 2024

I think all of these would be feasible, even with current technology. There's a couple of small and larger problems though.

- I doubt current tech is fast enough to generate ATC text as well as its spoken version while simultaneously doing other game calculations (e.g. graphics/physics), without causing unacceptable comms delays.

- Briefings and radio comms happened according to rather strict procedures and guidelines. This lends itself wonderfully to procedural programs rather than Neural Nets. I doubt a Neural Net would be of great added value.

- Missions need to strictly adhere to a certain format or the game won't be able to read them. This is something Neural Nets are not good at. I think a hybrid approach would be best, where a CNN decides the mission targets, enemy aircraft etc. while the actual implementation is left to a procedural method.

March 25, 2024

1 hour ago, AEthelraedUnraed said:

- Briefings and radio comms happened according to rather strict procedures and guidelines. This lends itself wonderfully to procedural programs rather than Neural Nets. I doubt a Neural Net would be of great added value.

I think that you overstate how strict the procedures and guidelines are. Putting in the unprofessional bits, variations, etc would make it much more real.

And if you look at the records of the Pacific war, there were constant issues with the spotter planes misidentifying the enemy.

But I think that it's too ambitious to try to fully tackle this right now.

March 25, 2024

1 hour ago, Aapje said:

I think that you overstate how strict the procedures and guidelines are. Putting in the unprofessional bits, variations, etc would make it much more real.

It differs a bit with time, place and airforce, but overall there was a relatively strict adherance to the radio code. E.g. "Angel Blue, this is Angel FAC. You're cleared to attack target. Over." "Angel FAC, this is Angel Blue One. Roger, cleared to attack target. Out."

The bold text should basically be without variation. That leaves the small part in the middle, but even there the possible variation is relatively little. All primary sources I have read seem to indicate a high level of professionalism in radio communications. Though again, it differs a bit on the exact period/country/circumstances.

It should be said however that even with "fixed" messages, there will of course be differences in intonation etc. and a Neural Net should work great there. But that's specifically the spoken part rather than the text. I've seen some pretty fast text-to-speech networks, so I think that if people can live with perhaps a 2 second delay on radio messages and the occasional glitch, it might be feasible to generate the audio in "real time" while also running the other game logic.

Just for the lulz of it, I let ChatGPT generate a mission briefing text for the following mission (all data would be known through the mission generator):

Spoiler

Squadron: No. 183 Sq. RAF.
Airfield: B.53 Merville.
Date: October 1, 1944.
Time: 16:30.
Aircraft: 4x Typhoon Mk.Ib.
Loadout: 8x RP-3 rockets, 4x 20mm Hispano Mk.II cannon
Crew: S/L Mulliner, Cpt. Lens, Lt. Bodley, F/O Lee-Warner.
Target: Attack gun positions and house north of Antwerp.

Briefing Officer: Good afternoon, gentlemen. Please gather round for the mission briefing.

Squadron Leader Mulliner: Steps forward, attentive

Captain Lens: Nods, standing at ease

Lieutenant Bodley: Listens intently, ready for orders

Flight Officer Lee-Warner: Stands at attention, awaiting instructions

Briefing Officer: Our mission this afternoon is another critical strike against the enemy's positions north of Antwerp. Squadron No. 183, you will be spearheading this operation.

Squadron Leader Mulliner: Understood, sir. What are our objectives?

Briefing Officer: Your task is twofold. First, you will target the remaining gun positions identified earlier today. Secondly, there's been intelligence indicating that a house in the vicinity is being utilized as a command post. That house is to be neutralized.

Captain Lens: Raises a hand Sir, any specifics on enemy resistance we can expect?

Briefing Officer: Intelligence reports suggest moderate to heavy anti-aircraft defenses in the area. We've observed increased activity since our earlier sortie.

Lieutenant Bodley: How do we approach the house target, sir?

Briefing Officer: Bodley, you and Lee-Warner will lead the attack on the house. Mulliner and Lens, you'll provide cover and engage any threats in the vicinity.

Flight Officer Lee-Warner: Nods Understood, sir.

Briefing Officer: Loadout will consist of the same configuration as earlier: eight RP-3 rockets and four 20mm Hispano Mk.II cannons per aircraft.

Squadron Leader Mulliner: Any specific rendezvous points or communication protocols, sir?

Briefing Officer: Maintain radio silence until reaching the target area. Once in position, use coded signals for coordination. We expect heavy radio traffic interference from the enemy.

Captain Lens: And what about weather conditions, sir?

Briefing Officer: Weather appears to be clear with minimal cloud cover. Visibility should be optimal for your operations.

Briefing Officer: Gentlemen, this mission is crucial. Success here will significantly disrupt the enemy's operations in the region. Your skill and courage will be tested, but I have full confidence in each of you.

Squadron Leader Mulliner: We won't disappoint, sir.

Briefing Officer: That's what I like to hear. Prepare your aircraft, check your gear, and be ready to depart by 16:30 hours. Dismissed.

Captain Lens: Salutes Thank you, sir.

Lieutenant Bodley: Salutes Yes, sir.

Flight Officer Lee-Warner: Salutes Understood, sir.

Squadron Leader Mulliner: Salutes Thank you, sir.

The pilots disperse, heading to their respective aircraft, ready to execute the mission with precision and determination.

This is actually pretty fun ?

Although of course there's no way to make sure all the "other things" ChatGPT came up with (e.g. "radio silence," "coded signals" and "radio traffic interference") are present in the mission.

March 25, 2024

@AEthelraedUnraed

As far as I know, the US air force made a big step in professionalization after WW 2. I wouldn't expect pilots of all countries back then to be all that professional in their communications.

I found a transcript of some intercepted Luftwaffe comms:

https://ww2aircraft.net/forum/threads/luftwaffe-radio-communications-dialogue.57749/post-1707620

It fits what I would think they would say, like:

"Wolfgang to Hans: Where in the heck are we?
Hans to Wolfgang: I dunno where we are either.
Wolfgang to Hans: Where are you then?
Hans to Wolfgang: Here!"

And:

"Fritz to Hugo: Spitfires due west of us
Hugo to Fritz: Be quiet
Fritz to Hugo: There by the clouds, there by the clouds
Hugo to aircraft: Look out, all Hugo aircraft, look out on our left.
Hugo to aircraft: Look out, attack coming from above.
Hugo to aircraft: Are you there? Are you there?
Fritz to aircraft: Paul! (top of his voice)
Hugo to aircraft: Hello, Hugo aircraft: one bandit orbiting right.
Aircraft to Hugo: Am breaking off, am breaking off!"

I also remember hearing in a documentary somewhere that unnecessary chatter was a major issue for the USAF in the pacific and that it caused a lot of confusion, because important messages could not even be transmitted because the radio kept being blocked by 'livestream commentary'.

March 25, 2024

5 hours ago, Aapje said:

I found a transcript of some intercepted Luftwaffe comms:

https://ww2aircraft.net/forum/threads/luftwaffe-radio-communications-dialogue.57749/post-1707620

It fits what I would think they would say, like:

"Wolfgang to Hans: Where in the heck are we?
Hans to Wolfgang: I dunno where we are either.
Wolfgang to Hans: Where are you then?
Hans to Wolfgang: Here!"

And:

"Fritz to Hugo: Spitfires due west of us
Hugo to Fritz: Be quiet
Fritz to Hugo: There by the clouds, there by the clouds
Hugo to aircraft: Look out, all Hugo aircraft, look out on our left.
Hugo to aircraft: Look out, attack coming from above.
Hugo to aircraft: Are you there? Are you there?
Fritz to aircraft: Paul! (top of his voice)
Hugo to aircraft: Hello, Hugo aircraft: one bandit orbiting right.
Aircraft to Hugo: Am breaking off, am breaking off!"

I also remember hearing in a documentary somewhere that unnecessary chatter was a major issue for the USAF in the pacific and that it caused a lot of confusion, because important messages could not even be transmitted because the radio kept being blocked by 'livestream commentary'.

My understanding was always that there wasn't much of official rules at first, until major communication issues appeared during the first big air battles (e.g. BoB) and at least the RAF started to introduce a heavily regulated radio code. Not sure what the USAAF radio procedures were.

Here's a transcript of the communications between Leutnant Autenrieth of II./NJG 1 and his ground controller Unteroffizier Adam on the night of 11-12 august 1942 ("Nachtjagd Combat Archive," T. Boiten):

Spoiler

Marie 80. Richard auf 120.
Viktor.
Marie.
Viktor.
Neue Rollbahn 120.
Viktor.
Marie 40.
Viktor.
Kappelle 32. Marie 50.
Viktor.
Einmal Rolf.
Angetreten 120.
Viktor einmal Rolf.
Viktor.
Marie 45.
Rolf beendet.

This is extremely scripted language, almost entirely in codewords. Granted, it's specialised usage, but on the other hand there's probably no situation in WW2 where strict adherence to radio code would matter as little as when having an entire frequency to yourself and no enemy fighters within a hundred miles.

The following is purportedly a transcript of the communication between Johnnie Johnson and two ground controllers over Normandy in 1944 (https://forum.dcs.world/topic/143195-communications-with-airfields-during-wwii/):

Spoiler

'Greycap to Kenway. Airborne with twelve Spits. What's the form?'

'Kenway to Greycap. Bandits active five miles south of Caen at low level. Please investigate.'

'Greycap to Kenway. Roger. Any definite height on bandits?'

'Kenway to Greycap. No, but they're below 5,000 feet. Out.'

[...]

'Greycap from Grass-seed. Twenty plus bandits climbing up inland. Steer 140.'

'Greycap. Bandits have crossed below you at 15,000 feet. Port on to 310. Buster.'

'Greycap. Bandits now seven miles ahead. 5,000 feet below. Gate.'

'Greycap. Another strong formation of bandits behind you. About eight miles. Exercise caution.'

It isn't a complete transcript (note that no responses by Johnson are recorded for the second part), but it's still relatively strict adherence to radio brevity code. Besides the inquiry into a more precise height as well as perhaps the last sentence, they're all short, concise messages with standardised phrasing and some codewords.

5 hours ago, Aapje said:

I wouldn't expect pilots of all countries back then to be all that professional in their communications.

Well as I said, AFAIK there were (major) differences in (adherence to) communication procedures between various countries, periods and situations. I don't doubt that there were times at which more "informal" communications were the norm, or that in case of emergency one would just shout "Greycap, break!" rather than "Kenway to Greycap, bandit closing on your six, break, out."

But I do think that a rather scripted, procedural AI does a good enough job of creating plausible communications that a Neural Net isn't really needed.

Edited March 25, 2024 by AEthelraedUnraed

March 25, 2024

Personally I really don't like the addition of AI slop to games.

There is also potential copyright problems since the datasets the AI uses often includes material from not so kosher sources.

March 25, 2024

25 minutes ago, [N.O.G.F]Leon_Portier said:

the datasets the AI uses often includes material from not so kosher sources.

So do half the forum users

March 27, 2024

On 3/25/2024 at 10:31 AM, Aapje said:

because important messages could not even be transmitted because the radio kept being blocked by 'livestream commentary'.

Sounds like a typical squad night on discord.

March 28, 2024

The AI of the IL-2, for now, has nothing to do with aerial combat simulation in a game (Simulator).

And if it were possible to use AI, which today what we know has a lot to do with text processing (BIGDATA) from Microsoft, the PCs we would need would be crazy.

Let's go back to the summer of 1999. It was the summer of MATRIX, that film that described a dystopian future, where machines had taken over humanity through artificial intelligence.

The fact is that 23 years later, at the end of 2022, ChatGPT was presented to society, that interface that allows you to interact with a “robot” that answers everything you ask or request. In fact, it was the first real and direct experience that many users have had with artificial intelligence. It is true that their answers can still be improved and that human beings can today offer a more reasoned and contextualized reply. However, artificial intelligence has a very similar analysis capacity in its favor and is, without a doubt, much faster. And this would be just a preview of what awaits us in the coming years.

AI is revolutionizing the airline industry by improving operational efficiency, increasing safety, reducing costs and delivering improved experiences for both airlines and passengers.

From the information and news that appear sporadically, we can deduce that its progress is taking leaps and bounds. Furthermore, the most uncertain thing for human beings is that we are talking about a silent giant, since we hardly have certainty about the real state of technology or what fields it already covers that affect us daily.

The reality is that artificial intelligence is expanding to all industries, including aeronautics, still raising some questions and concerns generally linked to lack of knowledge about its scope and control. Let's try to analyze if it is a risk or if, as is desirable, it comes to help us and be our collaborator.

First, to understand how it would revolutionize the industrial world, let us keep in mind that artificial intelligence analyzes data precisely to offer us a concrete answer. From here, there are endless possibilities on how this technology would make it easier to perform each specific task. From helping pilots during the flight, offering a faster response on industrial processes, preventing failures in certain processes or reducing the factor of human error.

On a more strictly labor-related level, artificial intelligence will undoubtedly cause job destruction. But just as happened during the industrial revolution that originated in the mid-18th century, there is a certainty that the employment it will generate will be greater. Of course, a large part of these new jobs will require improving the qualifications of employees, which is always desirable from a socio-labor point of view.

In the specific area of the aeronautical sector, artificial intelligence is playing an increasingly important role, transforming the way aircraft are designed, manufactured, maintained and operated. Let's look at some key areas where AI is impacting this industry

Design and simulation: AI can help in creating more efficient and streamlined design models. Powered by machine learning algorithms, AI can analyze data and simulate different conditions to find optimal design solutions.

Innovation in various fields: AI is used in the research and development of new materials, technologies and systems that drive innovation in the aerospace industry.

Predictive maintenance: AI is used to analyze real-time sensor data from aircraft and predict when technical problems are likely to occur. This enables proactive maintenance and reduces unplanned downtime.

Real-time anomaly detection: AI can identify anomalous behavior in aircraft systems in real time, helping to prevent accidents and improve safety.
Aircraft inspection and maintenance: Drones equipped with computer vision systems and AI algorithms can autonomously inspect aircraft structures for damage or wear.

Pilot Training: AI is used in advanced flight simulators to train pilots in a variety of situations and scenarios, including emergencies. This improves pilot safety and preparation.

Human-machine interaction: Assistance and automation systems in the cockpit benefit from AI to provide and analyze information in real time, make decisions and assist pilots in critical situations.

Route and operations optimization: Airlines use AI to optimize flight routes, air traffic management and flight scheduling. This helps reduce fuel consumption, minimize delays and improve overall efficiency.

Air traffic control: Air traffic management is also being optimized through AI to coordinate and direct efficiently.

nte flights, especially in areas with congested traffic.

Personalizing the customer experience: Airlines can use AI to analyze passenger preferences and behavior and offer personalized services, from seat selection to in-flight entertainment options.

Collectively, AI is revolutionizing the airline industry by improving operational efficiency, increasing safety, reducing costs and delivering improved experiences for both airlines and passengers.

We must also keep in mind that there are some applications of AI in aeronautics that require social acceptance by users, whether they are industry workers, pilots or passengers, among others. That would be the turning point of this revolution. As knowledge about technology increases, its benefits become more evident, and the number of detractors and fearers decreases, its acceptance will be a fact.

We are accustomed to the fact that the head and the heart usually intervene in human decision-making. It is true that in the case of the use of artificial intelligence it would be as if we were only using the head... although with an infinitely greater capacity and with incomparable speed. But let's not lose sight of the fact that human beings must always be there, ultimately, to apply a little heart when decision-making requires it. Or, at least, it would be desirable.

Edited March 28, 2024 by 1/JSpan_Guerrero

March 28, 2024

Indra crea un simulador de realidad virtual para formar pilotos