Do you think it is likely that we will start to see Large Language Models integrated in to major video games? If so, are there some examples within gaming already?
We won’t see large language models. We will likely see a stripped down version like a small language model (or Domain Specific models if you want the fancy marketing wank term) because a NPC in a fantasy game doesn’t need to know about 13th century Europe or 19th century Asia.
Do I think it’ll happen? Yes, even if it’s not good, because AAA companies are cheap and have no taste. They thrive on just spewing out more content than a smaller studio could make, quality be damned.
That said, whether or not it COULD be useful in the future I think depends on the context and how well you could tune the models.
I think it has absolutely no place in a narrative game where intentional authorship is what people come for. Even if it’s passable, I want to know that what I’m hearing or reading was something SOMEONE wanted to say.
But I think it could be interesting in more open ended, replayable sim games where you want to be able to try a wide variety of approaches and have different experiences each time, but it would be impractical for a dev to implement all those possibilities to the point where players would feel like the game adequately responds to their actions. However, I don’t think you could just drop a copy of chat gpt in there and call it a day. You want different NPCs to be different and you want some consistent reality that they all exist in and respond to. So you’d probably need to put in some constraints based on some hidden file describing a particular world gen’s state. A basic example would be the NPC knowing that the town you asked about is to their north or perhaps an existing relationship between 2 characters.
Idk how technically feasible this would be, but it’d be a cool tool in the right context if done right. I think the key here is it can be good when it enhances what you want to do and you put in the effort to make it work vs just using it as a lazy shortcut.
I think using LLMs to provide the dialog for NPCs in a RPG is a use case that’s just begging to happen. Ie townsfolk that don’t just give the same few replies every time, and who react to things you’ve done in the past beyond just whatever prewritten options the developer thought of.
That is…actually far better than I thought it would be. It’s clearly not ready yet, but I could see the potential.
The AI model is too happy to serve the whims of the player, but if there was a better model that could actually be hooked in to me hanics like personality scores or reputation, I could see that as an interesting gameplay system. It also needs more checks on what they are and aren’t supposed to know (e.g. why would a Skyrim NPC associate the name Batman with heroism, or why would they know who Gandalf is?).
A (digital) setup like Westworld is probably in the cards someday. Hopefully with more checks in place to keep the AI from rising up though!
A challenge game developers have talked about with integrating LLMs is keeping the dialogue matched to the game world, e.g. you don’t want a Skyrim NPC mentioning a cell phone.
If the feature actually worked as intended I could see myself ignoring the rest of the game and just chatting with the townsfolk.
In reality, I imagine the NPC would totally forget what we were talking about after a certain amount of messages pass. Limited context windows and all that jazz.
Not anytime soon. Nvidia tried, and nobody liked it. LLMs still suck at creative writing and need a ton of RAM/VRAM just to work. They also often get confused or trail off in any discussion/roleplay.
The only game that sort of made it work was Suck Up!, where you’re a vampire that has to convince an AI to let you in their house so you can suck their blood. It’s a fun concept but even that game gets repetitive quick and the LLM is very stupid and random.
Tiny models only get stupid like that because you’re taking a general purpose model that knows everything in the world and compressing all that knowledge too much. If you start with a model that only knows basic english and info about a few hundred things in the game, it can be much smaller.
I think so. New GPUs will be able to handle AI models running locally before too long. I think this will be used for NPC behavior as a replacement for procedural quest / dialogue generation. I have seen a lot of mobile games leveraging this but they don’t seem very good yet. Models need to be trained more specifically for each game I think
You can run language models on consumer cards right now. The only thing is depending on the size of the model and the amount of VRAM on your card, you might not be able to do much else.
Yeah there is likely still a ways to go before we can run high end modern games plus a local model, but newer nvidia cards are pretty crazy. It’s probably closer than I think
A good fit would be random background NPCs. For example, pedestrians in a GTA like game. Can potentially increase the variety in the things they can say, and maybe even talk about things the player has just done.
I think the PC vs. console divide is relevant here. I’m not sure how advanced text entry on consoles is these days, but I imagine PCs have the advantage with keyboards. Maybe if they use voice recognition on the consoles? But AAA games usually target both, and if interacting with the model is clunky for a big chunk of your market then the big developers might not use the technology.
Of course, indie devs that only target PC can go wild.
Whisper from OpenAI is pretty solid for speech recognition (at least English), and it is small enough to deploy on mobile devices. If I recall correctly, both PS and Xbox controllers have mics built-in, so input device is covered.
AI could also generate dialogie options for players, though. It could operate as traditional dialogue, with AI generating responses and possible doalogue paths ahead of time so you get a “normal” experience that just changes every time
Yeah, I’m generally not fond of using AI, but that clip where June goes fucking crazy convinced me that it might actually be viable. Not to that level of craziness, but having them go on like SC2 casters throughout the entire game would fit in so much.
There are a couple of mystery games that are available in early access/beta that already use LLM AI to generate dialogue.
I’ve messed with 1 and it’s alright. The problem with it is mostly in how it doesn’t necessarily regurgitate the info it knows into something coherent and logical to actually solve the mystery. That or they never ever tell you the truth. Since the only way to solve the mystery is by talking to NPCs to get enough info to piece the puzzle out, it often leads to unsolveable mysteries.
Now, if that problem with LLMs could be fixed, so it isn’t conflagrating multiple pieces of data into something new and novel, I think it would be awesome for making game worlds that are more alive and natural. Like instead of walking up to an NPC and clicking “rumors” you can actually just talk to them like a real person and they would respond in kind, as if the NPC was actually a person living in that world. It’s really the only thing I think would actually work well with generative AI chat bots, but the generative AI still has too many problems to make it truly viable.
It actually works better with trying to play D&D, because even IRL, players often misinterpret rules and argue. Which you genuinely have when trying to play with 3 AI characters.
Are you asking if company is willing to risk all their reputation to add software that generates random text based on user input, that can be exploited or just faked as hate, racists or whatever you could think of evil is ?
I don’t think so. Maybe it could be used to generate some large dialogue loops but putting random text generator to the actual game if it’s not your first game, you have nothing to lose or you want to feel the AI hype ? I don’t think so.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: [email protected]
Welcome to the largest gaming community on Lemmy! Discussion for all kinds of games. Video games, tabletop games, card games etc.
We won’t see large language models. We will likely see a stripped down version like a small language model (or Domain Specific models if you want the fancy marketing wank term) because a NPC in a fantasy game doesn’t need to know about 13th century Europe or 19th century Asia.
Do I think it’ll happen? Yes, even if it’s not good, because AAA companies are cheap and have no taste. They thrive on just spewing out more content than a smaller studio could make, quality be damned.
That said, whether or not it COULD be useful in the future I think depends on the context and how well you could tune the models.
I think it has absolutely no place in a narrative game where intentional authorship is what people come for. Even if it’s passable, I want to know that what I’m hearing or reading was something SOMEONE wanted to say.
But I think it could be interesting in more open ended, replayable sim games where you want to be able to try a wide variety of approaches and have different experiences each time, but it would be impractical for a dev to implement all those possibilities to the point where players would feel like the game adequately responds to their actions. However, I don’t think you could just drop a copy of chat gpt in there and call it a day. You want different NPCs to be different and you want some consistent reality that they all exist in and respond to. So you’d probably need to put in some constraints based on some hidden file describing a particular world gen’s state. A basic example would be the NPC knowing that the town you asked about is to their north or perhaps an existing relationship between 2 characters.
Idk how technically feasible this would be, but it’d be a cool tool in the right context if done right. I think the key here is it can be good when it enhances what you want to do and you put in the effort to make it work vs just using it as a lazy shortcut.
Thanks for sharing.
I did a bit of browsing online and it seems modders are progressing a lot in terms of technical feasibility in an open ended setting:
https://www.nexusmods.com/skyrimspecialedition/mods/98631
https://www.nexusmods.com/skyrimspecialedition/mods/126330
I think using LLMs to provide the dialog for NPCs in a RPG is a use case that’s just begging to happen. Ie townsfolk that don’t just give the same few replies every time, and who react to things you’ve done in the past beyond just whatever prewritten options the developer thought of.
This is my number one request for being AI to games, it would be so awesome for immersion like you say
For those that haven’t seen it, this produces hilarious results.
https://youtube.com/watch?v=vWMLVZF3pGc
Even with the annoying YouTuber trying to make silly content out of it I’d say it actually kept into the roleplay pretty well
Wow, even with its flaws, that went way smoother than I expected
Thanks for sharing, this set me off down the rabbit hole, and it seems this is now a popular and viable skyrim mod for organic dialogue with NPCs: https://www.nexusmods.com/skyrimspecialedition/mods/98631
Pretty mind blowing!!
Awesome find!
That is…actually far better than I thought it would be. It’s clearly not ready yet, but I could see the potential.
The AI model is too happy to serve the whims of the player, but if there was a better model that could actually be hooked in to me hanics like personality scores or reputation, I could see that as an interesting gameplay system. It also needs more checks on what they are and aren’t supposed to know (e.g. why would a Skyrim NPC associate the name Batman with heroism, or why would they know who Gandalf is?).
A (digital) setup like Westworld is probably in the cards someday. Hopefully with more checks in place to keep the AI from rising up though!
A challenge game developers have talked about with integrating LLMs is keeping the dialogue matched to the game world, e.g. you don’t want a Skyrim NPC mentioning a cell phone.
They did this in Free City, and one of the bank tellers became indistinguishable from a player.
What is Free City? I can’t figure it out from what I found in my searches.
https://en.wikipedia.org/wiki/Free_Guy
I feel like if an NPC doesn’t have something meaningful to say, then they may as well not exist. Otherwise they just serve to waste the players time.
“How dare there be an active world in my murder hobo wankfest?”
It is called “flavour”
If the feature actually worked as intended I could see myself ignoring the rest of the game and just chatting with the townsfolk.
In reality, I imagine the NPC would totally forget what we were talking about after a certain amount of messages pass. Limited context windows and all that jazz.
Not anytime soon. Nvidia tried, and nobody liked it. LLMs still suck at creative writing and need a ton of RAM/VRAM just to work. They also often get confused or trail off in any discussion/roleplay.
The only game that sort of made it work was Suck Up!, where you’re a vampire that has to convince an AI to let you in their house so you can suck their blood. It’s a fun concept but even that game gets repetitive quick and the LLM is very stupid and random.
NVIDIA not just tried, but still doing it, and apparently soon you’ll play with these NVIDIA ACE NPCs in PUBG and a few other games.
https://www.youtube.com/watch?v=wEKUSMqrbzQ
I don’t know what sounds more robotic, the AI or the script read for the player.
deleted by creator
They don’t need a ton of ram if you use a tiny LLM customized for the game’s use cases, and that’s what games would be doing.
The downside is the tinier the model the stupider it will be
Tiny models only get stupid like that because you’re taking a general purpose model that knows everything in the world and compressing all that knowledge too much. If you start with a model that only knows basic english and info about a few hundred things in the game, it can be much smaller.
I think so. New GPUs will be able to handle AI models running locally before too long. I think this will be used for NPC behavior as a replacement for procedural quest / dialogue generation. I have seen a lot of mobile games leveraging this but they don’t seem very good yet. Models need to be trained more specifically for each game I think
You can run language models on consumer cards right now. The only thing is depending on the size of the model and the amount of VRAM on your card, you might not be able to do much else.
Yeah there is likely still a ways to go before we can run high end modern games plus a local model, but newer nvidia cards are pretty crazy. It’s probably closer than I think
A good fit would be random background NPCs. For example, pedestrians in a GTA like game. Can potentially increase the variety in the things they can say, and maybe even talk about things the player has just done.
I think the PC vs. console divide is relevant here. I’m not sure how advanced text entry on consoles is these days, but I imagine PCs have the advantage with keyboards. Maybe if they use voice recognition on the consoles? But AAA games usually target both, and if interacting with the model is clunky for a big chunk of your market then the big developers might not use the technology.
Of course, indie devs that only target PC can go wild.
Whisper from OpenAI is pretty solid for speech recognition (at least English), and it is small enough to deploy on mobile devices. If I recall correctly, both PS and Xbox controllers have mics built-in, so input device is covered.
Not sure about Switch, but PS5/Xbox support mouse and keyboard inputs if a game is designed for them.
Voice recognition seems more likely to me, though. PS5 already has an advantage there because every Dualsense controller has a mic in it.
AI could also generate dialogie options for players, though. It could operate as traditional dialogue, with AI generating responses and possible doalogue paths ahead of time so you get a “normal” experience that just changes every time
The voice actors for The Finals are completely made with AI, there is no real actor behind the scenes they started from.
That’s not really what you had in mind, you’re probably imagining a live-response model, but it is something that’s happening now.
Yeah, I’m generally not fond of using AI, but that clip where June goes fucking crazy convinced me that it might actually be viable. Not to that level of craziness, but having them go on like SC2 casters throughout the entire game would fit in so much.
There are a couple of mystery games that are available in early access/beta that already use LLM AI to generate dialogue.
I’ve messed with 1 and it’s alright. The problem with it is mostly in how it doesn’t necessarily regurgitate the info it knows into something coherent and logical to actually solve the mystery. That or they never ever tell you the truth. Since the only way to solve the mystery is by talking to NPCs to get enough info to piece the puzzle out, it often leads to unsolveable mysteries.
Now, if that problem with LLMs could be fixed, so it isn’t conflagrating multiple pieces of data into something new and novel, I think it would be awesome for making game worlds that are more alive and natural. Like instead of walking up to an NPC and clicking “rumors” you can actually just talk to them like a real person and they would respond in kind, as if the NPC was actually a person living in that world. It’s really the only thing I think would actually work well with generative AI chat bots, but the generative AI still has too many problems to make it truly viable.
It actually works better with trying to play D&D, because even IRL, players often misinterpret rules and argue. Which you genuinely have when trying to play with 3 AI characters.
Interesting, thanks for sharing!
Are you asking if company is willing to risk all their reputation to add software that generates random text based on user input, that can be exploited or just faked as hate, racists or whatever you could think of evil is ?
I don’t think so. Maybe it could be used to generate some large dialogue loops but putting random text generator to the actual game if it’s not your first game, you have nothing to lose or you want to feel the AI hype ? I don’t think so.