As part of Microsoft’s push into artificial intelligence, one research team is launching a World and Human Action Model called Muse that has been trained to generate snippets of gameplay and controller actions. Muse was created by the Microsoft Research team in Cambridge, UK, with the team collaborating with the nearby Xbox first-party studio Ninja Theory.
The model was trained using data from Ninja Theory’s 2020 multiplayer game Bleeding Edge, with Microsoft claiming the data was gathered ethically, with all player-identifying information anonymized before use.
“It’s been amazing to see the variety of ways Microsoft Research has used the Bleeding Edge environment and data to explore novel techniques in a rapidly moving AI industry,” says Gavin Costello, technical director at Ninja Theory.
Katja Hofmann, lead of the Microsoft Research Game Intelligence team, shows me gameplay clips generated by the earliest versions of the model. These short snippets rapidly disintegrated, falling apart into blurry mush. With additional training, the model improved, more accurately replicating character movement, including flying and destroying power cells on the map.
The latest version of Muse generates gameplay clips (which can be seen on Xbox Wire) at a low resolution of 300×180, running at 10 frames per second. Hofmann says that this is done in order to ensure training works even for the “smallest possible models.”
By far, the team’s biggest excitement around Muse comes from how it reacts to real-time elements, with Hofmann showing a demonstration of real-time controller actions and the generated gameplay responding appropriately. Hofmann also demonstrates seamlessly dragging a power cell into the gameplay, which is instantly picked up by the model.
What are Muse and generative AI being used for at Xbox?
Fatima Kardar, corporate vice president of Gaming AI at Microsoft, says this kind of research is being used to “empower” game developers, letting them choose if or how they want to use the technology. Microsoft Gaming CEO Phil Spencer shared that he’s excited about the possibility of using AI models to aid game preservation, letting a model study games that are trapped on depreciated hardware.
Dom Matthews, studio head at Ninja Theory, notes that the team won’t be using AI-generated content in its games but that the team is interested in any tools that allow faster ideation and iteration.
“I think the interesting aspect for us that is exciting is ‘How can we use technology like this to make the process of making games quicker and easier for our talented team’ so they can really focus on the thing that I think is really special about games which is that human creativity,” Matthews says. “Games to me are really creators using the medium of games to talk to their audience, to communicate through the medium of interactivity, and that is core to game creation for me.”
Reaction: Potentially interesting developments
For me, AI usage has a lot of potential, and any kind of tools that can make game developers work easier and faster should definitely be evaluated. When it comes to raw gameplay footage being generated by some model or game narratives being adjusted using AI, I’m less enthused.
I’m sure there are some interesting things that’ll come out of these advancements, such new ways for testing player navigation and pathfinding, but I do hope Microsoft leadership listens to creatives like Matthews and doesn’t lose sight of the human focus at the core of game development.
