In the future, Microsoft suggests, you may be playing AI. No, not on the battlefield, but on games that actually use AI to simulate the entire game itself.
As a first step, Microsoft has developed an AI model, called WHAM, that “beta tests” games early in the development cycle using AI instead of human players.
Gamers know that realistic AI can turn a good game into something great, like how the older F.E.A.R. games would realistically model how soldiers might react to a hostile, armed player. Microsoft’s World and Human Action Model (WHAM) takes the opposite approach — it tries to figure out how human players will react in a given situation, right down to a specific frame or setup within the existing game world. Microsoft calls this WHAM by the name “Muse.”
The point of Muse’s WHAM, Microsoft said, wasn’t to improve the way NPCs or in-game monsters necessarily reacted to players. Instead, WHAM was developed to make a game “feel right” — not too hard, not too easy, with interactions that felt realistic. That’s something that normally takes hours upon hours of beta testing and evaluating how gamers interact with the environment. WHAM was designed to help automate that, the company said.
Simulating video games with Muse’s WHAM
Microsoft said Wednesday that it has released the WHAM model to huggingface.com, alongside a “WHAM Demonstrator” to essentially place the AI player in a specific spawn location, and then test and evaluate what would happen if the AI made different decisions. Microsoft also published a paper describing WHAM to the Nature scientific journal, which was made available to PCWorld before publication.
To develop the model, Microsoft used about 500,000 anonymized gaming sessions (over all seven of the game’s maps) from Ninja Theory’s Bleeding Edge, a 4×4 multiplayer combat game that Ninja Theory released in 2020 but halted development on less than a year later. Each frame of the session was reduced to 300×180 resolution, then encoded into 540 AI tokens. Likewise, each motion on the Xbox controller, including the buttons, was reduced to 16 different inputs based on the stick direction and button input.
Microsoft said that the GIF below was generated by the Muse WHAM.
Microsoft
Microsoft encoded all of this gameplay into a 1.6-billion parameter model, condensing essentially seven entire years of gameplay into a single transformer. The company also developed smaller models based upon a single map, Skygarden, with 128×128 images used instead, with parameters ranging from 15 million to 894 million. (In AI, a larger number of parameters usually generates more realistic outcomes, at the cost of additional computing resources.)
Microsoft then built a concept prototype, known as the “WHAM Demonstrator” — sort of the AI chatbot based upon the WHAM model. In this case, the user was able to “place” the AI player upon a map, in relation to various objects around it. When enabled, the WHAM Demonstrator then sketched out how the “human” player was likely to respond. In this case, the developer could run and then re-run the Demonstrator to see various outcomes, then select an outcome to continue to see how the AI “human” would respond.
Microsoft
From its training, Demonstrator understood the gameplay rules and physics, though it took more training iterations to understand that some players could achieve flight, depending upon game conditions.
The idea is that the WHAM Demonstrator could be used to run different scenarios from the same starting point. In the Nature paper, Microsoft showed how WHAM, beginning with the same eight frames, could produce 16 widely divergent endpoints, based on the AI decisions that WHAM made. Even more interestingly, WHAM was developed so that users could add additional enemies or objects, and the AI would react accordingly.
Microsoft
Forget fake frames: Is the future of gaming entirely AI?
Draw a line through WHAM/Muse into the future, and you arrive at a “game” which is generated more and more in real time using AI. According to Microsoft’s vice president of gaming AI, Fatima Kardar, that’s where Microsoft hopes to go — apparently following Google, which has already demonstrated consistent game worlds from a prompt.
“Today, countless classic games tied to aging hardware are no longer playable by most people,” Kardar said in a statement. “Thanks to this breakthrough, we are exploring the potential for Muse to take older back catalog games from our studios and optimize them for any device. We believe this could radically change how we preserve and experience classic games in the future and make them accessible to more players. To imagine that beloved games lost to time and hardware advancement could one day be played on any screen with Xbox is an exciting possibility for us.”
Microsoft is also exploring the idea of “modding” games using AI, and making those early experiences available to players via Copilot Labs.
Microsoft said, however, that it does not necessarily plan on using AI as part of game development. That will be up to the company’s creative leaders, Kardar said, and any AI work will be shared “earlier on” with players and creators.
