There are plenty of artificial intelligence agents that have learned to beat video games, but now a team of scientists have taken it one step further by creating an AI that can actually make its own games just by watching someone play them.

Researchers at the Georgia Institute of Technology published a new paper called “Game Engine Learning from Video,” which describes an AI that can recreate the features of a game simply by watching several minutes of gameplay.

“Our AI creates the predictive model without ever accessing the game’s code, and makes significantly more accurate future event predictions than those of convolutional neural networks,” said Matthew Guzdial, lead researcher on the project and a Ph.D. student in computer science. “A single video won’t produce a perfect clone of the game engine, but by training the AI on just a few additional videos you get something that’s pretty close.”

The research team used Nintendo’s classic “Super Mario Bros.” to train the AI using videos of two different play styles: a “speedrunner” play style, where a player rushes through the game as fast as possible, and an “explorer” play style, where a player spends time exploring each level to find powerups or hidden areas. Based on this data, the AI was able to recreate several of the game’s core mechanics, including jumping, defeating enemies, avoiding pits and other environmental hazards and so on. The AI also learned more specific rules about how certain mechanics function, such as Mario being unable to jump again once he is already in the air.

“The technique relies on a relatively simple search algorithm that searches through possible sets of rules that can best predict a set of frame transitions,” explained Mark Riedl, associate professor of interactive computing and co-investigator on the project. “To our knowledge this represents the first AI technique to learn a game engine and simulate a game world with gameplay footage.”

The game created by the AI is still not perfect, but the project demonstrates how an AI can learn and replicate complicated concepts simply by observing them in action.

“Intelligent agents need to be able to make predictions about their environment if they are to deliver on the promise of advancing different technology applications,” said Guzdial. “Our model can be used for a variety of tasks in training or education scenarios, and we think it will scale to many types of games as we move forward.”

Photo: Super Mario Maker courtesy of Nintendo