Xbox Executive Dreams of AI Taking over QA Testing

As video games continue to grow, the number of potential bugs, glitches, and crashes grows with them. In recent years, we’ve seen numerous buggy games launch that resemble the saying ‘the bigger they are, the harder they fall.’ Speaking at PAX West 2022, where he’d already had a bit to say on the state of multi-studio AAA game development, Matt Booty, Xbox Game Studios’ head, has rendered his thoughts on the state of QA, and where he sees it going. It was his feeling, that the current QA testing model may struggle into the future as game sizes continue to expand, and he thinks that AI solutions may be a large part of addressing this challenge.

“Some of the processes we have, have not really kept up with how quickly we can make content, one of those is testing,” Booty stated.

“You think about a game, one of the biggest differences between a game and something like a movie, is if we’re working on a movie and you come in and say ‘hey, this ending, let’s tighten this up, let’s edit this, let’s cut that scene,’ it usually doesn’t break anything at the beginning of the movie. But in a game, you can be ready to ship, and a designer’s like, ‘I’ve got this one little feature, I’m just going to change the color on this one thing’ and then it somehow blows up something and now the first 10 minutes of the game doesn’t play.”

So that testing aspect, every single time anything new goes into a big game the whole game has to be tested, front-to-back, side-to-side. My dream – there’s a lot going on with AI and machine learning right now, and people using AI to generate all these images. What I always say when I bump into the AI folks, is: ‘Help me figure out how to use an AI bot to go test a game.’”

He continued. “Because I would love to be able to start up 10,000 instances of a game in the cloud, so there are 10,000 copies of the game running, deploy an AI bot to spend all night testing that game, then in the morning we get a report. Because that would be transformational. I always kind of laugh a little bit, people always say ‘the game shipped on Tuesday but I hear they were fixing bugs on Saturday night’ – there are months of testing and things that have to happen before a game goes out.”

A really interesting dimension to the comments Booty makes involves the current attempts by Xbox and greater body Microsoft, to acquire Activision/Blizzard/King, who has only recently become home to its first unionized QA team, the staff at Raven Studios. Members of that same team had a bit to say about the state of QA and how it impacted Call of Duty: Vanguard. The idea that the newly unionized team would then be replaced completely, or even somewhat by AI would be controversial considering the work that went into unionizing in the first place.

Source