Xbox Game Studios head Matt Booty has said it’s his “dream” to turn to “transformational” AI and machine learning as an alternative to human QA testing.
Booty dove into his desire for AI testers during his “Storytime With Matt Booty” panel during PAX West. He said QA testing has “not really kept up with how quickly we can make content,” as reported by VGC, and went on to compare changing something minor in a game as opposed to a movie.
“If we’re working on a movie and you come in and say ‘hey, this ending let’s tighten this up, let’s edit this, let’s cut that scene,’ it usually doesn’t break anything at the beginning of the movie,” he said. “But in a game you can be ready to ship and a designer’s like ‘I’ve got this one little feature, I’m just going to change the colour on this one thing’ and then it somehow blows up something and now the first 10 minutes of the game doesn’t play. So that testing aspect, every single time anything new goes into a big game the whole game has to be tested, front-to-back, side-to-side.”
Booty continued by saying it’s his “dream” to see AI testers become a reality, adding “there’s a lot going on with AI and machine learning right now, and people using AI to generate all these images. What I always say when I bump into the AI folks is ‘help me figure out how to use an AI bot to go test a game,’ because I would love to be able to start up 10,000 instances of a game in the cloud. So there’s 10,000 copies of the game running, deploy an AI bot to spend all night testing the game, then in the morning we get a report. Because that would be transformational.”
It seems unlikely Booty is advocating for doing away with human QA testers entirely, so much as theorising about about how a new technology could be used in the QA process. It’s not the best look at a time when the wider industry faces ongoing actions and controversies over QA testing practices: tester roles are a notoriously underpaid, underappreciated and overworked part of the development process (and sometimes don’t even get appropriate in-game credits). QA units at both Raven Software and Bioware have formed unions this year to tackle unfair pay and dismissals. To suggest that the dream future of QA should be all about machine learning feels somewhat tone-deaf in such a context.
