Study Shows Game Developers Focusing on Wrong Subjects

Posted by Dan on June 9, 2009

I recently did an article at Stratics on a new study from Absolute Quality on bio-sensory feedback testing in games. I want to expand on it a bit here, and discuss some of the development and financial implications for the industry. First, the original article:

In a report to be released later this week, a company called Absolute
Quality (a subsidiary of gaming services company e4e) has completed a
bio-sensory study of how “engaged” gamers really are when they play.*
The study concluded that while game developers get the biggest reaction
out of cutscenes, exposition, and scripted ecnounters, their target
audience does not and prefers the simpler gameplay aspects of “running
and gunning”. Using a wireless headset from EmSense that scans subjects
brain waves (EEG), visual reactions, and the like from dozens of
sensors, the test involved subjects playing through a game, with
moments of emotive activity being recorded and tagged with the in-game
action that they correspond to. According to Absolute Quality, the
system measures “objective and continuous emotional and cognitive
responses.” 

So what does this mean for MMO Gamers? It means that we need to
start questioning what it is that makes MMORPGs fun. Obviously, that’s
a colossal topic, and that is precisely why Absolute Quality’s test
results are so exciting. “Brainwave and bio-sensory measurement
technologies give developers insight into the visceral engagement and
emotion of gamers, all with the precision of real-time measurement,”
said Tim Hong, SVP Product Development and Analytics, EmSense. 
“Traditional focus groups or surveys have their fair share of biases
and only try to get at the thrill and fun of a gamer experience, but
using this powerful technology clients have quantitative, benchmarked
measures for game engagement.”

According to Absolute Quality, in future studies, Absolute Quality
and EmSense will continue testing to explore gamer experiences in other
genres of the game industry. They intend to probe deeper into the issue
of developing games which are engaging for women, who now represent
more than 40 percent of gaming consumers. In the future, this could
mean that we’ll see games that are more tailored towards the mythical
“what women want”. But the potential runs deeper than that. For
example, free2play model gamers have different likes and dislikes than
subscription model gamers. Future tests can help reveal more detail
into what these differences are, and theoretically this will lead to
improved games. Right?

Maybe. The core result of the test showed that gamers prefer
gameplay to cutscenes; solid mechanics vs. plot exposition; and that
women prefer story and cutscenes more than men, who want more visceral
engagement.  Is that really all that surprising? Over the past years,
WoW has refined their gameplay model to focus much more on play
mechanics and solid design choices over forcing plot and cutscenes on
players (of course, it still has a rich lore available for the people
who want that). EVE has rock-solid gameplay and freeform environments
putting less importance on story and plot exposition, and more on the
tactical aspects of gameplay. In the shooter genre, we’ve found that
the most successful games included Counter-Strike with nearly no plot;
and old-schoolers will remember the buzz around the Serious Sam series,
which was hugely popular in bringing gameplay back to the simple roots.
The concept that this is surprising is a little bit of a stretch, but
it is nice to have some raw data to back up these numbers.

The study won’t be the end of the “story/plot vs.
gameplay/mechanics” fight in game development. Far from it. That debate
will probably rage until hell freezes over. But this, and future
studies, can provide a glimpse into the intangible answer to that
question. “The gaming industry has changed considerably in response to
changing demographics. Factors such as age, gender and socio-economic
conditions play a significant role in gamer experience”, said Shirish
Netke, President, Interactive Entertainment, e4e. “Better gamer-market
segmentation is crucial to delivering the right gaming experience for
the right audience.  It is also clear that game developers appear more
inclined to develop games based on their own personal preferences,
rather than truly understanding the needs of their target market.  By
more thoroughly exploring the target consumers’ gaming preferences,
game developers can significantly increase the success rates of new
titles in the market.” Greater success rates in new titles means more
choices for us as gamers, and that’s always a good thing.  A copy of
the report can be viewed here.

*Full disclosure: I was one of the subjects of this study at GDC 2009.

The implications of this technology on the industry are really quite fascinating for a number of reasons. Game development is expensive — AAA games can take $12-20 million to develop. MMO games can be even more expensive due to the huge frontloading of capital outlay required for the actual operation, and the long-tail marketing costs needed to keep a continually growing user base. More importantly, game development follows a production-possibilities frontier for the allocation of scarce development resources. I see some very fascinating implications for this technology in helping to make more efficient choices along that curve, especially in relation to man-hours spent on story and immersion, versus on mechanics and game design.

For example, let’s somewhat facetiously take a hypothetical IP: “My Little Pony Adventures.” The creative director, production heads, and the senior investors are deciding on what kind of game to make utilizing that IP. They determine that the target market is 8-year old children, and that they want to make the game a semi-MMO involving 100-1000 players on a server at a time. The teams come up with two, distinctive concepts.

Concept A is called “My Little Pony Adventures: Pony Panic”. Rather than focusing on story, this concept intends to feature many kinds of simple, refined game mechanics that will appeal to 8-year olds. In addition to puzzles, it will have many options for how to groom your pony, play with them, and take them to shows and ride them in a virtual field with other players.  The intent is to create an series of fun mini-games as well as a sandbox experience that will be accessible to the skill levels and interests of 8-year olds.

Concept B is called “My Little Pony Adventures: Princess Universe”. The concept will focus many hours into developing an immersive story and universe that captures the spirit of the My Little Pony IP. Players will take the role of a princess, and follow her through a series of adventures around the universe. Through the use of cut-scenes, merchandise tie-ins, and extensive world building and scripted events, players will take part in a relatively static (i.e. not sandboxy) experience that will hopefully be immersive for the interests of 8-year olds.

Both games potentially could be big-money winners for the target audience, but come from wildly different approaches. Which will be more successful? It will be important for the company to find out early in the development cycle, so that resources can be efficiently allocated, and time, energy and money are not being wasted.  That’s what makes the EmSense headset and AbsoluteQuality’s analysis so fascinating to me. It bypasses the focus group process, and significantly slims down the percent chance of error from inaccurate statements. And, it allows for extrapolation of information from other datasets. The My Little Pony team can take studies from other games like Lego Universe, and Puzzle Pirates, and the like, and see what the 8-year olds are thinking while they play. They can identify directly what mechanics cause feelings of joy, and what don’t; what stimulate brain activity, and what they gloss over. The end result, theoretically is that they can make better informed choices early in the development cycle, and continue these choices throughout the cycle, allowing for an agile solution based on quality feedback.

Assuming it works, this is good for the industry. Better decisions make better games; better games mean more money for the developers, which means better quality of life for employees, more development, greater risk taking and a broader selection. Now, it is likely to be several years before the technology permeates the market and there are enough datasets to make these kind of choices. But the fact that Absolute Quality is thinking outside the box, and working to improve the industry as a whole is just absolutely fascinating to me. It’s such as simple choice, yet it’s so innovative. “Lets find out what triggers various emotions in gaming, and then we can exploit it as design choices.” It seems so common sense, so why isn’t anyone else doing it?  I think, as an industry, we can be somewhat jaded about developments in technology. Oh look, a new program that lets you render graphics 5% faster. Ho hum. A new motion-capture software lets you do skeletal modelling on the fly. Yawn.  It’s so…..boring. I may be overly excited about this technology, but the optimism comes from the innovative nature of what the technology does.  It’s not “let’s do this faster, or make this easier for our designers.” It’s “how can we make games better?” And in the end, isn’t that the goal?

Popularity: 3% [?]