User Experience (UX) research is a critical aspect of video game design that can significantly…
What’s Wrong With the RITE Method?
A critique of a common method used in video game usability research
Many video game usability practitioners employ a method to test usability within video games, called the ‘RITE’ method, short for Rapid Iterative Testing and Evaluation (RITE). Pioneered at Microsoft Games Studios and Microsoft Research, the RITE method has been adopted by many usability research organizations besides the teams at Microsoft.
While the RITE method has some advantages, such as the ‘rapid iterative’ ability to suggest changes to designers and test them in successive passes, it may fall short when looking for usability issues that lie beneath the surface.
The RITE method has its benefits: for instance, the ‘rapid iterative’ research design allows usability researchers to detect problems and inform the game design team. This is a nice fit with highly communicative ‘agile’ development processes: allowing usability errors to be corrected and retested in successive iterations. As cyclical (iterative) rounds of testing go on, the number of usability problems originally detected typically goes down with each iteration.
However, one should consider the quality of the usability problems being found by the RITE method. Does a rapid and iterative approach dive deep enough into the user experience to find problems that are not otherwise being found by existing quality assurance teams?
Those unfamiliar with usability research can often confuse it with quality assurance testing, which is not the case. In a recent conversation with Jakob Nielsen, he described an analogy that explains usability and user-experience research for designers:
“usability research is to design, as quality assurance is to testing”
When hiring a usability research team, game companies should expect more than basic usability testing. While basic usability testing is very important, these findings are aimed at problems that lie closer to the surface, such as issues dealing with the user interface. While these issues are important to find, they probably can be found through traditional quality assurance or game testing teams.
While Situated Research is interested in finding basic usability issues, we are equally interested in finding issues that lie deep beneath the surface. We feel these deeply-rooted issues may not be suitably addressed by the RITE method, for its ‘rapid iterative’ approach fails to conduct an open-ended, in-depth analysis. Uncovering patterns in player motivation, which ultimately form much of players’ opinions of a game, can provide high-value findings for game companies aiming to improve a game and its future sales. A mixed quantitative-qualitative approach can identify irregular problems that may have a more serious impact on gameplay, in addition to frequently occurring problems that might be identified by the RITE method.
Patterns in player motivation require a research method that dives deeper than traditional usability testing, and should incorporate theory blending research from fields such as human communication, behavioral psychology, and human-computer interaction. Situated Research maximizes a game company’s ROI by examining both the individual and social plane to find deeply-rooted issues that point to the core of the player’s experience. Going beyond traditional usability heuristics, which test the usability of specific portions of an interface, is necessary to find those deeply-rooted design issues that can go undetected by traditional quality assurance teams.
Written by: Matthew Sharritt, Ph.D.
Posted by: Situated Research
Related Posts
Hi, I’m one of the collaborators on the RITE method paper. Standard quality assurance testing cannot and does not go into the reasons the users struggle with interface, being aware of the game situation in order to make decisions, or have a holistic understanding of the relationship of the player and the game. The reason the RITE method was developed and continues to be successful is that it brings a decision making team around a common experience. Adapting the product actively between test subjects in order to get the best experience and solve problems has an immediate and positive effect on the user experience. These are things that are not and should not be altered based on the feedback of professional testers through QA testing. I think to suppose that the RITE method does not both encourage and support in-depth usability research and deep analysis of game design issues might reflect defects in how the testing is performed and not the methodology. RITE works best when it is a collaborative exercise between the usability researchers and the game designer.
As a game designer with over 20 million shipped titles I can say that the RITE method does work, is relevant to design, and does actually identify deep as well as surface problems that impact the user experience.
-Mark Terrano (Design Director & Founder, Hidden Path Entertainment)
Hi Mark, thank you for your comment. AOE II is one of my favorite games, and motivated me to incorporate RTS games into my dissertation research. I believe the RITE method is a useful tool, and my critique of the method was meant to show how it could be complimented with other forms of usability research to inform game design. Is it usability research, or an extension of QA testing? Our clients often confuse the two, which was part of the motivation for my response. Often, QA is brought in at the end of the product development life cycle to iron out gameplay problems or to do some final balancing in the game, where usability research can be incorporated from early stages of design to compare major alternative game designs (Sony, Disney Interactive and EA have hired us to do just that).
The rapid-iterative nature of RITE can result in chasing a rabbit down a hole: in effect, refining a single design without comparing or merging the benefits of alternative designs (see Jakob Nielsen’s discussion of parallel & iterative design: http://www.useit.com/alertbox/design-diversity-process.html). We have seen usability researchers tout the benefits of methods such as RITE without understanding the multi-disciplinary background that went into designing the method. This can reduce the effective application of the method (and interpretation of the results).
I think the best professional game developers I’ve worked with (I visited over 100 game development studios when I was in the Xbox Advanced Technology group) are already actively using parallel and iterative techniques to make their games – with multiple versions of features and mechanics existing in the same product being switch-configurable.
Zynga is a company that lives and breathes parallel & iterative design backed up by one of the most advanced metrics systems in the world. I’m glad to see these techniques being embraced more broadly in the industry.
RITE as we have practiced it didn’t result in tightly iterating on a single feature (rabbit hole) but knocked off the worst sticking points and most confusing spots in the interface – allowing players to immerse more fully in the content and have the best possible experience. As described in the paper – addressing the most significant issues that the multi-disciplinary team was in unanimous agreement on had the best results.
Quality assurance testing as practiced in the game industry is often done by 3rd party teams – or internally at large studios like EA – by teams that are not associated with the product. They do white and black box testing (against a script), functionality and playtesting. While the individual testers often make suggestions based on their play of other games, Quality Assurance testing in the video game industry typically does not and is not expected to do design analysis and comparison. RITE where I have seen it used has always been a tool for usability engineering to work collaboratively with design to make better products.
As the RITE Method paper clearly outlines that much of the benefit comes from the cross-discipline collaboration of decision makers, I’ll still contend that ” usability researchers tout the benefits of methods such as RITE without understanding the multi-disciplinary background” is a flawed application of the method, not a flaw in the RITE Method itself as someone might imagine from the title of your article “What’s Wrong with the RITE method”.
Before we came up with RITE – the methods used to test video games were borrowed from the commercial software testing usability methods – what seemed to me to be a rigid focus on documentation of the number of failures with a statistically relevant number of participants, rather than eliminating the failure spots when possible; was not the best use of the usability testing resources.
After using the techniques described in the paper, the player-relevant data and progress toward a finished title was immediate, obvious to everyone involved, and startling by contrast to other methods that were in use.
I am not a usability engineer – but a game designer by profession – perhaps the understanding of testing interactive software has advanced sufficiently in the 15 years since RITE was developed that it is outdated and these concepts are obvious and ubiquitous in the usability community. Is there a dominant documented method that you have found to work well for interactive games?
Having worked with a number of large and small third party QA testing organizations previously and on our current titles I can say that I have not found any where I could expect consistently useful design feedback as part of any QA testing.
I think you hit the nail on the head when you say that QA is not expected to do design analysis. I’ve been pretty amazed at the confusion out there when approaching people in the industry about usability research. Many dismiss it as an overlap with QA, which is aimed at debugging and basic functionality (white / black-box testing as you mentioned).
Another distinction that we make (at least internally) is to describe differences between usability testing and usability research. Many use the terms interchangeably, but we consider ‘testing’ as something closer to QA – looking at the user-interface and the ability of players to use the functionality in the game interface in intended (meaningful) ways. However, usability research goes further: rather than ‘testing’ an interface for common usability issues, it is more of an open-ended pursuit of whether users / game players are able to satisfy game goals, engage in strategy formation, and maintain flow while playing the game. These are often less quantitative in nature (e.g., testing an interface’s efficiency) and relate more to the overarching game design, and whether it is fun (or fulfills whatever goal it was meant, such as educational games), engaging, etc. This type of research often aims at uncovering the quality of game interactions, and therefore has much more relevancy in determining whether a game will achieve success.
I would agree that there is not necessarily a common approach to game usability research that consistently works. RITE is an excellent tool in the toolbox, but part of the motivation for writing the post was seeing others take it as gospel. Every usability method was developed with a specific purpose in mind, and often the best approach needs to be custom-designed for the project in mind. Each game, and development process, has it’s tradeoffs and associated advantages and disadvantages. Custom-designed research methods often are difficult to design – especially for games, due to their interdisciplinary nature. This often requires a deep understanding of game design and the technical background that supports the games (art, narratives, ludology, level design, audio, programming) as well as the gamer communities and behaviors surrounding the games themselves (gamer psychology, industry standards, game communities, collaborative play, etc.) and a multidisciplinary blend of social science research methods.
While definitely not industry standard, we feel that an open-ended holistic research approach (in a situated environment) is best for allowing gamers to act naturally, while avoiding pre-determined hypotheses (tests) that may impede the discovery of something unintended (often the powerful observation that means the most).