Interviews

Much like a questionnaire — a topic covered in the last installment — an interview is for collecting subjective data. However, the face-to-face nature of an interview means that you can be more interactive in your data collection, which if done correctly, can lead to very rich data. However, it is also obviously quite time-consuming, and it is harder to analyze and quantify the data you get at the end.

The quality of what you get out of an interview will also depend greatly on your own skill as an interviewer, so here are some tips.

The Setup

There are some basic things here. First, choose a good setting. Generally speaking, it should be comfortable, with as few distractions as possible.

Also, plan to generally only interview one (or perhaps two) people at a time. One is usually best, so as to avoid one person’s opinion dominating, but sometimes two people can have a nice dynamic and prompt each other — particularly if discussing a cooperative or multiplayer game.

Obviously, if you are going to be interviewing people in their own play environment, this gets harder, but still do you best to make sure extra distractions are not around (e.g. ask politely if the door can be closed, etc.).

Also it is useful to set up a way to record the interview; this doesn’t have to be video, but it is important that you at least record what is said. You will not remember everything, and even if you take great notes it is a good idea to have the recording to refer back to. This is also preferable to making notes constantly, as this tends to distract the person you are interviewing and can create a feeling that they are being assessed personally — rather than the game.

You will have to alert the person you are interviewing to the recording and get their permission. You will also have to explain to them what you are interested in asking them about, and also as with all of these methods make it clear that this interview is not about evaluating their performance, but to look at their experience of, and the performance of, the game.

You should also make an interview script; i.e. write down the questions you want to ask, and the order in which you want to ask them. However you should also be prepared to ask follow-up questions, although take care and don’t be confrontational when you do. It may be your game they are badmouthing, but you need to play it cool.

When coming up with these questions many of the same rules creating questionnaires apply (see part 1 of this series). In other words, try to be clear and precise with your questions, and check they are not leading, loaded, and only have one meaning. Also take care to avoid questions that can be answered by a simple yes or no; if you are after that kind of information use a questionnaire instead.

During the Interview

Start off with some easy stuff. Much like with a questionnaire, this warms them up, and will get them in the mood to talk to you. Once you get going also take care to only ask one question at a time, and usually only move on when you are sure the person you are talking to is finished. You should practice listening empathically and encourage responding by doing things like nodding your head and saying “mm-hmm” to acknowledge you are listening to them.

It is fine if they go off on a tangent; that is part of interviewing. However, if they go too far off (say, talking too much about a great design idea they have) or spend too long, be prepared to gently nudge them back into talking about things you are interested in.

Also, try to be as neutral as possible. One bit of advice given to interviewers is to act as if you have heard it all before. Not as in you are bored, but as in that you are not shocked or overly reactive to what they are saying. This may sound like it is against the idea of empathic listening, but if done correctly, it is perfectly possible to signal that you are listening carefully without also implying you are judging.

Finally, it can be useful to finish up the interview with a short period of time where your interviewees can add anything that you may not have asked about.

After the Interview

Make a transcript. Be aware this will take some time! Then, if you conducted multiple interviews, look for common themes and threads in what people are saying. One good thing about interviews (which is also a good thing about gameplay observation and think out loud methods) is that they can generate great sound bites or quotes that can be quite convincing when given to others on the development (or even management) team.

Pros

• Rich data source for subjective impressions
• Can ask follow up questions

Cons

• Not easy to quantify
• Time-consuming
• Not objective

Observational Studies

Observational studies (also sometimes called Behavioral Observation or Ethnographic Observation) are where you get some people to play your game and either record them doing so to watch later, or observe them directly. This technique is probably the most valuable of the traditional user research testing methods because nothing beats actually watching someone play your game. You can see how they progress, how they deal with challenges, and where they appear to become stuck or unmotivated.

You can also attempt to pay attention to their body posture and expression in order to try and gauge their emotions. Be careful, however, and only write down what you actually see — don’t try and infer any inner thought processes. Also the observer or facilitator should be as neutral and hands-off as possible. You want to see where people get stuck or make mistakes, so don’t help them unless you really, really have to.

Observation can be done in large groups if you have the correct lab space. But it is probably better done on a one-on-one basis so that players do not distract each other and the observer is also not distracted.

Again, there are several things to remember when setting up an observational study, and I will try and go through some of the steps below:

Step 1 – Designing the Observation Session

Work out where you will be testing, if you will be recording the session or not, and what part or parts of the game you will be testing. It would be ideal if you could test exhaustively, however with modern non-linear and emergent games this is obviously difficult. Still, you should plan to test at the very least large representative chunks of your game, and certainly make sure that the core mechanics are fun.

One useful thing to do when running an observational type test is to create and use test scripts. Test scripts, like a movie script, or the recipe for a delicious cake, lay out how the test will go and typically include the order of events, what the events are, and even scripted dialog for the tester to use.

Be specific with this. What happens when the user comes into the room? Do you offer them coffee? Where do they sit? Do you tell them where the bathroom and emergency exits are? Do you talk to them at any time during the test, or only at set points? What build do you load up? In what order to you present the game portions you are looking at?

This may sound overly scientific, and in fact some argue that it is a bad way to run research, as it scares the player and gets them in a mindset that makes them feel tested. This is a valid point. At the same time, however, if you want the best data, it is good to have at least some consistency, and certainly you need to have some plan for the session. Good researchers can do this and still not come across as a robot.

Ultimately, it is no good if at the start of the day you welcome a user happily, chat about their ride over to the lab as you offer them a coffee, and then later in the day you just rush the next user into the room and try and get things done as fast as possible. Then you may find that it is not your game they are rating, but you.

A good test script should include welcoming the participants, administrative stuff, and some introduction to the procedure (what will happen, how long it will take, etc.) Remember, again, at this point it is good to be casual and relaxed to make it clear that the game is being evaluated, not the player.

The script should also clearly describe the tasks they are going to be doing during the observation session, this can be as simple as “start the level and then get to the next level” but it is important that they have clear defined start and end points. This is because you want to know exactly when in this process certain issues or behaviors occurred.

You should also think about what kind of behaviors (in the game, and perhaps outside of it, depending on the game) that you expect to see, just so you know what to look out for. It can be helpful if many people are observing to agree on a clear sentence (or two) that outlines the behaviors you are observing — so what exactly does a double jump look like? What about a failed jump? A list of such agreed-upon descriptions of behaviors is called an ethogram, and it helps to cut down on confusion when discussing what was observed with others.

There are quite a few software packages out there aimed at helping out at this point (for example Morae or Observer, although there are many others) which can handle the recording of the screen and the participant, the timing of the test and also allow you to carry out quick predefined coding as you observe (or when you go back later to watch recorded video). This said, if you can’t afford to record and use fancy software then having a nice well-defined ethogram and somewhere to write down observations (sit somewhere the user can’t see you — even if that is sitting behind them) can still produce useful results.

Once you have everything ready to go, you should test out the setup you have to make sure everything runs smoothly and the instructions are clear. This is also important as the only way to become a good observer is get experience observing.

Finally before you can start you need to look into recruiting people to test. I covered this in part 1, but just to be clear, the most important thing at this point is that you want representative users. It’s no good testing it with the dudebro in the dorm next-door if your target market is young girls five to eight years of age. And if you are going to be testing with children you are opening up whole other box of issues (parental permission, testing in groups vs. alone, communication, and so on).

Step 2 – Running the Observation Sessions

What do you do once the session is running? Well, observe, of course. Watch the participants and take notes on what they do in the game, in real life, and what they say. As mentioned above, there is software that can help you do this, and certainly recording the play session will help you later if you can go back and review it. Plus the recordings can be great material to show other folks who might need convincing that something needs to change.

Be careful to note not only what the player is doing, but also the time at which they do it, and the situation in which it occurs. This context is important for interpreting the data.

Also when observing, write down only what you see and hear. So, write: “the player is laughing at the cutscene and smiling” not “the player is happy” as this latter description makes assumptions about the user’s internal states.

Again this may seems overly scientific, but in the long run the fewer assumptions you make, the better, and it helps when communicating with others what you found if you stick to clear descriptions of behavior.

In addition to the behaviors you see, you should also record down performance measures from in the game. In other words, you should note things like the time taken to clear a particular stage, the in-game score, the amount of damage taken and ammo used, and so on. This is especially important if you are not tracking these variables through a gameplay metrics system.

Finally, I should note that during this time it is good if the player is isolated from others (including you) and can concentrate on the game. This can be as advanced as an observation room at Microsoft or just you simply sitting in the same room out of the player’s line of sight.

The important thing is that you let the player play without feedback from you. This can be frustrating when you see someone getting stuck in a place where the solution is “obvious”, but this is exactly the type of thing you want to see. This means that you shouldn’t really talk to the player once the session has begun, and you should politely ask them not to talk to you either. Having them wear headphones can help as it adds to the sense of a self-contained environment, and can help put the player in the right frame of mind.

This doesn’t mean you can’t step in and ask a question, or help if things are going really wrong. However, generally speaking, it is better to note down periods where you are not really sure what was going on and then ask the player about them later, after the test is done.

Step 3 – End of the Session

This part is pretty straight forward. Thank the player for taking part, and perhaps take the opportunity to give them a final questionnaire or even conduct a quick interview with them. This interview/debrief gives you the chance to ask about any interesting observations you made and try and get some feedback on what was going on. Again, be careful to do this in a non-confrontational and neutral manner; don’t jump down their throat about how they should have just used the jetpack to get over that hole they kept falling in during level 3.

Finally, if you are running multiple sessions, it is usually advisable to give yourself about 15 to 30 minutes at this point to reset the room, and gather your thoughts before the next player/group of players. Observing for long periods can be very draining, so these little breaks can help recharge your batteries a little, and also allow you to think about and perhaps review the observations you have made so far.

An Aside: Contextual Inquiry

Another type of observation study is a contextual inquiry. This is essentially fieldwork where you play David Attenborough and go out and observe your players in their natural habitat. The idea being that you can observe them in their normal play setting going about their normal routines when using your game, and therefore get insights that may not be available in your own test situation.

In this case the test scripts I mentioned above are somewhat less useful, as the point here is to see how players use your game in their normal routine. However being prepared in terms of what you will say when you show up and how you will carry out the observation is still important.

Contextual inquiries also tend to be more useful for improving existing games, or observing similar games to yours to learn about how people play them in the real world. As such, testing prototypes or games in development in this fashion can be less useful, as players haven’t had an opportunity to develop a routine with them. Still, it is certainly nice to see how your game will be used (and abused) in the wild.

Pros and Cons of Observational Methods

That is all I will cover as far as observations go (with the exception of think out loud, which I will discuss in a bit). The biggest advantage of observation studies is that you really get to see what new players do when playing your game. This is invaluable and makes observation studies the best methodology, in my opinion. This method can also give you unexpected insight, and reveal issues that you may have never seen without it.

However, observations are obviously quite time-consuming, especially if you are recording everything. Also, it can take quite some work to be a good observer. As I already said, a good observer has to be as neutral and non-intrusive as possible. But also they have to know what to look for, and when to ask follow-up questions.

What typically helps is first practicing, and making a list of typical behaviors you would expect to see. Then you know what to look for before you go in to a real session. Remember you want to primarily be looking for their behavior in the game, so think about the types of things they will be doing. So for example do you notice them jumping repeatedly in one place? Or running into walls?

Also, avoid coloring what you see with your own expectations. Don’t try to infer internal states; just write down what you observe, and then if you want to know what was going on in their head, ask them later.

Pros

• Provides you with objective data. You are seeing what players actually do, not what they say they would do/have done
• Players may surprise you
• When recording equipment is used, the material collected can be used for other purposes

Cons

• Time-consuming, especially if you record and go back through everything carefully
• It takes experience to be a good observer
• Going into a test environment can make people nervous or at least aware that they are being monitored, and therefore they may not produce completely natural behavior

Think Out Loud Protocol

An expansion of the observation method, this is where you carry out observations as detailed above, but ask the players to talk about what they are thinking as they go along. You then record this while observing what they are doing. This adds information about the internal states of the player, and gives you an idea of what assumptions they are making based on the game. Again, you may hear things that are wrong, or that you don’t like, but resist the urge to correct the player.

This method can lead to unexpected insights that wouldn’t be gained otherwise, or at the very least gives you a little more insight into how people are mentally interacting with your game. However, it does have one big downside, and that is that it is quite unnatural to talk about what you are doing and feeling as you do it. This might change how people will actually play the game and affect how much or little fun they are having.

Related to this, you may find that some people talk non-stop, whereas others hardly say a word, a finding that may reflect more the differences in people than in your game.

Pros

• Allows for the collection of objective (behavioral) data while also getting access what players are thinking and feeling
• Can result in unexpected insights

Cons

• Time-consuming
• Unnatural
• What people say is still subjective

Gameplay Metrics

Similar to, and a great addition to, observational methods are gameplay metrics. This is basically is statistics generated by the game itself. which tell you about what the players did. For example think of the statistics systems that Bungie and 343 Studios have for Halo, or the Autolog system in Need for Speed.

Such gameplay metrics give you hard, objective, statistical facts about how the game plays. The collection of this data has to be built into the game, but once it is, you have a very large data source at your fingertips. Want to know what weapons take down players most on a map? What power-ups they use? Where they tend to die? Gameplay metrics make that possible. Good metric triggers and hooks can also really compliment observations as they take some of the load off you in terms of recording the in-game details about the player’s performance.

The power of metrics: on the left is where people are standing when they make kills with a weapon and on the right is deaths by this weapon in Halo Reach. With just a basic knowledge of FPS games, you can still probably work exactly what kind of weapon this is and where the elevated and the open spaces are in the level.

However, metrics can be time-consuming to collect and you need a good number of players to really take advantage of them. Plus metrics lack subjective information about emotion and what is going on with the players — it is just a mass of data.

Also, you can end up with data overload if you don’t analyze what you collect carefully. For example, load up any heat map in Halo Reach and turn on deaths by all weapons, and compare it to kills by all weapons. The result will generally give you a little information in that the kills and deaths seem pretty well spread out over the player area, but is not particularly useful for working out finer details of the player experience.

Pros

• Objective data that can be collected remotely and discretely
• Can be used to see trends and provide data that supports other methods
• Allows for continuous data recording without interrupting the player

Cons

• Time-consuming
• No subjective feedback
• Needs large sample sizes to get meaningful data
• However, at the same time, you can get too much data

Biometrics

I will finish up by covering biometrics. I have already done a primer on this topic, so I am not going to go into too much detail here. Biometrics are basically when you record signals from the body, such as brain waves via EEG, or where people are looking via eye tracking, to try and give us a clue as to your player’s internal body states. In this way they are again like think out loud is, and like metrics can be: an add-on to observation-based testing.

Much like observation and gameplay metrics, biometric measures can be useful because they give you objective data on what is going on. They also allow for data on emotions and excitement to be collected continuously during play without having to stop and ask questions about how people are feeling.

However, the ways you can collect biometrics are currently quite invasive in that you have to wire people up, and they cost quite a bit of money and time to use, not to mention you have to be trained in how to use them — as well as how to analyze the data that you get.

Also they have problems in that you can get artifacts or fake signals in your data, and that there is no real solid agreement on what you are measuring actually means. For example, heart rate may go up in a certain part of your game, which could mean it is exciting, but it could also mean it is scary and unpleasant, or that your player is so bored they are now thinking about last night in bed with their partner, or even just that the person you are recording took a deep breath (which also increases your heart rate).

The above issues mean that you have to combine biometric data with other sources of data (observations, interviews, questionnaires) in order to work out what is going on with the physiological signals you are collecting. This has led some people to doubt its usefulness, arguing that it doesn’t add anything that you couldn’t already get but does add a bunch of complications and extra work (see an excellent discussion on this point, and on games user research in general, by Bungie’s John Hopson and Valve’s Mike Ambinder here).

Proponents of biometrics, on the other hand, suggest that the strength of biometric data is that it gives you access to emotions and reactions that the players may not know they are having, and therefore helps you look at other data sources in a way you haven’t before. Ultimately, however, it is just another tool and you will have to judge if it is appropriate for you or not.

Pros

• Gives objective quantifiable data
• Allows for continuous data recording without interrupting the player

Cons

• Invasive
• Costs a lot of time and money
• Problems with specificity, artifacts, inference and validity can make it difficult to interpret

Final Words

As I stated in the introduction these two articles are intended as rough primers, and are in no way comprehensive. I am sure that others will disagree with what I say, and perhaps others will be happy to add more or provide material of their own. However, I do hope though that this document can be of some use.

If you are interested in further reading on this topic then I can recommend checking out this excellent document from Mike Ambinder at Valve and watching this discussion between Mike and John Hopson from Bungie. As mentioned in the last article, you may also consider checking out the IGDA Special Interest Group for Games User Research on LinkedIn.

Furthermore a wiki-based games user research primer is in the works. This primer will arise out of a workshop at the CHI 2012 conference and is planned to be an evolving wiki based site that can provide the community with up to date information on game research methodologies. So watch that space.

Finally, please remember that the above methods are here you help you develop your games. The data they give can be rich and interesting, but it should not be domineering. In the end, it is your creative vision that guides the game; the data that can be collected via games user research is there to help you, NOT limit you.

Written by: Ben Lewis-Evans, via Gamasutra
Posted by: Situated Research

The post Finding Out What They Think: A Rough Primer To User Research, Part 2 appeared first on Situated Research.

This article, and its forthcoming followup, is intended to give a rough idea to developers of several different methods that can be used in games user research.

However, many, many books have been written on research methodology and I cannot cover everything. Therefore these two articles cannot be taken as completely comprehensive.

In the first of the articles I will be covering a few general points about Games User Research and then discussing three methods, focus groups, heuristic evaluation and questionnaires in some detail.

What is Games User Research?

So, before getting really started, what is games user research in the first place? Well, let’s start by comparing it with Quality Assurance (QA). QA is a well-established part of software development, and is often carried out by professionals within a development team. These folks are, generally speaking, aimed at finding bugs and making sure the game runs smoothly.

However, because those working on a project usually carry out QA, it means they have an investment in it, and are familiar with it. This can cause problems when it comes to evaluating the usability and experience of a game. What seems obvious and fun to someone that has been working on a project may be completely alien and frustrating to a new user.

This is where games user research methods come in, a field that is all about the user and their experience of the game — in particular, the big question, “is it fun?”

To really (over) simplify things, it could be said that QA and test is about the software, and how it functions when dealing with users, and game user research is about the user and how they function when dealing with the software. Notice I say just it is about how the user functions when dealing with the software, and that it is NOT about testing the user (more on that later).

So how is this done? And what is fun, anyway? Well, there has been plenty written on fun already, so let us just say that fun has quite a few dimensions. Fun can be something that is easy to use, but it can also come along with a struggle and a challenge. It can arise from an engaging experience, a compelling experience, or a relaxing one.

All this means fun is a subjective variable, which changes from person to person, and situation to situation. However, due to the emotional component of fun, what can be said is that if your players are having fun, then it is likely they can tell you about it — if only you know how to ask.

So How Do We Ask?

Okay — I’ll get to that. But before I get into the fine details, please bear with me as I outline some general principles to keep in mind:

Get the right players

Whatever methods you choose, make sure you get the right sample. For most methods, this means getting representative users, aka the people that you expect will play your game.

If you have the time, perhaps there is some advantage to getting as wide and large a user group as possible (if you really think that everyone will want to play your game), but given that you are likely to be constrained by time (and money) it is generally best to concentrate on getting as close a sample of the target type of player you are after as possible.

The game is being tested, NOT the user

Secondly whenever doing this type of research make sure it is clear to the user that they are not being tested. The research is about improving the game, not the user, so the user shouldn’t be made to feel inadequate if they can or cannot do something. In principle it is all valuable information.

This can be hard to do sometimes, as you are the ones designing the game, and it can be uncomfortable to hear others criticising it. But try your best to not be defensive or judgemental (i.e. avoid thinking the problem exists between the keyboard and chair.)

What do you want to know?

Whenever doing research, you should be clear about what you want to know. You will be playing the game yourselves as you work on it, and you should have design documents, so you know how things are supposed to work. So don’t just plonk people down with the game and come up with questions on the fly. Work out what areas you think might be problems and know what you want to ask about before it’s time to test.

Please note: I am not saying you go out there with preconceptions and already “know” the answers you want; rather I am just making the point that you should be at prepared and know what you’re looking for. Otherwise, you get a mass of data that may not be of any use to you at all. At the same time, be open to surprises. You never know what might pop up.

Test early and test often

The next point is considered one of the most important by user researchers, and that is to test as early as you feel it is possible to do so. This can be difficult, as it feels bad letting your baby out into the hands of users before it is 100 percent compete. But really, the sooner you can test the better — test with paper prototypes, for instance!

The primary reason for this is that it is much easier to change the game if you find an issue early in development rather than late. Once you have made the changes, you should test again. That said, it is also a good idea to make sure that the product isn’t too buggy; you want to test the experience of playing the game, not the experience of crashing due to bugs.

One extreme example of this approach is the Rapid Iterative Testing and Evaluation (RITE) method, developed by games user researchers at Microsoft. In this method a test is run (usually via behavioural observation — which will be covered in the second article in this series), and changes are made to the game as soon as problems or issues are detected, before the next participant arrives. This can occur perhaps even after just one user has been tested.

Listen to and act on problems, but not necessarily the solutions

When dealing with your users, you should be open, and listen to the problems they are raising. You can also listen to the solutions they give to those problems — but they are likely to be less useful to you. You are the game developers, and you know what is possible with the technology, time, and resources you have. The users won’t. So observe, do the research, and treat it seriously when it reveals issues, but take suggestions from users as to how to solve the issues with a grain of salt.

Games user research is just another source of data

Often when articles such as this appear online, there is much gashing of teeth and angst about taking the art out of game design and instituting “design by committee”. I can understand this worry; however, much like QA, user research is just another tool to improve your game. It shouldn’t dominate your design or suppress your artistic talent; rather, if done correctly, it should augment your talent and give you new insight.

Time for the Crunchy Stuff

Are you still with me? Great. Here, then, are the research methods that I will cover, in varying degrees of depth, in the rest of this document: focus groups, heuristic evaluation, and questionnaires/surveys.

Focus Groups

This method can be something of a dirty word (and is definitely part of that whole “design by committee” problem). So let us deal with it first and get it out of the way.

You are probably familiar with focus groups, even if you’ve never seen them used in person. Basically this is where you get a bunch of people, have them play your game, and then put them in a room to talk about it. They can be free to talk about what they like and what they didn’t like, but you also have a facilitator in the room who can ask specific questions of interest.

This process can also be used quite early on in development, where instead of getting people to play the game, you give a presentation or talk about ideas for the game and get feedback on that.

The advantages of focus groups are that they involve a lot of people, so you can get more feedback. They can also be somewhat efficient, as everyone is together in one place, and the facilitator can ask follow-up questions. So if someone mentions they like or dislike something in particular, you can gain a bit more detail on why.

Although at the same time if you are not careful, focus groups can get away from you, and end up wasting far too much time. To avoid this last problem you really need a good facilitator to run a focus group. The facilitator has to be strong enough to guide the conversation to areas that are useful, but not dominate the discussion.

Probably the biggest risk with a focus group, and one of the reasons why focus groups are not often used, is that just one or two members of the focus group can dominate the discussion. Due to group pressure, you may not hear from other people who have valuable insight into the game. Focus groups also have a tendency to get more into discussing solutions for issues rather than just the issues themselves. This is not what you want.

Finally, focus groups are a subjective method, and all you have to go on is what people say — and as much as we like to judge people on the “attitudes” they hold, and think that they predict behaviour (they usually don’t), we all know that what people say is not always what they actually do.

Pros

• More people can mean more feedback (although see cons…)
• Gets everyone together in one place
• Allows for follow-up questions
• Can be useful when discussing concepts

Cons

• A good facilitator is required
• Strong voices may take over and reduce feedback overall
• Too many solutions, not enough issues
• What people say is not always (or even often) what they do

Heuristic Evaluation

Heuristic evaluation is where you get an expert (or experts) in games user research, get them to play your game, and then they evaluate it on a set of criteria (heuristics). Kind of like a scientific game review… Kind of.

Basically, to do this, the expert(s) will use a list of heuristics, which are basically rules or mental models, and give you feedback on whether your game fits these heuristics, and where problems might come up. These heuristics can vary, but here is a selection of some possible heuristics listed in a 2009 article [PDF link] by Christina Koeffel and colleagues:

• Are clear goals provided?
• Are the player rewards meaningful?
• Does the player feel in control?
• Is the game balanced?
• Is the first playthrough and first impression good?
• Is there a good story?
• Does the game continue to progress well?
• Is the game consistent and responsive?
• Is it clear why a player failed?
• Are there variable difficulty levels?
• Are the game and the outcome fair?
• Is the game replayable?
• Is the AI visible, consistent, yet somewhat unpredictable?
• Is the game too frustrating?
• Is the learning curve too steep or too long?
• Emotional impact?
• Not too much boring repetition?
• Can players recognize important elements on screen?

The article itself lists over 29 of these heuristics, and goes into much more detail than I have provided here, so I recommend reading it if you have the time.

The good thing about heuristics, in my opinion, is that even if you aren’t an expert, they can provide a list of things for you to think of when you look at your game. For instance, does it provide enough feedback to players that their actions are affecting the world? Does it force players to hold the controller in an awkward fashion? And so on. Again, this may seem like common sense stuff, but it really is amazing how often so-called “common sense” is not common at all.

Now, one obvious advantage to heuristic evaluation is that you only need to use a small number of experts (just one in some cases), and being experts they know what they are talking about. This also leads to the problem that you do need experts, and where do you find those? Plus have you found the right one(s)? The types of heuristics used can vary by expert, and of course should fit your type of game — so this is obviously important.

Also sometimes experts can be a bit too expert and miss stuff that might be a problem for novices. This is because as we become more experienced at something we no longer need to consciously consider everything that we perceive and do, whereas someone who is still learning is still thinking about what they are doing all the time. This is why generally if you want to see something done well you watch an expert, but if you want to learn how to do something it is often best to ask a novice.

Pros

• Smaller number of people needed
• Relatively fast turn around
• Experts are expert

Cons

• Where do you find experts?
• Did you find the right one?
• Experts can be too expert

Questionnaires & Surveys

I am sure you know what a questionnaire is, but do you know how to design and use one correctly? Mountains of books have been written on this, but hopefully I can make at least some important points clear.

First of all, when do you use questionnaires? Well, they are usually used to evaluate subjective views about your game, particularly value judgements. This may vary from specifically asking players about their favorite weapon to open-ended questions asking for general comments on the experience.

Questionnaires can be given to players during the game, which means the experience is fresh, but this risks interrupting the flow of the game (if possible, find natural down times to ask). They can also be used after a gameplay session is over. The big advantage of questionnaires is that they can be given to many people, and as such you can end up with lots of nice data to examine (in theory).

Before I go into the detail of constructing your own questionnaire, there are some pre-existing questionnaires out there aimed at evaluating the fun of gameplay experiences.

Examples of such questionnaire are the Game Experience Questionnaire (GEQ) for examining gameplay experiences and the affect grid [PDF link] or the manikin system [PDF link] for rating emotions. These pre-existing questionnaires can be great, because they are usually well-written and reliable. However, they also have a tendency to be more academic in nature, and care should also be taken that they fit your game, so modify them if necessary.

So how do you go about designing your own questionnaire? Here are four steps that I hope can help you.

Step 1. Work out what you want to know

As stated already, all of these methods require that you know what information you are after. But this is extra important when it comes to questionnaire design. You usually don’t get any chance to follow up on people’s answers, so you want them to be as clear as possible, and for the information you gain to be what you are after (and neither too little nor too much!)

So, brainstorm, make lists, do whatever is best for you in terms of getting down what you want to know. Then cut it down to only what you really, really need to know. Be focused!

A mistake that novice researchers often make is to ask for everything simply because the opportunity is there. However what this results in is a mess of data that will take forever to be analyzed, and may not produce a meaningful result. You don’t really want your questionnaire taking more than 15 to 20 minutes to answer (this is not a target, by the way, but a maximum).

Step 2. Design the content

The design part is the meat of the process and can be further broken down into a few things to consider.

Questions or statements? Do you want people to answer questions, or rate statements? This is pretty straightforward but still important. Basically, questions are good for gaining information, for example:

How challenging was the Horrible Bog Beast?

1 2 3 4 5 6 7
Very easy              Very Hard

Questions can also be worded in the form of an instruction. For example, “Rank the levels you just played in order from 1-6, where 1 is the one you enjoyed the most, and 6 is the one you enjoyed the least.” It’s effectively just like asking a question about the ratings for each level individually.

On the other hand, getting people to rate statements is more typically used to assess value judgments or agreement with ideas, so an example would be something like:

The Horrible Bog Beast is an interesting enemy to encounter.

1 2 3 4 5 6 7
Strongly Agree                   Strongly Disagree

Either questions or statements are fine; however, use them where they are best, and don’t switch between the two types too often.

Language use. It is incredibly important that you use clear, everyday language. Avoid jargon. This is vital because you want to be sure that the people you are testing understand what you are asking. Many people will just answer anyway, even if they don’t understand a question (be honest — I am sure you have done it) and the data you get may not be useful at all. So be blunt, be direct, and ask for exactly what you want to know.

When you offer alternative answers to your questions (for example if they have to select from a list of consoles they own), these should be relatively exhaustive. So in other words there shouldn’t be any alternatives that you haven’t thought of. Adding an “other” where additional options can be filled in can help here, but it is best if this “other” option isn’t used too often.

You should also be careful not to ask what is essentially the same question, but in a different way. Remember the goal is to keep the number of questions down. Also avoid asking questions that are phrased negatively. So “I like the jumping mechanic” and a gradient from agree to disagree rather than “I don’t like the jumping mechanic”. In the case of negatively-phrased questions, people filling your questionnaire have to select “agree” to disagree, and “disagree” to agree. It seems silly, but this can confuse people.

Also avoid leading, double barreled, and loaded questions. A leading question is one where the person filling it in is lead or biased to give a certain answer, e.g. “This game was fun. How fun was it?” This example is, of course, over the top, but you would be surprised how often leading questions can slip into questionnaires, so just try and keep your wording direct, and neutral. Don’t assume. Ask!

Double (or multiple) barreled questions ask about more than one thing at a time. Here’s an example from outside the world of games — from a national referendum held in New Zealand:

“Should there be a reform of our justice system placing greater emphasis on the needs of victims, providing restitution and compensation for them and imposing minimum sentences and hard labour for all serious violent offences?” YES/NO

As you can see, there are at least six questions here; should there be a reform, should it place greater emphasis on the needs of victims, should it provide restitution and compensation, should it impose minimum sentences, should it impose hard labour, and should it be for all serious violent offences? But you only get to say yes or no once. As I say, this question was put to the whole of New Zealand, and really ruined my first ever experience of voting.

Loaded questions are ones that make moral judgments or make assumptions that are unfounded; an example of this type of question also comes from a New Zealand referendum where it was asked:

“Should a smack as part of good parental correction be a criminal offence in New Zealand?” YES/NO

This question is loaded in that it uses a moral term like “good”. It is also ill-defined, as it uses the term “good parental correction” (whatever that is), and finally it is misleading in that if you are anti-smacking you have to say yes (agree) and if you are pro-smacking you have to say no (disagree).

Closed or open? Questions can be either open, as in the player can say whatever they like, or closed, where there are only a few limited options to select.

Open questions allow for richer data to be collected, as they let players give as much feedback as they want. However they can also give as little as they want, and often without direction, the answers may be vague.

One way to get around this if you do use open questions is to make sure you have time to read them before the end of the session, in case you want to ask for clarification on any points people bring up. However this is difficult if the questionnaire is being given remotely, such as when it is being used via email or online.

Closed questions on the other hand let you have much tighter control on the answers your participants give, and come in several flavors (scales):

• Dichotomous: This is for simple yes/no questions. So usually it used to collect stuff like yes/no, true/false. It is nice, direct, and precise. However, it is also not very data rich. So generally speaking such questions are used just to collect demographics, or when you really want to force a choice between two options.

A dichotomous choice in Diablo III

• Continuous: This is where you ask people to give ratings along a continuum or sliding scale (like when you are doing face customization in games). The big advantage of a continuous scale is that it gives a lot of resolution. However that much resolution is hardly ever needed (is there really a difference between a rating of 96.43 and a 93.21?).

A (semi) continuous scale in Saints Row: The Third

• Interval: This is probably the type of scale we are most familiar with. Here, you give a rating along some sort of scale with each step being separate/discrete from the other (rather than continuous).

This does lower the resolution of the scale when compared to a continuous scale, but it is much clearer in terms of allowing for comparison of results. In my opinion, a 1-7 scale is probably the best way to go (over say a 1-5, or something larger) as it allows for some good resolution, while not being too broad. Others may disagree

Several interval scales in Arkham Horror

If an interval scale is used, then it can also be broken down further into several types: numeric/categorical, Likert, or semantic.

• A numeric scale is simple; it just asks for a number, and is used often to rank things.
• Likert scales you have probably seen many, many times and are usually for when you want to see how much someone agrees or disagrees with a certain statement,e.g. 1-7 where 1 is strongly disagree and 7 is strongly agree
• Semantic scales are when you want to simply ask for a rating related to something or a value judgment, e.g. 1-7 where 1 is poor and 7 is good.

Each of these types of scales can be used where necessary, although generally speaking, it is best not to mix scale types too much. Otherwise people filling in the questionnaire might get confused and provide a rating that they think is using an agree to disagree dimension but is actually asking for poor to good.

Finally, with interval scales, you can either make them unipolar, where you ask for varying levels of one variable e.g. 1-7 where 1 is not very exciting and 7 is very exciting, or they can be bipolar where you contrast two different variables, e.g. 1-7 where 1 is very boring and 7 is very exciting. Unipolar scales zoom in on one area a bit better, but bipolar scales give people filling in the questionnaire more room to express their opinions. As with before, try to not mix these styles up within the same questionnaire too often, or at all.

Step 3. Put it together

You have your questions, so now’s the time to put it together. First, work out what medium you are going to use. Will it be collected via a computer (say a through a web interface), or via paper? Collecting via a computer is preferable if possible as it means that there will be less data entry later on, however sometimes the portability and ease of paper can win out.

Give yourself plenty of time to do this, and if you are planning on using a computer-based survey, there are many companies online that offer this kind of service. In particular I hear that SurveyMonkey is a popular choice, although I have not used it myself.

You should next consider what order to put your questions in. Generally speaking I advise putting the easy questions at the start. This gets people rolling, and as long as your questionnaire isn’t too long, they should be more prepared to tackle the bigger questions you want to ask later.

Then try and cluster the remaining questions in a sensible fashion, such as by subject, or what they refer to within the game. Don’t ask a bunch of questions about the weapons, then move on to the boss, and then go back to the weapons.

Also check to see if answering some questions excludes other questions or could cause new questions to be asked later on. If so, make sure this occurs. In other words, if you ask “Do you own an Xbox” yes/no, make sure later on the people who answered “no” aren’t asked to list the top ten Xbox games that they own.

Step 4. Test it

No plan survives its first contact with the players, so first test your questionnaire yourself. If you are using a computer-based questionnaire, make sure the data that comes out the end looks okay. Give it to a few other people (preferably people who are not too familiar with the game, or are not designers) and ask them to fill it in (with you outside of the room so you can’t help them), and then ask for their comments on the clarity of the questions. Essentially, user test it, and then change anything that might be wrong (and then test again…) This is a great deal of work, I know, but once you have a good questionnaire, you can possibly use it again in the future (or at least cannibalize it for juicy parts).

Pros and Cons

There is much more that could be said, but perhaps this is already too much for a “primer” — so I will move on. The best thing about a questionnaire is that you are asking the same questions to everyone, so you get consistent quantifiable data that you can compare between people.

However they lack follow-up, in that you can’t ask people why they selected a certain rating, they are not objective, and you do need quite a few people if you want to draw completely solid conclusions from them. That said, even when testing with just a few people, providing a questionnaire can give you some ideas about what those individuals thought.

Pros

• Consistent
• Quantifiable
• Relatively quick to administer
• Can be used on a large scale

Cons

• Can lack follow-up
• Not objective
• Really at their best with large sample sizes
• It can take a while to put together a good questionnaire

Hopefully that is more than enough for this first article. While again this article is in no way comprehensive, I hope it has proven to be useful. In the upcoming part of this rough primer, I will be covering interviews, observational methods, game metrics and biometrics.

Finally, if you are interested in games user research or work in the area, then please consider checking out the IGDA Special Interest Group for Games User Research (GUR-SIG) on LinkedIn (just search for the group). It is a great place to get together and discuss GUR with others working in or around the industry.

Written by: Ben Lewis-Evans, via Gamasutra
Posted by: Situated Research

The post Finding Out What They Think: A Rough Primer To User Research, Part 1 appeared first on Situated Research.

Many video game usability practitioners employ a method to test usability within video games, called the ‘RITE’ method, short for Rapid Iterative Testing and Evaluation (RITE). Pioneered at Microsoft Games Studios and Microsoft Research, the RITE method has been adopted by many usability research organizations besides the teams at Microsoft.

While the RITE method has some advantages, such as the ‘rapid iterative’ ability to suggest changes to designers and test them in successive passes, it may fall short when looking for usability issues that lie beneath the surface.

The RITE method has its benefits: for instance, the ‘rapid iterative’ research design allows usability researchers to detect problems and inform the game design team. This is a nice fit with highly communicative ‘agile’ development processes: allowing usability errors to be corrected and retested in successive iterations. As cyclical (iterative) rounds of testing go on, the number of usability problems originally detected typically goes down with each iteration.

However, one should consider the quality of the usability problems being found by the RITE method. Does a rapid and iterative approach dive deep enough into the user experience to find problems that are not otherwise being found by existing quality assurance teams?

Those unfamiliar with usability research can often confuse it with quality assurance testing, which is not the case. In a recent conversation with Jakob Nielsen, he described an analogy that explains usability and user-experience research for designers:

“usability research is to design, as quality assurance is to testing”

When hiring a usability research team, game companies should expect more than basic usability testing. While basic usability testing is very important, these findings are aimed at problems that lie closer to the surface, such as issues dealing with the user interface. While these issues are important to find, they probably can be found through traditional quality assurance or game testing teams.

While Situated Research is interested in finding basic usability issues, we are equally interested in finding issues that lie deep beneath the surface. We feel these deeply-rooted issues may not be suitably addressed by the RITE method, for its ‘rapid iterative’ approach fails to conduct an open-ended, in-depth analysis. Uncovering patterns in player motivation, which ultimately form much of players’ opinions of a game, can provide high-value findings for game companies aiming to improve a game and its future sales. A mixed quantitative-qualitative approach can identify irregular problems that may have a more serious impact on gameplay, in addition to frequently occurring problems that might be identified by the RITE method.

Patterns in player motivation require a research method that dives deeper than traditional usability testing, and should incorporate theory blending research from fields such as human communication, behavioral psychology, and human-computer interaction. Situated Research maximizes a game company’s ROI by examining both the individual and social plane to find deeply-rooted issues that point to the core of the player’s experience. Going beyond traditional usability heuristics, which test the usability of specific portions of an interface, is necessary to find those deeply-rooted design issues that can go undetected by traditional quality assurance teams.

Written by: Matthew Sharritt, Ph.D.
Posted by: Situated Research

The post What’s Wrong With the RITE Method? appeared first on Situated Research.