[The following is the second of two articles by college professor and researcher Ben Lewis-Evans on games user research methodology (see Part 1, which covered focus groups, heuristics, and questionnaires, as well as giving a grounding in the topic of user research in general. In this article, Lewis-Evans covers interviews, observational methods (including think out loud and contextual inquiry), game metrics, and biometrics.]

Interviews

Much like a questionnaire — a topic covered in the last installment — an interview is for collecting subjective data. However, the face-to-face nature of an interview means that you can be more interactive in your data collection, which if done correctly, can lead to very rich data. However, it is also obviously quite time-consuming, and it is harder to analyze and quantify the data you get at the end.

The quality of what you get out of an interview will also depend greatly on your own skill as an interviewer, so here are some tips.

The Setup

There are some basic things here. First, choose a good setting. Generally speaking, it should be comfortable, with as few distractions as possible.

Also, plan to generally only interview one (or perhaps two) people at a time. One is usually best, so as to avoid one person’s opinion dominating, but sometimes two people can have a nice dynamic and prompt each other — particularly if discussing a cooperative or multiplayer game.

Obviously, if you are going to be interviewing people in their own play environment, this gets harder, but still do you best to make sure extra distractions are not around (e.g. ask politely if the door can be closed, etc.).

Also it is useful to set up a way to record the interview; this doesn’t have to be video, but it is important that you at least record what is said. You will not remember everything, and even if you take great notes it is a good idea to have the recording to refer back to. This is also preferable to making notes constantly, as this tends to distract the person you are interviewing and can create a feeling that they are being assessed personally — rather than the game.

You will have to alert the person you are interviewing to the recording and get their permission. You will also have to explain to them what you are interested in asking them about, and also as with all of these methods make it clear that this interview is not about evaluating their performance, but to look at their experience of, and the performance of, the game.

You should also make an interview script; i.e. write down the questions you want to ask, and the order in which you want to ask them. However you should also be prepared to ask follow-up questions, although take care and don’t be confrontational when you do. It may be your game they are badmouthing, but you need to play it cool.

When coming up with these questions many of the same rules creating questionnaires apply (see part 1 of this series). In other words, try to be clear and precise with your questions, and check they are not leading, loaded, and only have one meaning. Also take care to avoid questions that can be answered by a simple yes or no; if you are after that kind of information use a questionnaire instead.

During the Interview

Start off with some easy stuff. Much like with a questionnaire, this warms them up, and will get them in the mood to talk to you. Once you get going also take care to only ask one question at a time, and usually only move on when you are sure the person you are talking to is finished. You should practice listening empathically and encourage responding by doing things like nodding your head and saying “mm-hmm” to acknowledge you are listening to them.

It is fine if they go off on a tangent; that is part of interviewing. However, if they go too far off (say, talking too much about a great design idea they have) or spend too long, be prepared to gently nudge them back into talking about things you are interested in.

Also, try to be as neutral as possible. One bit of advice given to interviewers is to act as if you have heard it all before. Not as in you are bored, but as in that you are not shocked or overly reactive to what they are saying. This may sound like it is against the idea of empathic listening, but if done correctly, it is perfectly possible to signal that you are listening carefully without also implying you are judging.

Finally, it can be useful to finish up the interview with a short period of time where your interviewees can add anything that you may not have asked about.

After the Interview

Make a transcript. Be aware this will take some time! Then, if you conducted multiple interviews, look for common themes and threads in what people are saying. One good thing about interviews (which is also a good thing about gameplay observation and think out loud methods) is that they can generate great sound bites or quotes that can be quite convincing when given to others on the development (or even management) team.

Pros

• Rich data source for subjective impressions
• Can ask follow up questions

Cons

• Not easy to quantify
• Time-consuming
• Not objective

---

Observational Studies

Observational studies (also sometimes called Behavioral Observation or Ethnographic Observation) are where you get some people to play your game and either record them doing so to watch later, or observe them directly. This technique is probably the most valuable of the traditional user research testing methods because nothing beats actually watching someone play your game. You can see how they progress, how they deal with challenges, and where they appear to become stuck or unmotivated.

You can also attempt to pay attention to their body posture and expression in order to try and gauge their emotions. Be careful, however, and only write down what you actually see — don’t try and infer any inner thought processes. Also the observer or facilitator should be as neutral and hands-off as possible. You want to see where people get stuck or make mistakes, so don’t help them unless you really, really have to.

Observation can be done in large groups if you have the correct lab space. But it is probably better done on a one-on-one basis so that players do not distract each other and the observer is also not distracted.

Again, there are several things to remember when setting up an observational study, and I will try and go through some of the steps below:

Step 1 – Designing the Observation Session

Work out where you will be testing, if you will be recording the session or not, and what part or parts of the game you will be testing. It would be ideal if you could test exhaustively, however with modern non-linear and emergent games this is obviously difficult. Still, you should plan to test at the very least large representative chunks of your game, and certainly make sure that the core mechanics are fun.

One useful thing to do when running an observational type test is to create and use test scripts. Test scripts, like a movie script, or the recipe for a delicious cake, lay out how the test will go and typically include the order of events, what the events are, and even scripted dialog for the tester to use.

Be specific with this. What happens when the user comes into the room? Do you offer them coffee? Where do they sit? Do you tell them where the bathroom and emergency exits are? Do you talk to them at any time during the test, or only at set points? What build do you load up? In what order to you present the game portions you are looking at?

This may sound overly scientific, and in fact some argue that it is a bad way to run research, as it scares the player and gets them in a mindset that makes them feel tested. This is a valid point. At the same time, however, if you want the best data, it is good to have at least some consistency, and certainly you need to have some plan for the session. Good researchers can do this and still not come across as a robot.

Ultimately, it is no good if at the start of the day you welcome a user happily, chat about their ride over to the lab as you offer them a coffee, and then later in the day you just rush the next user into the room and try and get things done as fast as possible. Then you may find that it is not your game they are rating, but you.

A good test script should include welcoming the participants, administrative stuff, and some introduction to the procedure (what will happen, how long it will take, etc.) Remember, again, at this point it is good to be casual and relaxed to make it clear that the game is being evaluated, not the player.

The script should also clearly describe the tasks they are going to be doing during the observation session, this can be as simple as “start the level and then get to the next level” but it is important that they have clear defined start and end points. This is because you want to know exactly when in this process certain issues or behaviors occurred.

You should also think about what kind of behaviors (in the game, and perhaps outside of it, depending on the game) that you expect to see, just so you know what to look out for. It can be helpful if many people are observing to agree on a clear sentence (or two) that outlines the behaviors you are observing — so what exactly does a double jump look like? What about a failed jump? A list of such agreed-upon descriptions of behaviors is called an ethogram, and it helps to cut down on confusion when discussing what was observed with others.

There are quite a few software packages out there aimed at helping out at this point (for example Morae or Observer, although there are many others) which can handle the recording of the screen and the participant, the timing of the test and also allow you to carry out quick predefined coding as you observe (or when you go back later to watch recorded video). This said, if you can’t afford to record and use fancy software then having a nice well-defined ethogram and somewhere to write down observations (sit somewhere the user can’t see you — even if that is sitting behind them) can still produce useful results.

Once you have everything ready to go, you should test out the setup you have to make sure everything runs smoothly and the instructions are clear. This is also important as the only way to become a good observer is get experience observing.

Finally before you can start you need to look into recruiting people to test. I covered this in part 1, but just to be clear, the most important thing at this point is that you want representative users. It’s no good testing it with the dudebro in the dorm next-door if your target market is young girls five to eight years of age. And if you are going to be testing with children you are opening up whole other box of issues (parental permission, testing in groups vs. alone, communication, and so on).

---

Step 2 – Running the Observation Sessions

What do you do once the session is running? Well, observe, of course. Watch the participants and take notes on what they do in the game, in real life, and what they say. As mentioned above, there is software that can help you do this, and certainly recording the play session will help you later if you can go back and review it. Plus the recordings can be great material to show other folks who might need convincing that something needs to change.

Be careful to note not only what the player is doing, but also the time at which they do it, and the situation in which it occurs. This context is important for interpreting the data.

Also when observing, write down only what you see and hear. So, write: “the player is laughing at the cutscene and smiling” not “the player is happy” as this latter description makes assumptions about the user’s internal states.

Again this may seems overly scientific, but in the long run the fewer assumptions you make, the better, and it helps when communicating with others what you found if you stick to clear descriptions of behavior.

In addition to the behaviors you see, you should also record down performance measures from in the game. In other words, you should note things like the time taken to clear a particular stage, the in-game score, the amount of damage taken and ammo used, and so on. This is especially important if you are not tracking these variables through a gameplay metrics system.

Finally, I should note that during this time it is good if the player is isolated from others (including you) and can concentrate on the game. This can be as advanced as an observation room at Microsoft or just you simply sitting in the same room out of the player’s line of sight.

The important thing is that you let the player play without feedback from you. This can be frustrating when you see someone getting stuck in a place where the solution is “obvious”, but this is exactly the type of thing you want to see. This means that you shouldn’t really talk to the player once the session has begun, and you should politely ask them not to talk to you either. Having them wear headphones can help as it adds to the sense of a self-contained environment, and can help put the player in the right frame of mind.

This doesn’t mean you can’t step in and ask a question, or help if things are going really wrong. However, generally speaking, it is better to note down periods where you are not really sure what was going on and then ask the player about them later, after the test is done.

Step 3 – End of the Session

This part is pretty straight forward. Thank the player for taking part, and perhaps take the opportunity to give them a final questionnaire or even conduct a quick interview with them. This interview/debrief gives you the chance to ask about any interesting observations you made and try and get some feedback on what was going on. Again, be careful to do this in a non-confrontational and neutral manner; don’t jump down their throat about how they should have just used the jetpack to get over that hole they kept falling in during level 3.

Finally, if you are running multiple sessions, it is usually advisable to give yourself about 15 to 30 minutes at this point to reset the room, and gather your thoughts before the next player/group of players. Observing for long periods can be very draining, so these little breaks can help recharge your batteries a little, and also allow you to think about and perhaps review the observations you have made so far.

An Aside: Contextual Inquiry

Another type of observation study is a contextual inquiry. This is essentially fieldwork where you play David Attenborough and go out and observe your players in their natural habitat. The idea being that you can observe them in their normal play setting going about their normal routines when using your game, and therefore get insights that may not be available in your own test situation.

In this case the test scripts I mentioned above are somewhat less useful, as the point here is to see how players use your game in their normal routine. However being prepared in terms of what you will say when you show up and how you will carry out the observation is still important.

Contextual inquiries also tend to be more useful for improving existing games, or observing similar games to yours to learn about how people play them in the real world. As such, testing prototypes or games in development in this fashion can be less useful, as players haven’t had an opportunity to develop a routine with them. Still, it is certainly nice to see how your game will be used (and abused) in the wild.

Pros and Cons of Observational Methods

That is all I will cover as far as observations go (with the exception of think out loud, which I will discuss in a bit). The biggest advantage of observation studies is that you really get to see what new players do when playing your game. This is invaluable and makes observation studies the best methodology, in my opinion. This method can also give you unexpected insight, and reveal issues that you may have never seen without it.

However, observations are obviously quite time-consuming, especially if you are recording everything. Also, it can take quite some work to be a good observer. As I already said, a good observer has to be as neutral and non-intrusive as possible. But also they have to know what to look for, and when to ask follow-up questions.

What typically helps is first practicing, and making a list of typical behaviors you would expect to see. Then you know what to look for before you go in to a real session. Remember you want to primarily be looking for their behavior in the game, so think about the types of things they will be doing. So for example do you notice them jumping repeatedly in one place? Or running into walls?

Also, avoid coloring what you see with your own expectations. Don’t try to infer internal states; just write down what you observe, and then if you want to know what was going on in their head, ask them later.

Pros

• Provides you with objective data. You are seeing what players actually do, not what they say they would do/have done
• Players may surprise you
• When recording equipment is used, the material collected can be used for other purposes

Cons

• Time-consuming, especially if you record and go back through everything carefully
• It takes experience to be a good observer
• Going into a test environment can make people nervous or at least aware that they are being monitored, and therefore they may not produce completely natural behavior

---

Think Out Loud Protocol

An expansion of the observation method, this is where you carry out observations as detailed above, but ask the players to talk about what they are thinking as they go along. You then record this while observing what they are doing. This adds information about the internal states of the player, and gives you an idea of what assumptions they are making based on the game. Again, you may hear things that are wrong, or that you don’t like, but resist the urge to correct the player.

This method can lead to unexpected insights that wouldn’t be gained otherwise, or at the very least gives you a little more insight into how people are mentally interacting with your game. However, it does have one big downside, and that is that it is quite unnatural to talk about what you are doing and feeling as you do it. This might change how people will actually play the game and affect how much or little fun they are having.

Related to this, you may find that some people talk non-stop, whereas others hardly say a word, a finding that may reflect more the differences in people than in your game.

Pros

• Allows for the collection of objective (behavioral) data while also getting access what players are thinking and feeling
• Can result in unexpected insights

Cons

• Time-consuming
• Unnatural
• What people say is still subjective

Gameplay Metrics

Similar to, and a great addition to, observational methods are gameplay metrics. This is basically is statistics generated by the game itself. which tell you about what the players did. For example think of the statistics systems that Bungie and 343 Studios have for Halo, or the Autolog system in Need for Speed.

Such gameplay metrics give you hard, objective, statistical facts about how the game plays. The collection of this data has to be built into the game, but once it is, you have a very large data source at your fingertips. Want to know what weapons take down players most on a map? What power-ups they use? Where they tend to die? Gameplay metrics make that possible. Good metric triggers and hooks can also really compliment observations as they take some of the load off you in terms of recording the in-game details about the player’s performance.

The power of metrics: on the left is where people are standing when they make kills with a weapon and on the right is deaths by this weapon in Halo Reach. With just a basic knowledge of FPS games, you can still probably work exactly what kind of weapon this is and where the elevated and the open spaces are in the level.

However, metrics can be time-consuming to collect and you need a good number of players to really take advantage of them. Plus metrics lack subjective information about emotion and what is going on with the players — it is just a mass of data.

Also, you can end up with data overload if you don’t analyze what you collect carefully. For example, load up any heat map in Halo Reach and turn on deaths by all weapons, and compare it to kills by all weapons. The result will generally give you a little information in that the kills and deaths seem pretty well spread out over the player area, but is not particularly useful for working out finer details of the player experience.

Pros

• Objective data that can be collected remotely and discretely
• Can be used to see trends and provide data that supports other methods
• Allows for continuous data recording without interrupting the player

Cons

• Time-consuming
• No subjective feedback
• Needs large sample sizes to get meaningful data
• However, at the same time, you can get too much data

Biometrics

I will finish up by covering biometrics. I have already done a primer on this topic, so I am not going to go into too much detail here. Biometrics are basically when you record signals from the body, such as brain waves via EEG, or where people are looking via eye tracking, to try and give us a clue as to your player’s internal body states. In this way they are again like think out loud is, and like metrics can be: an add-on to observation-based testing.

Much like observation and gameplay metrics, biometric measures can be useful because they give you objective data on what is going on. They also allow for data on emotions and excitement to be collected continuously during play without having to stop and ask questions about how people are feeling.

However, the ways you can collect biometrics are currently quite invasive in that you have to wire people up, and they cost quite a bit of money and time to use, not to mention you have to be trained in how to use them — as well as how to analyze the data that you get.

Also they have problems in that you can get artifacts or fake signals in your data, and that there is no real solid agreement on what you are measuring actually means. For example, heart rate may go up in a certain part of your game, which could mean it is exciting, but it could also mean it is scary and unpleasant, or that your player is so bored they are now thinking about last night in bed with their partner, or even just that the person you are recording took a deep breath (which also increases your heart rate).

The above issues mean that you have to combine biometric data with other sources of data (observations, interviews, questionnaires) in order to work out what is going on with the physiological signals you are collecting. This has led some people to doubt its usefulness, arguing that it doesn’t add anything that you couldn’t already get but does add a bunch of complications and extra work (see an excellent discussion on this point, and on games user research in general, by Bungie’s John Hopson and Valve’s Mike Ambinder here).

Proponents of biometrics, on the other hand, suggest that the strength of biometric data is that it gives you access to emotions and reactions that the players may not know they are having, and therefore helps you look at other data sources in a way you haven’t before. Ultimately, however, it is just another tool and you will have to judge if it is appropriate for you or not.

Pros

• Gives objective quantifiable data
• Allows for continuous data recording without interrupting the player

Cons

• Invasive
• Costs a lot of time and money
• Problems with specificity, artifacts, inference and validity can make it difficult to interpret

Final Words

As I stated in the introduction these two articles are intended as rough primers, and are in no way comprehensive. I am sure that others will disagree with what I say, and perhaps others will be happy to add more or provide material of their own. However, I do hope though that this document can be of some use.

If you are interested in further reading on this topic then I can recommend checking out this excellent document from Mike Ambinder at Valve and watching this discussion between Mike and John Hopson from Bungie. As mentioned in the last article, you may also consider checking out the IGDA Special Interest Group for Games User Research on LinkedIn.

Furthermore a wiki-based games user research primer is in the works. This primer will arise out of a workshop at the CHI 2012 conference and is planned to be an evolving wiki based site that can provide the community with up to date information on game research methodologies. So watch that space.

Finally, please remember that the above methods are here you help you develop your games. The data they give can be rich and interesting, but it should not be domineering. In the end, it is your creative vision that guides the game; the data that can be collected via games user research is there to help you, NOT limit you.

Written by: Ben Lewis-Evans, via Gamasutra
Posted by: Situated Research