The Interview Methodology — Centering the ‘Human’ in ‘Human-Computer Interaction’

I’ve been learning how to research at Microsoft Research, India for a year and a half. In this time, my primary and favoured methodology for qualitative human-computer interaction research has been semi-structured interviews. I like interviewing people about how they use technology because I get to witness the humanness in ‘human-computer interaction’ — I become privy to the experiences, feelings, and inner dialogues that define, challenge, and shape the future of technology. As a social media researcher interested in platform architectures, I’ve interviewed 40+ influencers, journalists, and disabled advocates about their use of Twitter and Instagram (and to a lesser extent, Facebook and LinkedIn). In the process, I’ve iterated on my approach several times over. While I have by no means arrived at a conclusion about what constitutes a great interview, or perfected the methodology, I’ve learnt interesting lessons which I wish to outline in the three steps mentioned below:

1. Understanding what you’re investigating (research question, themes, sub-questions)
2. Building the questionnaire (guidebook approach, iteration, non-assumption)
3. The interview (rapport, information, engagement, direction, empathy, closing)

I would like to note here that these aren’t individual and isolated learnings — they have emerged from collaboration and conversations with fellows, interns, my mentors at MSRI and my participants over multiple studies. This is my own unique approach, so if you are looking for tips, you may wish to pick and choose elements that best suit you.

Understanding what you’re investigating

I first try to narrow down the research question, major themes, and sub-questions which will underlie the interview questionnaire.

Research questions

This is not a section about research questions in general, instead it is about those that will ultimately lead to a semi-structured interview. I’ve found that one does not ‘pick’ or ‘choose’ a research question as much as one carefully ‘constructs’ one. An RQ is ideally narrow enough that it allows you to obtain in-depth qualitative findings, and broad enough that you don’t hole yourself into a kind of over-specificity which would force you to overlook important, even peripheral but valuable, aspects of what is being investigated.

Say you have a population in mind and a phenomenon or experience that you want to study. Let’s take for example: female Indian social media users and online trolling. A broad RQ is: how do female Indian social media users experience online trolling? Language and intent start to become important in the first step of construction. If you were to investigate WHY female Indian social media users experience online trolling rather than HOW, you would perhaps include perpetrators and platform administrators in your list of interview participants. The HOW keyword, on the other hand, outlines the intent to recruit a specific population and allows you to shape the focus of your investigation — which in this case could be the method, degree, and impact of trolling, rather than the intent behind it.

Themes

Once you’re happy with your research question, it is a good time to dive into prior work to identify themes relevant to your RQ that have already been studied. Then, you can start delineating various aspects within your RQ that are (a) of interest to you, and (b) at the same time under-explored in existing scholarship. Here are some things to consider: Do you want to know the ways in which your phenomenon of choice occurs? Are you investigating how it affects the user? Do you think that the academic community would benefit from novel insights on how it is managed by the user? These themes will help you decide a specific direction of research. Of course, you may pursue multiple and overlapping themes, which is fine. This exercise simply brings clarity, both for the purpose of the interview and for the paper you eventually wish to write.

Sub-research questions

Once you have identified major themes, you can move on to sub-research questions. This is a space to ask as many questions related to each theme as you can, and to explore your curiosities. You might want to ask yourself: what is it that my participants can tell me, with the most confidence, authority, expertise, and self-knowledge? Is there a kind of self-reflection I can guide them towards, which I will be able to wield into findings, which will be beneficial to my field?

For example, under ‘ways in which’, an interview can help you understand the frequency of trolling. You may find out who trolls, how trolling occurs (e.g., individually or collectively), the type of trolling (such as slurs or threats), where it occurs — different platforms, DMs, comments, emails, and such, and when it occurs (for example, after the user posts specific kinds of content). These insights, bundled into who-how-where-when, expand on the methods of attack. Under affect, an interview can help you learn about what happens during and after trolling. How does it make the user feel in the moment? What about after some time has passed? What does this experience mean to her, and how does it affect how she views the world and her own self? Then, under management, you might be interested in understanding: How does a trolled user respond? Why does she respond the way she responds? How has it impacted her online behavior? What do these changes look like? The interview can also help you understand the larger context of the specific thing you are studying. While some findings may be inferred by you as the researcher, you can assume that she is the authority on her experience as well as the culture she is immersed in, and ask: does she get trolled more than her male counterparts? Are platform privacy features adequate? Why does she think her trolling occurs? And so on.

Once you write these down, you will have a good idea of what you want your questionnaire to look like — because you now know what you’re looking for. This exercise, ideally done collaboratively with a research team and/or mentor, will help you understand the vital question: What is it about this particular phenomenon that I want to know? In other words, what am I really researching?

Building a questionnaire

At this point, you can pick out specific sub-themes that are of the greatest relevance to your RQ, and begin directly translating these sub-themes into actual questions directed at the interview participant. You will want to make sure that you have at least 2–5 questions under each theme. The questionnaire, at this point, may look like this**:

Theme 1
Sub question 1
Sub question 2

Theme 2
Sub question 1
Sub question 2

Theme 3
Sub question 1
Sub question 2

There are three approaches I suggest here: the guidebook approach, iteration, and non-assumption.

Guidebook approach

Since these are semi-structured interviews, you can view the questionnaire not as a strict checklist on which each item must necessarily be ticked, but instead as a guiding document for themes you want to address in the interview. This means that if you can’t fit all sub-questions into an interview you are conducting, that is okay — you can instead make sure that you ask at least one or some questions from each theme. However, for easy referencing, all of your relevant sub-questions are always present in front of you during the interview. You may find in the process of research that it is actually counterproductive to force each sub-question into every interview. All participants have a range of unique experiences that may be more relevant to one theme than the other. Centering the ‘human’ means allowing them to expand upon the experiences that are most important and relevant to them, and probing further into those very experiences. Your questionnaire is hence a guidebook, not a to-do. This approach also helps you remain engaged in the interview, so that you can choose each question and construct each in-the-moment and not-in-the-questionnaire follow-up according to its relevance.

Iteration

This questionnaire is a document upon which you can, and should, continuously iterate. If a participant has particularly interesting insights on one theme, you can focus a majority of the interview on that theme and come up with newer sub-questions in the process of the conversation. If you believe that this is an experience that more members of the population may have, and if it may make your findings more comprehensive, you can include the newer sub-questions in your questionnaire for your next interviews. On the other hand, you can also start eliminating sub-questions that participants tend to address in answers to other sub-questions. If you have adequate findings within one theme, you can choose to simply focus on other themes in successive interviews. Once you’ve conducted half of your interviews, you can start thinking about whether you want to zoom in on only one or two themes for the rest of your interviews, expand your questionnaire, or stop asking questions that have reached a point of saturation (i.e. participants keep offering the same answer). This is important because it means that you are finding, investigating, and exploring, rather than going in with a biased understanding of what you want or hope to find and write in your research paper. In other words, we are not probing participants until the perfect answer arrives — we are going in without the assumption that such a perfect answer or finding exists.

Once you build the first draft of your questionnaire, it is generally a good idea to rely on your research team to help you refine it. The coolest thing about research is that at every step, collaboration really is the best approach — your team will bring newer perspectives that may lead to the development or modification of sub-questions, and even themes. In this way, you can approach your interview questionnaire as a living document in a systematic manner. This form of structuring allows you to incorporate ideas that emerge in the process of interviewing, which reduces bias, and allows for more robust — and importantly, novel — findings.

Non-assumptions

I also suggest trying, to whatever extent is possible, not to make assumptions rooted in personal biases and perceptions. For example, there is a general assumption that disabled social media users face tremendous online harassment. However, when I interviewed disabled users, a majority told me that they do not face much outright harassment — but that ableism creeps into the online experience in different ways such as pity and pedestalization. Had I let my assumptions shape my questionnaire and jumped to, ‘what kind of slurs have been hurled at you online?’ two things would have happened: first, participants would dig into their memories and offer information about isolated incidents that could be deemed useful to this study, in which case I would have a skewed understanding of the frequency and qualitative nature of online ableism, and second, my research would leave an important aspect of the disabled online experience unaddressed. When I interviewed Instagram influencers, I asked them very many times about online shaming, and heard so many stories of Twitter engagement being majorly hostile that I was sure quantitative data would back this finding up. However, Twitter data revealed that the frequency of positive engagements with influencers is actually much higher than negative engagements — it’s just that negative tweets are afforded more visibility through retweets. Had I not assumed that asking questions about positive engagements is largely irrelevant to my study, I would have found this out much sooner.

The interview

As an interviewer, everything you do and say — the tone of your voice, the phrasing of your questions, your facial expressions (even over video), and your new relationship with the participant — will have an effect on the outcome of the interview. Making the participant feel comfortable is the key to an honest interview, and this is achieved by the researcher being comfortable, confident, and empathetic. There are a few ways to go about this.

Rapport

My go-to strategy in the first 5 minutes is to have a non-recorded chat with the participant. I try to create an atmosphere of ease by making a joke, telling them that I’ve seen (or stalked) their work on social media, and/or bringing up easy, familiar topics — mutual interests, fun facts about the city they live in, or even the miserable weather. In other words, I try to show them a little bit of my own self. This approach, accompanied by a tone that indicates levity, conveys a couple of things: one, that the interview is a candid conversation rather than a strict and formal interrogation; two, that I have time for the participant and care about their experiences beyond their immediate usefulness to the research project; three, that I’m striving to be a ‘safe’ person who is open and approachable — that I want to share a little bit of my own vulnerability to show them that I will be non-judgemental when they share theirs. This, which is perhaps my favorite method of centering humanness, reminds me that inculcating values of empathy, safety, care, and community in conversations with users is an important precedent to inculcating the same values in technologies that will ultimately serve these users.

Information

Post the initial chat, I find that offering as much information about the study as possible builds trust. Explain to them the purpose of your research, why it is personally important to you, and what you hope the study will accomplish — whether it is new interventions or novel perspectives that will benefit the larger scientific community. Let them know, at each step, what you are about to do — whether it is beginning a recording/transcription, if there are any potential triggers in the work, and the fact that they are free to candidly tell you if they wish to skip any questions (all of this, even if it is outlined in a consent form). If needed, check in with them in the middle of the interview — can they hear you, are they okay with the pace, is this approach working for them? In essence, keep them in the loop — this suggests to the participant that their time is respected, and that their contribution is valuable (both of which are very true). That, in turn, may motivate them to provide answers that are adequately lengthy and self-reflective.

Engagement

Next, when I am asking questions, I make a genuine attempt to be highly engaged in the conversation. When a participant gives me an answer, I write down and then repeat the key points of the answer back to them before moving on to the next question. I tell them when I find an answer interesting or surprising, I tell them when I’ve learnt something new, I tell them when I am curious about the answer to my next question, and I tell them why. When the participant mentions a particular phenomena in their answer, I try to maintain a conversational nature by telling them something I’ve read about that phenomena. In these small but meaningful ways, I’m once again conveying to the participant: I am interested in what you are saying, and I care about your experiences beyond their essential utility. Humans, when they talk about their own selves, are the kind of creatures that respond to disinterest with disinterest, and curiosity with generosity. Try to let your passion for your research drive your curiosity, and generosity will follow.

Direction

There are three notes I would like to share on how to give an interview direction. First, if a participant says: “I’ve experienced/felt XYZ at times” ask about specific instances and examples — the more specific, the better. ‘Participants experience/feel XYZ at times’ is an analysis that you as a researcher have to provide. The instance/example is the finding, the paper quote, the basis of the analysis. Because of this, it is essential to ask, and ask again for each vague comment. Second, sometimes, participants do not have clarity on the fact that you’re asking for their own personal experiences. This is particularly true for senior professionals. I have heard many versions of the following statements: “I don’t have the data to comment on this”, or “I’m not an expert, so I can’t say”. I have also heard participants comment on and analyze the broader culture of things and other people’s experiences. It is good to note these comments. However, again, the analysis is your job — and you may have to share with them that you are more interested in their intimate experiences. Further, sometimes, participants can wander and end up giving a long-winding answer — the kind that will not contribute to your findings, but is clearly emotional and important to them. I try to not interrupt them the first time it happens — because I may learn something new, I want to hold space for them, and I don’t want them to feel invalidated. If it happens again, I gently guide the conversation back to the research topic, using sentences such as: “I want to pivot for a moment”, or “There is something else I’m also curious about”.

Empathy

Researchers are often critics, and to critique a phenomena, it is not unlikely that you will investigate a pathology at some point. This means that your participants may touch upon sensitive topics that can be difficult to explain — and that your curiosity may center around negativity. In my experience, the best way to navigate this is by not being awkward — because awkwardness doesn’t make for a very good safe space. Instead, allow yourself to be human. Be respectful in your investigation, give them the option to withdraw from a specific question, and probe with the skill of gentleness. Assess whether a follow up question to a sensitive incident is appropriate (if done with adequate respect, it usually is), and go for it.

Closing

Before ending the interview, I look back at my questionnaire and check whether there are any major sub-questions or themes I want to address. Once I’m satisfied with the data I’ve collected, I ask the participant a final question: “Is there anything relating to your experience I may have missed out on, that you want us as researchers studying XYZ to know?”. Very often, participants are waiting for the right question to share an experience of significant importance and relevance (either to them, or to the research, or both) — and very often, that question never comes. I believe it is essential to give them an opportunity to share such thoughts, because they give us as researchers new perspectives, and often entirely new dimensions, relating to our research question. This also gives participants an opportunity to zoom out and provide a broader, meta-commentary on their answers, experiences, and the culture they are immersed in. These are some of my favorite moments.

When you end the interview, end the recording/transcription, thank the participant for their time, address any remaining formalities, and reiterate what you plan to do with the data. I also like to take this time — away from the surveillant gaze of automated transcription — to have a final, candid 2-minute chat with the participants. I tell them what I liked about the interview, how their answers shaped my thinking, and once again, I engage in some light jokes/banter. Leaving on a positive note doesn’t just mean that the participant will be available for a follow-up. It means that we as researchers have done our job of articulating — through our actions, not words — that every conversation and confession surrounding the usage of technology is incredibly valuable to the HCI community, and that we are committed to handling their experiences — whether in the form of data, or a memory — with care.

Thanks for reading. I am thinking of writing more detailed pieces on my lessons surrounding qualitative HCI research, such as:

• Exploring theoretical lenses from sociology in HCI
• Interviewing highly-visible populations/celebrities — privacy and sensitivity
• My initial mistakes in writing related works sections
• What it means to use a “quote” in an HCI research paper

…and more, as I learn more. If you have any other ideas, let me know!