Remote Research & Testing


What are the benefits of doing remote research and testing? Well, they are numerous, that’s for sure!


Matthew: Although, honestly I kind of want to be one of those people, that wears sunglasses all the time.

Matt: I don't think you do, because I-- I've met those people

Matt: So Matthew, I think we were going to talk about remote usability testing.

Matthew: Or just, remote everything, right?

Matt: Yep, alright. Bring it on.

Matthew: Not all things, but like I think I do want to talk about testing, but also I think a lot of the things that we plan to talk about can be applied to just remote research in general, like we're just interviewing people remotely. That kind of thing.

Matt: Sure, sure, sure.

Matthew: Um, Literally mean everything?

Matt: That's where I thought you were going, and I was curious to see where you led us, in this conversation. So where would like to start?

Matthew: Well, I think I would like, as usual, and you're not going to be surprised by me saying this, I think I would like to start by setting a little context. For the most part, I want to focus on moderated, where the researcher is an active participant in the testing of the research. Go.

Matt: End, scene.

Matthew: End scene. Alright, goodnight everybody.

Matt: So we know Matthew prefers moderated, which, yeah I do too. I, am I inferring that correctly?

Matthew: You, you infer, and I imply. Uh, the same concept.

Matt: So, so let's talk about moderated remote usability testing.

Matthew: Okay.

Matt: Let's see how many more words we can jumble in this subject.

Matthew: I know

Matthew: All these adjectives

Matt: That's makes it much more interesting.

Matthew: Moderated remote generative usability.

Matt: Yes.

Matt: So lets start at the beginning.

Matthew: Okay.

Matt: If, if you're working one of these projects and this method that you've chosen to, get to your study,

Matthew: Right.

Matt: Where do we start?

Matthew: Well, and you're gonna kick me again, but like I think I'd like to briefly talk about why you might choose to do this.

Matt: Okay.

Matthew: And,

Matt: This is the beginning.

Matthew: It's still the beginning.

Matt: Right.

Matthew: But, why you might choose to do remote. People who fit the profile, who are the potential customers, or the potential users, may be in places that are either difficult for you to get to in a timely manner, or difficult to get to because of budgetary concerns, and so,

Matt: Or flat out undesirable.

Matthew: Right, yeah you just don't want to travel

Matt: Right.

Matthew: For whatever reason, I mean

Matt: Right.

Matthew: You know, I--I think it usually ends up being a time and budget thing, and I think that that's why doing a moderated remote session can be really helpful.

Matt: Right.

Matt: But what are some the, the potential risks of doing it remote, in suppose to in person.

Matthew: Well, obviously if you're not in situ, as the professionals say, no they don't. No one say that, no one say that.

Matt: You're fired.

Matthew: If you're not in context, on site, you miss out on a higher level of fidelity of understanding. You might, you might be looking at the-- at some slice of interaction or experience, but to be able to truly understand that; it is sometimes better to be in context, and--and watch it happening so you could say, oh, I can see how their physical environment is impacting their ability to do this. And you don't necessarily get that on remote, uh, remote testing or remote interviews.

Matt: You quite literally have a very small window.

Matthew: Right, right.

Matt: Into their life or their work.

Matthew: And you definitely have those moments, you know, where I've had those moments of hearing, hey can I pause, because my baby's crying, and I've just got to go, it'll take me a few minutes to get them back to sleep. So one, the answer is, of course, of course. Go take care of that. And two, oh not literally exactly like this, but this is what life is going to be like for this person, and people like this person. We can't imagine the users of Product A are gonna be like, alright, I'm gonna sit down, and I am going to focus on completing this task, and nothing's going to stop me at all. Because, realistically, that never happens. And so, we do want to see, what is that experience like when life interrupts? How easy it is for them to get back on track. And I think you see that more when you're in context, but you do still get some of it with remote testing. It's just, I wouldn't say its a low fidelity, it's just a bit lower. So it's often you still get to see some of their environment and sometimes they even walk you around and show you things. So you get some of it, but it's just not as robust.

Matt: So safe to say, that it's not our preference to do remote. In many cases, we prefer to be in context, when appropriate, when available.

Matthew: Yeah, I would say my decision tree is, I wanna talk to the people who are most represent who we're trying to target, and then I say, what is the easiest way to get to those people? What is the, and--and then pair that with the, what is going to give us the most, the best outcome?

Matt: Right, that's where I was heading.

Matthew: And try and find that balance.

Matt: Yep

Matthew: Because it could be that a remote is enough, and it leads you down a path that is positive for the team, positive for the product or service.

Matt: I think one of the reasons we want to focus on remote, is cause, it does have, I don't wanna saay stigma, but the impression I hear from other people is that it's a little bit scary, it's an unknown. When I talk to people and tell them I've done remote usability testing, I get a lot of, "Wow, can you, tell me more about that. How'd you manage that? What's-- What's the logistical set up that you use?" Um, I guess it's just a little bit more of an unknown quantity, so

Matthew: I understand that, especially if you don't have any or very little experience doing it, How you could look at it as, wow is that really going to work? But ultimately if you could get a set up like what we're doing right here, where we can have a conversation and then if--if we were doing a usability test, we--we would have this conversation where we get to know each other a little bit and I'd probably have some pre we're getting going on testing questions to ask you, and--and then I say, "alright, there's a little green box, you know, if you wiggle your mouse, this, I always have to say this, so wiggle your cursor over this--this window we're talking to each other, there's a green share button. Click that, and then share your desktop, and then we'll get started on the usability test. And then you still get to watch them, their facial reactions, you get to obviously see what they're doing on their screen. So, I think, it's--it's still valuable stuff. Its kind of like in the old school days, or the school days now, if you're still working in a lab, where you've got an observer's room, except you're in the observer's room, but you can still communicate directly with the person doing the test. So, there's--there's a level of displacement there, but I don't think it--it ends up being something that you can work around.

Matt: So, once you've selected your participants, or your--your segments let's say, and you've made the decision to do remote, what's next for you as you build out your project?

Matthew: That's a good question. It depends.

Matt: (laughs)

Matthew: Just kidding. It depends, but I have an answer.

Matt: Okay.

Matthew: It's--it's kind of the, instead of it depends, let's refer to the if then table. If time is of the essence, and depending on the extent to which the, or the robustness or the availability of the prototype, if you're testing out a concept, if you're testing out robust software that exist in production, those are all going to be decision points that you're going to have to take into consideration for how you want to plan the test, and what technology is gonna be able for you to use the test or to--to carry out the test, that's what I mean. So if you're doing, you know, we want to test the unboxing of our physical product, you're going to have to mail the participant this product. You're gonna have to wait and you know, instructions, don't open this yet. And then they have to let you know that they received the package, and then you set up a call like this, where ideally they can set up their camera or whatever, if they have one. That you could both see the box and their face, and they have to go through the unboxing. It's not something that you can always say, something like what we're doing now and say, here click this URL and we'll start the test. The it depends part is, you know, what are you testing? What level of fidelity is that thing that you're testing? And what do you hope are some of the outcomes that you achieve from this? Are you just looking for, show me where the bugs are? Or are you looking for, will this resinate with people?

Matt: So let's talk about the people, you mentioned finding the right participants.

Matthew: Right.

Matt: Early on. So do you do anything different when it's a remote test versus an in-person test, as far as recruiting?

Matthew: Not, I would say not for the most part,

Matt: No? Would you be doing the recruiting yourself? Or do you use a third party vendor?

Matthew: I do almost all my recruiting myself.

Matt: Okay.

Matthew: The times where I find myself reaching out to a third party, a vendor recruiter, is when the profile is so specific. Most recently, people sixty-four and a half to seventy, who are just about to enter Medicare or have only been in it a couple of years. That was a rough profile to try and recruit for. And I reached out to two agencies who specialized in recruiting, and both of them were like, no we can't help you. And that's too specific. But eventually we found the people, but it took a long time. There really isn't too much of a difference in how I approach it for in-person versus non-in-person, other than where I go to ask. So if it's here in Portland, I might use Craig's List. If it's very specifically in other cities, I might use the Craig's List in those other cities. If it's, we just want people, whose primary language is English, who are between the ages thirty and fifty, and have some established income level, and live in the United States, I just start asking on Twitter or Linkedin, because knows somebody whose aunt knows someone, and I've had good success with finding people that way when it's very broad. So for that I don't see that big of a difference.

Matt: What if the difference, I was going to bring up is, the difference when you're going, when you're not doing it remote, when you're in person, is just the logistics of the travel, um if you're going out of your city.

Matthew: Oh I see what you mean.

Matt: You're in Portland, let's say you're going to San Francisco to do a study, so it's--it's different. Potentially more questions. Where are you located? Are you available? You know, between these times for, are you comfortable with people coming to your house or business?

Matthew: Right, yeah, I see what you mean yeah. Yeah but I feel like those, I classify those as technical constraints.

Matt: Mhmm, for sure.

Matthew: And so far, I would say I don't think I've ever run into a situation where we've been going to someone's home, or someone's place of work, and had anyone say, no I'm not comfortable with that. Because, you're sort of getting the population that self selects to be part of this kind of thing, like they enjoy doing this kind of stuff. The may want their $150 gift card, but

Matt: I wish I could share that. I've run into people who have a change of mind

Matthew: Oh really?

Matt: Oh yeah, where they've offered to allow research in their home or business, and day of, day prior I say, I'm not comfortable with this.

Matthew: Okay.

Matt: Um, or things come and was maybe a little questionable whether they were telling the truth, or they were making up an excuse.

Matthew: Right. But, then you just drop them.

Matthew: Fun fact, it's much easier and less expensive to recover from a missed session, if you're doing a remote, then if you're doing it in context or in a lab.

Matt: Oh, exactly that's what you do. It's just, sheer point, I have run into that situation, where people are more comfortable talking on the phone or a video chat.

Matthew: Yeah--yeah, I see what you mean.

Matt: Because they were like, they feel safer, or just privacy is more intact, I don't know, but there is a different level of involvement and commitment from someone to just have a video chat, versus letting a team, or, you know, two, three, four people coming to them.

Matthew: Right, right. Yeah, because it is a weird experience to have at least one person come in who's gonna be doing all the talking. Someone who's probably not gonna talk very much, and will just be shoving a camera in your face the whole time. Not literally, but,

Matt: It depends on the study.

Matthew: Depends on the study.

Matthew: Do you like cameras coming at your face like this?

Matt: Right, right. Please, answer on the sniker skin, okay.

Matthew: So, yeah--yeah, I see what you mean about the differences, but I find that for most part, they're--they're workable, you know, and again, it goes down that path of, it depends, life if it's this, then it's this, oh then that means we need to do this. I think ultimately we need to come up with one of those decision trees, that will probably be really convoluted.

Matt: You touched on one of our tenants, of every problem has a solution.

Matthew: That's right. I just may not like it.

Matt: You may not like it. But there's always a solution, you just have to find it.

Matthew: Yeah. And I think that's the big thing, research in general, testing in general, is remaining positive, as positive as possible, and remaining flexible and resilient. There's gonna be times where things just do not go well, and you just need to shut it down. Not the whole study, but that individual's, and you need to, go, if that's your thing, go grab a beer, and go, "That was terrible."

Matt: That's right. Alight, do another one.

Matthew: Tomorrow! But, we're gonna do better, we can do better.

Matt: That's right.

Matthew: It happens, it's rare

Matt: It is rare, but it does happen. And--and I think it pays to be somewhat mentally prepared for the possibility of that happening,

Matthew: Mhm Hm

Matt: Um, because, especially if it happens when you're in the middle of the interview,

Matthew: Right, you have to have a plan in place.

Related Post

What Is UX?What Is UX?

S01E01 Yep, we went there. Well … a little. We talk about pay rates, responsibilities, and career paths for people in “UX.” You can watch the episode or listen to it. Transcript Matthew: Set a reminder for a month from now for a check-in. Matt: Yeah. Matthew: And be like,