Iterative Research. Small Bites.


Minimum Viable Research? Yes. Matthew waxes on the topic of moving forward with research in an iterative manner. Building up to a level of confidence, rather than committing to interviewing/testing with a set number of participants.


Matthew: You're not hearing your voice, are you?

Matt: No, sounds normal.

Matthew: I like how I said that and you're really quiet for a second. 

Matt: Am I? Hold on, hold on, hold on. Nope, nope. Yeah, what should we talk about?

Matthew: Well, let me tell you, I've got thoughts.

Matt: Yeah, what are your thoughts?

Matthew: Well, I just have one thought.

Matt: What's your thought?

Matthew: I'm gonna open the Airtable to remind me of my thought. No, I did have something that was rolling around in my brain this morning.

Matt: Awesome. I'm on the topics tab.

Matthew: Oh, wait, I'm not even in there yet, come on.

Matt: I'm still logged in from last time we talked.

Matthew: Something about this project with the Google thing. No. My phone was like, oh, the Google thing? Here, let me search for what thing means. Got it, got it, got it. That was a rough slog. So, it wasn't the Google thing, it was the bank thing. So, they wanna do a bunch of research, right? And it's all generative and they want to talk to people in multiple locations and I propose to them taking an iterative approach, as I usually do. Let's start with, my proposal said six people. We'll stop, or we'll pause, and we'll be like, is this enough, do we wanna keep going, do we wanna change tacks? And what I found is, when I proposed this iterative approach, even though, when we all sit down and we're not trying to get anything done, anybody I ever talk to about this is like, "Oh, that's a really great way of approaching it." But every single time that I propose that, I get pushback because they're like, "Well, six people won't be enough." And I'm like, "Well, I know that six people won't be enough, "but is 60 enough?" "Oh, no, 60 is too much." Well, what's enough? And then, they're like, "Oh, okay, well, "an iterative approach makes sense." But, I think in some respects, you can still make the claim that it's research, but with usability testing, with Mayo, we've done like, oh, well, we have to talk to 20 people or whatever. Instead of taking an iterative approach, we committed to doing those 20 people. Did we need to do that?

Matt: I'm trying to think of a scenario for a project where I, or we, started super small, like six, did the six. I guess two examples, one where that was enough and one where we said, "Okay, yeah, let's do six more "to get more data." 'cause Mayo, I think we haven't started at that small, at six, I think we're doing like nine or 12.

Matthew: Yeah, we always did bigger numbers and for the most part, once we got past the first few, it was the same conversation over and over again, which, on the one hand, is like, okay, well this is really validating. Could we have stopped at three? In some respects, you don't know you could stop at three until you do the 12, but I think, at the same time, both you and I can tell or get a really good sense that, you get that third, fourth person, and either we're not asking the right questions, but we're trying to follow some rigor to this so that we're having everybody go through the same thing, not to jump back on the how many users is enough to test with. I think my pitch is, sometimes, three is enough, or what I put in my proposal was, we'll do six. Talking about the iterative approach, it's like, I don't wanna sell you on doing 60 interviews, but if we altogether determine that 60 interviews are what we need to build a high-level of confidence that X, Y, and Z are the right things to do, then that's what we're going to do. It was something was on my mind that I felt like could bridge beyond both sides of the perspective of the practitioner and the people who would be hiring this to talk about how much effort do we have to put in, especially doing generative stuff, how much effort do we have to put in to get out value? And the answer is, there is no number that is consistent go-to and that's why we do three, we do six, we stop, we assess, we do more, or we do more, but on different stuff, or we stop, or... And then, that also is related to my whole wanting to test the test or test the research by running it through with people who are not the have a pulse people.

Matt: Yeah, I like the tactic of not coming out. I mean, there is no right number, A. I kinda feel like three, three is never enough, unless it's a super, super focused idea kinda thing?

Matthew: Yeah, if you're testing a tiny feature or something.

Matt: Then maybe, I'd say.

Matthew: Got one user type who's ever gonna use it.

Matt: Even then, I don't think I would feel right recommending only three. If that's the case, it would be a really quick test. It's a really small feature. You could pump that to six without a lot of expense or cost or time. The underlying intention is that we test it until we're confident and it might be six, it might be 60, it might be a hundred, but if it's a hundred, then we're doing something wrong.

Matthew: Or it's a thousand, because we wanna get statistically significant.

Matt: Right, which neither of us do, but yeah.

Matthew: Anyway, so, that was what was rattling around in my head this morning.

Matt: I like it. Like you, I've pitched this. We pitched it with Upwork votes and the research that I have done, it was just, here's the number, we're gonna talk to 15 people or whatever it is, and that's what we're gonna do without a sense of re-upping. Now, obviously, if we did, I'm gonna just say 15 'cause it was around that, and we were not able to draw conclusions, then I could see us having said, "Okay, let's do 10 more or something "with a different design." But we didn't go into it with that expectation and 15, I wouldn't say that's an iterative approach, that was a larger population.

Matthew: One of the things of the iterative approach is, if you do, not you, but if one does 15 interviews and gets to the point where it's like, we can draw no conclusions from this, how do you look, as the researcher? You've done 15, you've charged someone $30,000 or whatever, and you know, you asked good questions, you had people who were appropriate to the profile, but you still aren't very confident with what direction to take. To me, there's a risk there of looking really incompetent.

Matt: Right, which, I'm glad to say, I've never been in that situation. Each time, I've always gotten conclusions. Yeah, it's a great point. On the flip side, what if you say we're gonna do 15 and after five, you're hearing the same thing over and over again? Do you cut it short and refund their money?

Matthew: Right, right, yeah.

Matt: The answer's no, but anyway. That's a trick question.

Matthew: That was a trick question. Yeah, you do stop and you don't refund the money.

Matt: Right. You have a nice dinner because you saved time.

Matthew: Right, right. Let me bring to your attention where I said up to 15 people.

Related Post