When should you do qualitative testing?

When most product teams think about qualitative usability testing it’s often last-mile type stuff. Too often it’s used in an attempt to reinforce that the project you just spent months to deploy was indeed worth the effort. This is wrong.

If there was ever such a thing as vanity testing that would be it. True testing should be applied in two main forms: generative, and iterative. Generative being the design research type stuff that you do before you even start, aka the front of the double diamond (see below). Iterative testing is the kind of testing you *should* do during the development, delivery and eventually, throughout every product development cycle.

In reality most teams don’t do this. If you’re already doing both generative and regular frequency iterative testing give yourselves a pat on the back and go have a celebratory beverage because you’re in the top 10%.

For most teams, reality bursts through the door like the kool-aid man at some point during the project. Whether it’s budget constraints, timeline constraints, lack of buy-in from leadership – for one reason or another you end up bolting on testing at the end of the project in a mad scramble just to say that you did it.

And yet, as UX designers we all know the importance and potential impact of testing. And whilst you know you should be doing more, you probably feel like your hands are tied by variation of the constraints I mentioned above.

In this article I’m going to talk about one potential solution to this problem, and how you could implement it into your current workflow without having to completely overhaul things.

Recently I attended a UX Design Research conference in Melbourne, and one of the speakers talked about doing something called ‘customer Tuesdays’ where every Tuesday they’d invite a few of their customers in and talked to them about the product. Seemed like a straightforward concept but the more I thought about it the more it became clear to me that this should have been higher up on the list of potential solutions to the testing problem.

For starters, designers who work with development teams are probably already working in a agile (or some version of agile) sprint model. QA and testing is already a part of that process so it seems logical enough to add some additional data points into the mix. But maybe this was also the solution to so many of the issues that crop up around time constraints.

Often the complications that arise around time is caused by the ad-hoc nature of how teams currently implement testing. Since traditionally it’s difficult to predict when a product or feature will be ‘ready’ for a customer to test, it doesn’t get scheduled until the last minute. But by baking it into a sprint and upping frequency with something like Customer Tuesdays, it let’s you get away with testing smaller pieces and making smaller, incremental improvements to the product.

While you still may want to keep doing larger scale ad hoc product testing rounds and bring in higher level stakeholders, doing bite sized testing means you get a lot of the benefits without having to worry about some of the problems with scheduling and /or investing huge amounts of time into prep.

But what about cost? If you’re working on a b2b type product, or you’re lucky enough to have super passionate customers you may be able to get away with having to make chunky incentive payouts. For them, a coffee and the opportunity to be part of the product development process may just be enough. For those less fortunate, there are 2 potential solutions.

  1. Budget it in as a development / QA cost

One of the common challenges to qualitative testing is not having enough room in the budget. Often the gap between the research and insights to actionable features is just too big. And that’s probably true if your insights are big. But by cutting down the size and scale of what it is you’re actually testing, much of the kind of feedback you’re getting becomes easily actionable fixes or enhancements. Rather than huge transformative design research programs, you’re doing QA but directly with the customer. This makes it a lot easier to convince stakeholders to make room for in the budget since it’s something that most product teams already do (and in some cases is a legal requirement to do).

2. Efficiency

Whether you’re looking to talk to the same customers each time, or get a new customer in the door each week, having a process set up means you can run it much more efficiently. You can run off a pre-defined set of questions and have templates for everything. You could even have a permanent setup in the office (computer, video recording, screen recording etc) where you run the tests. Depending on how you prefer to do the recruitment, you could schedule an invite to automatically go out each week. All these little things add up to a huge increase in efficiency and lets you cut down on the operational costs around usability testing by a non-trivial amount.

Implementing something like ‘Customer Tuesdays’ into your current workflow begins by first looking at the product team’s process. If your team already does agile sprints it makes sense to try and pick a point along the sprint where it makes the most sense to put what you’ve built in front of the customer. Otherwise you’ll need to figure out with your team what time and day makes the most sense to add this in but in general you want to be doing it just before you get stuck into the bulk of your weeks tasks (which is why Tuesday works well for most teams).

Change is hard for any organisation, even for small teams it can be challenging to take on new processes or workflows, so it might be worthwhile thinking about how to ease the transition process. Think about how other products and companies coerce you into developing high-frequency habits through “30 day trials” or “6 week challenges” or designing ways of increasing buy in through including other people into the experience.

Rather than attempt to introduce ‘Customer Tuesdays’ as something new that’s going to be implemented forever, try coming in from a “let’s pilot this for 4 weeks” angle and see if it sticks. It’s a lot easier to get buy in from people if they know there’s a way out.