How to improve the UX research process in b2b

Respondent recruitment, team engagement, and research insights management

Maks Korolev

Follow

Published in

Acronis Design

12 min readDec 29, 2020

--

In this article I’d like to share three techniques that we use to increase the effectiveness of UX research at Acronis:

1. Recruiting and finding respondents for b2b product testing

2. Involving a team in the research process

3. Storing and managing research results

Some of them are widely known, others are hardly used. I hope by the end of this article you will have some new ideas on how to improve the UX research process in your own company, especially if you are working on business products.

1. Recruiting and finding respondents for b2b product testing and research

Recruitment via account and sales teams for a rapid research session

UX researchers in b2b often face difficulties when searching for respondents — it’s not easy to find specialists for interviews, access to them is complicated. Recruitment turns out to be very expensive and labor-intensive. At the same time, almost every b2b company already has established processes of communication with customers through other channels — through account managers, sales managers, support team, and so on. The researcher can use these channels to access clients for their purposes.

This means that for small research and interviews we can combine our sessions with those of account managers. It looks simple: before the meeting, the account manager asks the clients if they can spare an extra 20 minutes to discuss some topics with the product team (a new solution or personal experience with the product). Clients often agree to it. It is important to clarify here — we often encounter a situation where we need to check a small product hypothesis, test a short scenario, or learn small details about customers (process typical for agile development). Hour-long sessions for such research are not necessary, and a small interview on the topic within an account manager session is quite appropriate.

Such micro-sessions are important to moderate clearly — it’s best to convey to the client at the beginning that while their entire experience is important, in the next 20 minutes you want to focus on a very specific aspect of the user experience or a specific scenario, and discuss all other issues outside of this meeting. To do this, it’s advisable to look for customers who don’t have active problems at the moment or to arrange for two separate sessions: with you and with the support person.

Of course, such interviews do not replace full-fledged research, but they allow you to quickly check small hypotheses and clarify the necessary data. Sometimes this allows you not to waste time on recruiting at all and to clarify the necessary information within a couple of days.

Recruitment through the support team

If it’s important for us to know about a specific scenario or experience with a specific product feature, we can get such customers through the support case database. Support managers usually do a summary of cases, so customers who mentioned a specific scenario or feature during the conversation with the support specialist can be found by keywords.

It is important to take those customers whose problem has already been solved and those who are currently satisfied with the service.

We usually write to the customer and refer to past support cases, specifying that we are members of the product team and would like to learn more about the client’s experience with the feature. Between 2% and 10% of the contacted clients usually agree to an interview. It’s important to understand that customers who contact the support team may be atypical, and not reflect the whole user base, but their experience still can tell us a lot about the context of using the product.

Optimizing recruiting letters and Calendly for interview appointments

We look for respondents not only with the help of account managers but also through other points of contact with clients — we post links to the recruiting questionnaire on social networks, in the community on Reddit, in email newsletters, on some pages of the website, as well as on the final page of the NPS survey. We then select clients who fit our criteria and arrange interviews with them via Calendly.

Calendly is a public calendar that syncs with the researcher’s calendar.

The researcher can specify available interview slots, and the respondent can sign up for a suitable slot. After signing up, he or she receives an email with an invitation to an interview and a link to a Zoom meeting. Such recruitment automation is very important to us — we do research with people from a dozen different time zones, and without a such tool we would spend a lot of resources on scheduling interviews and timing with the respondent.

Conversion of recruiting emails is quite low (from 2% to 10%). We tried many variations in an attempt to find the best wording. Here is what we discovered:

Specific topics work better than general ones — “we want to talk about product notifications” is better than “we want to talk about your experience.” The topic itself can be expanded during the interview
While Calendly helps us save recruiting time, some recipients are afraid to click on a link, especially on a shortened one, or the one embedded in the text. So now we more often insert the Calendly link on a separate line without shorting it, and we also emphasize in the email that the client can email us about the preferred time directly if that’s more convenient for them
As mentioned above, you can refer to the client’s previous experience (e.g., recently closed support cases). This makes your request more relevant

*Calendly allows respondents to choose a convenient slot from those offered by the researcher*

Lee Munroe (Mesosphere) wrote an awesome description of the recruiting process and UX research process in general
Tomer Sharon (Goldman Sachs, ex WeWork) also has a good article on the subject (the article is much broader, but it covers the topic well)

2. Involving a team in the research process

Five Steps for involving designers in the research process

We support the trend toward democratizing research — teaching teams how to formulate and test some hypotheses on their own to focus on more complex questions, involving designers in research in 5 steps:

The first step — sharing the research results with the team. We try to make the results useful and understandable, while still generating interest. We add user quotes and anecdotal facts for the virality of the reports.

The second step — we bring in designers as observers. Gradually, the product team starts coming to the research sessions. We usually do interviews and usability tests remotely, in Zoom, so there’s no problem with the remote observation of the sessions. Initially, we had concerns that a large number of participants in the session would confuse respondents and distort their behavior, but this does not seem to be the case.

The third step — Active observation of the session and processing the results. Passively observing research session can be boring, you want to duck into your phone, or switch from Zoom to the browser. So after a few observation sessions, we involve the designers in more active work. We ask them to take notes about the problems they see right in the course of the session and enter those notes into a collaborative note-taking template in Miro. At the end of each session, the whole team has a follow-up call for 15 minutes and discusses the results, based on the notes made.

This format has several advantages:

It helps observers be more involved, as everyone feels personally responsible for the results contributed
It allows designers to learn to better describe problems because they get feedback on their notes during each follow-up session
It also allows us to have an up-to-date list of problems and insights as we go along, which helps to make changes faster and, in some cases, eliminate the need for a formal report

*Notes from several research sessions in the Miro template. The researcher’s notes are green, and the designer’s notes are purple*

The fourth step is for the designers to conduct individual sessions under the supervision of the researcher. After several sessions as an observer and several completed protocols, the designers feel more comfortable with the research situation and have a better understanding of what to pay attention to. After that, we gradually let them conduct separate sessions with respondents. In this case, the Miro protocol is completed by the researcher, so that the designer can focus on the communication with the client itself. Each session is also followed by a 15-minute follow-up session in which the team discusses the results and the researcher gives the designer feedback on the moderation.

The fifth step is for designers to conduct interviews entirely on their own. In this step, the researcher helps formulate hypotheses and questions and helps find respondents for testing, while the product team members conduct the research on their own. We do “open sessions” — inviting a few respondents without a specific agenda, and announcing these sessions to the product teams, so managers or designers can prepare questions or prototypes in advance. Then at the sessions, they ask questions and moderate themselves.

This has not yet become common practice, and only individual designers and managers are ready for this format, but we are moving to make it a regular process.

Involving account managers to get quantitative data from clients

In b2b, it can be difficult to get quantitative data about your clients. Let’s say you need 200 responses to test the frequency of problems found and consider your data reliable. If you send out a survey through the mail, or even through in-product notifications, you’ll get a response rate of about 5–10% (that’s optimistic), meaning you’ll need to do a mailing to 4,000 people. For some b2b companies, this is quite acceptable, for others, it’s not.

There is another channel for “delivering” surveys with a much higher response rate than emails: account and sales managers we’ve worked with before. Most of them meet with a dozen clients every week and can ask your questions right in the meeting if you structure them in the right way. That way you can dial in quantitative data in a reasonable amount of time.

What is important to keep in mind in this format:

It is important to clearly state the problem or question you want an answer for, it should not be semi-structured interviews
Structured formats like highlighter testing or first click testing and regular surveys work well
The survey should be short, no more than 5 minutes, so that managers can really build it into the structure of the meeting
It is important to describe the problem you want to solve with this survey, it will increase the motivation of both the manager and the client
Do not give questions about emotional evaluation, loyalty, and conveniences, like NPS or SUS, because in the presence of the manager it is difficult for the client to answer objectively

This format does not fully replace quantitative research, but it allows you to quickly quantify problems or scenarios. And you may encounter resistance. For managers, such a task may be unfamiliar. Some of them will be happy to try new formats of communication with the client, and some will find it less comfortable. Sometimes they may simply not have time for such sessions.

Try not to push — there are always people among the sales and account teams who are self-motivated to improve the product. Try the new format with them first. Let them be early adopters of your process.

One way to involve them is to explicitly describe what specific product feature (or user scenario) you are working on improving, and how the results of the work will translate into changes in the product. You can also come to some sessions with the client and fill the first surveys yourself so the manager can see how to do it.

3. Storing and managing research results

Usually, each research session gives us insights that are not related to the current topic — comments of respondents on other scenarios, a description of how they use other products of the company, etc. One of the popular ways to preserve such data is to keep a database of insights and fill it with the clients’ comments, adding tags for a quick search. It helps to create ad hoc reports later when the mentioned topics become relevant.

We tried to maintain a database of insights in Airtable, but encountered a number of problems:

It takes a lot of time to maintain the database — not only do we have to input all the comments, but we also have to delete irrelevant ones and maintain the taxonomy
The product team doesn’t often request the data itself for a particular scenario. At the current level of research culture, the researchers often have to initiate the changes themselves

So now we try to discuss the found insights immediately during a project, even if they were not related to the original topic, and “push” them into the work, rather than just putting them in the knowledge base, where they remain lost and unclaimed

Full-text database of all sessions with respondents instead of insights database

We use Otter as a partial replacement for the insights database. Otter integrates with Zoom and automatically transcribes, records all conversations with the client in real-time, and then stores the transcripts (of course, there is an explicit indication of the recording, and we only turn it on with the explicit consent of the customer).

That way we gather a huge database of interviews in which we can search by keywords. For example, we can search for anything that users said about reports, or about compliance, and get a collection of quotes on the topic.

In comparison with the usual bases of insights a database of transcripts contains much more “garbage” and irrelevant quotes, does not allow us to make statistics on the frequency of occurrence and the frequency of problems, and it takes more time to make one ad hoc report. But it does not require maintenance and management time, it’s more complete and gives us the full quotes on the topic.

*Dashboards are mentioned in 37 interviews. We can look up specific quotes, listen to them, or go to the full transcript*

List of quotes for each client — target interviews on a precise topic

In Otter, not only the texts are saved, but also the names of clients, who produced these quotes (they are pulled from the name of the meeting in the calendar or Zoom). So, we can find not only quotes but also specific clients who described a particular problem or task. When we get around to solving a particular problem or improving a scenario in a product, we can quickly find people who have mentioned that problem before, contact them and get more complete feedback, or even test extreme scenarios.

This is good for us because it allows us to get more complete and meaningful feedback, and it makes users happy because we come to them in connection with topics that are important to them.

*The base of the interview transcripts allows to create of quick ad-hoc reports on each user related topic and to find the specific clients for interviews on narrow scenarios*

In conclusion

Recruitment through account managers and support team, the involvement of the design team in the research process, and full interview transcripts allow researchers to have more influence on the quality of product decisions. Some of these processes we came up with from scratch, and for some of them, I am grateful to the research teams mentioned in the text.

And I’m sure your team has such unusual tools or processes as well, and I’d be happy if you’d share them in the comments.