Six ways to make remote usability testing more human

A few days before health officials advised against large gatherings because of COVID-19, we held our first annual Summit. It was surreal to go from being in a room with hundreds of civic tech and digital government community members, to working from home with no real sense of how long we’d all have to be apart.

As this new reality started to sink in, it became clear that as manager of the GRIT Toronto program, I was going to have to decide whether or not the usability testing we had scheduled was still something we could actually do — and do right — under the circumstances.

The next GRIT test in our pipeline was a collaboration with the team at Normative, and together we made the call to pivot from our usual in-person testing to doing everything remotely. We quickly sent word of the change in plans to the participants, and then set to work finding the right digital tools to help us make this happen.

We only had five hours to set everything up.

So behind the scenes at Summit, I was hunkered down in the green room, putting measures in place to make sure that we could do these tests while still maintaining the core principles that make the GRIT program what it is: a space where residents feel heard, comfortable, and respected while they test technology that affects their lives. Laura Shaw, Director of Research at Normative, was just as committed to mindfully pivoting as I was, and thanks to our strong collaborative relationship, we were able to think fast.

Of course, none of the planning would have amounted to much without the trust and commitment our GRIT testers brought to the challenge. We all moved as a unit, and it was a success.

It’s my hope that during the COVID-19 crisis, builders of technology remember that usability testing is an essential part of building products and services that help as many people as possible. In that spirit, here are tips on how to approach the challenge based on what we found useful during our first remote testing experience.

1. When you see a human, ask them how they are

Before talking about anything else, acknowledging the reason we’re all working remotely is the most human thing we can do in this circumstance. After all, we’re not truly working from home, we’re all stuck in our homes trying to work.

By agreeing to do the testing offsite, participants all but invited us into their homes, so naturally, the tone and structure of our conversations took on a different quality — and a lot of the time that newness was heartwarming. Who doesn’t love a meeting with a guest appearance from a curious toddler?

We’re all feeling uncertain, so make more space than usual for feelings before getting down to the task at hand. What is the participant doing to prepare for social distancing? Is their dayjob on hold? How are their kids adjusting to finishing off the school year at home? Taking time to name what’s unusual about our day to day lives right now is a great way to create a space that’s comfortable and open.

2. Get step by step consent when recording humans

Whether it’s in person or online, we offer tiered consent whenever we record GRIT test participants. Everyone’s comfort levels are different, so we try to create an environment that factors in as many different preferences as we can. Check-ins to reconfirm consent are built into the flow of the test so people are aware of their options at all times. Especially when working remotely, consistent communication is essential to making sure that everyone is on the same page. The online tools we used made it possible for us to offer a good range of options. Folks decided between no recording, audio only, video only, or all of the above!

3. Choose the right tools

Google Hangouts Meet was our virtual meeting tool of choice for a few reasons. G Suite tools are familiar to a lot of people, it’s free to create an account, and because we use G Drive, it was really easy to organize files from each participant after each test.Hangouts also has a screen sharing function, which we asked each of the participants to turn on. That allowed us to see what the testers were doing in real time so we could take notes on their interactions with the tool being tested. It also, of course, allowed us to talk to the participants during the test. That was perfect for the part of the GRIT test where we introduced users to the tool, and set up a few hypothetical scenarios they might use it in.

Then, using Invision — a prototyping tool designers might be familiar with — we gave participants a mock up of the tool to play with so we could observe how people interact with the design and iterate. We sent the links out to Invision before the test started so participants would have everything they needed on hand. This was a small drawback, since ideally the tester hasn’t seen the tool before they start giving feedback. We just had to hope participants didn’t peek before they were with us.

4. Encourage participants to vocalize their thoughts

It isn’t quite enough to just see how folks interact with a prototype. Having people talk through why they’re clicking on a certain area, or how they feel about colours or layouts, is an extra layer of insight that helps inform conclusions drawn by observers during testing. Again, this is something we’d do whether we were in the same room as the participants or not, but especially since we weren’t, hearing the participant think through their interactions out loud helped us feel more confident in our understanding of what the participant was thinking and feeling. Remote conversations can be a challenge, since the natural rhythm of a conversation can be lost to technical delays and the general difficulty of reading cues through a screen. This makes it doubly important to encourage people to share their thoughts in a remote context. A reasonable reaction to uncertainty during a conversation is to err on the side of not speaking at all, so a gentle reminder that you’re listening could make all the difference!

5. Let the participant know who’s in the room, just like IRL

Don’t forget to introduce everyone present! It’s easy to forget when you’re looking at a computer screen that you aren’t watching TV — there’s a real human on the other side watching back. So even if there are people who will mostly be observing during the remote test, be sure to introduce them. Again, our goal is to build a space where everyone feels heard and comfortable. That’s not likely possible if there’s an unexplained presence in the room.

6. Pay with e-transfers

Where possible, paying with an e-transfer instead of a gift card is always our preference. It’s even more advisable now with everyone trying to minimize time spent outdoors. With an e-transfer, participants get paid in a timely manner, and they don’t have to leave the house to access funds since the money goes straight to their account. Plus, we don’t run the risk of paying the participant with a gift card they don’t need or want. Paying in cash gives testers freedom to use the money they earned as they see fit.

In some ways, it feels like this is the perfect time to do more work like this. Since business as usual has slowed significantly, there’s a rare opportunity right now to take stock of the technology we’re using, and reconfirm it’s doing what we’re hoping it does by user testing. Plus, there are likely more users available at the moment than ever before, since so many of us across the country are waiting for the OK to return the work. In the meantime, why not offer the community a few dollars to test your tools?