Usability Testing under Pandemic Conditions: First reports from the field

A “normal” usability test in our laboratory: A test user travels to our lab at Steinstrasse in Zurich. A receptionist greets her and leads her from the reception area into the lab itself. There, she is introduced and guided through a simulation or tasks with the product we are testing, such as software, websites or devices. Our test manager often sits directly next to her. Our clients observe the test from the adjoining room via a large screen or directly through a mirrored window.

Quite a lot of dangerous contacts! Most test users travel by public transport. They are greeted politely with a handshake. The test manager and test user sit directly next to each other for an hour. All of the test users handle the same product. And in the observation room, our clients sit comfortably, but perhaps a bit too closely together.

The extraordinary situation caused by the COVID-15 pandemic is clearly forcing us all to take rigorous measures. Fortunately, we have been offering our clients the opportunity to monitor tests remotely via streaming for years and had already performed several remote usability tests. This has always been necessary to some degree, since our clients and interview partners sit all over the world. In today's situation, we benefit from this practical remote testing experience. For our clients, this means that we can carry out usability tests with our normal quality and with even more flexibility. And because our test users can participate from home, we maintain the social distance painlessly.

Instant pivot from in-person to remote testing? Our team made it possible
 

The need for remote testing became clear last week when we were in the last steps of preparing a round of 12 test interviews for Swisscom. Shortly before the first round of testing, we decided to carry out the tests remotely as a precautionary measure. We managed to overhaul our plans, preparing for a full remote operation in less than half a day. Of course, Liz and Zuzanna had their hands full during this, our first test under pandemic conditions! The most time-consuming part was contacting the test users who were signed up (some of them set to participate that very day) and communicating to them that they did not have to travel to our lab, but could (and should) carry out the test comfortably at home. 

Despite this tight lead time, our first tests under Corona conditions worked smoothly. Liz and Zuza were able to contact all of the test users in time and test worked almost perfectly from the beginning. Stefanie Luder, our client from Swisscom, agreed: 

"Even under these conditions, the insights gained from the interviews led to a better understanding of our clients and their needs. Since we were able to observe the tests directly as usual, the initial results are already available. Our team can now continue working on our project from the home office with a clear conscience. We are confident that we will be able stick to the ambitious project plan despite Corona". 

Remote tests can deliver equally valuable results

We are convinced that well-prepared remote tests can work as well as "normal" tests in the usability lab. There are certainly limitations and potential disadvantages, like a potential bias in the panel of test users due to equipment limitations at home. But there are clearly also advantages to remote testing, like the lessened demand effects due to the physical absence of the test manager next to user. By the way, we aren’t just saying it: the power of remote testing has also been demonstrated in scientific studies (disclaimer: our own Klaus Heyden was involved in this study during his studies).

Of course, more challenges await us. Today, we are in the midst of preparing a remote focus group. In the coming weeks, we will conduct hardware tests and deep dives at the user's home – all virtually of course. And we will test the usability of even more websites. We are happy to be well prepared for this unusual situation, and that we can keep up our daily testing routine for our clients.