How to Do Usability Testing on Your Mobile Data Collection App

Why should we do usability testing?

The short answer is that we (the designers and implementers) are not the user. We might empathize with them, understand their workflows, and think we know how to simplify the app to make it usable, but ultimately, we aren’t the ones who will be using it day to day.

Therefore we need to perform some simple check-ins with actual users, to ensure the app meets their needs and suits their level of knowledge.

Sometimes during the scoping phase where we gather all the requirements, we miss some aspects. These will be picked up during user acceptance testing with a version 0 of the application, an app which is working and built, but not yet deployed. For example, in a project in Malawi, the community health workers follow paper-based protocols to screen sick children for malaria. During scoping, we understood their needs and based the app on the paper workflow. Everything seemed to work perfectly.

But during user testing, a community health worker noticed that the app didn’t let him pause the screening for one child while he waited for the malaria test result. In real life, when they have to wait 30 mins for the test to confirm, they carry on with screening the next child, to save time. But the app didn’t let the health worker exit at that point, rather they were forced to wait the full 30 minutes without moving onto anyone else. This important issue was flagged, and during version 2 of the app design, we built in a pause functionality that enabled them to submit half the form, and carry on with another child, and fill in the test results later.

What does usability testing look like?

The foundations for usability testing begin long before sitting down with users. In fact, it is made up of five key components: user personas, user stories, a usability testing plan, a usability testing survey, and your final report. Let’s take a deeper look at each of these components:

User Personas

User personas categorize your expected users by their common characteristics, describing at a high level the background and needs of each participant. These personas are used as a basis for writing user stories and defining the tasks that each participant will perform with your mobile app during testing.

User personas typically cover:

• • User Description
  • Background
  • Language
  • Technology Experience/Digital Literacy
  • App Tasks (What this user will do in the app)  
  • App Device (What device will this user use to access the app)

User Stories

User stories provide examples of how each user persona will use the app. They are intended to capture the high-level user requirements for the app, both for an individual user and across multiple users, ensuring it supports all required use cases. These user stories help your team approach app development from the perspective of the user, making your main aim to support their needs.

In addition to framing your perspective during the development phase, user stories can also help to flag questions, highlight inconsistencies in project workflows, uncover logistical issues, confirm protocols, and track open issues or questions during user testing. The scenarios you outline as part of your user stories should directly inform your usability testing plan, as well as the question you ask as part of your user testing survey.

Usability Testing Plan

Of course, you don’t want to jump into usability testing with just a list of user stories. Defining your usability testing plan might be the most important part of the entire process. 

The purpose of usability testing is to ask people to use your app for realistic tasks in order to identify its well-functioning (and not-so-well-functioning) areas. How you do that is a different question. You need to define the number and type of participants, the length of the session, and what specifically you hope to learn from it.

The following is an example of a brief usability testing plan from one of our projects (which we have edited to remove identifying details):

• The purpose of the usability testing of [this CommCare application] is to ask people to use the software for realistic tasks so that we can identify what parts of the software work well, and what parts need to be improved.
• Each usability test will be a one-on-one session with the participant and the test facilitator.  Each session will likely take 30 minutes to 1 hour. 
• There will be 5 – 7 participants of each user persona.
• In the test session, the participant will be using the appropriate software (e.g. smartphone, tablet, computer) [Editor: this is as opposed to using printed versions of the screens, for instance]
• Each participant will be given an introduction to the purpose of the test and will sign a consent form for the usability test.  Demographic data will be collected.
• The participant will then be given specific tasks to do using the software (i.e. workflows derived from listed user stories).
• During the test, participants will be observed and sometimes the hands of patients will be videotaped while they use the software. Observations of usability problems will be recorded, without identifiers, by the usability tester. At the end, all participants will respond to an interview assessing ease/difficulty-of-use and suggestions for change. [Editor: Videotaping is not expected to be included in all projects though it is considered the industry standard.]
• Specify the language the usability tests will be conducted in.

Look at how this usability testing plan includes notes on not just the user personas and their use cases, but covers additional considerations, such as the materials, equipment, and software required, as well as the location and setup. You own usability testing plan may also touch on any partner support needed and other contextually-relevent requirements.

Usability Testing Survey

Once your users’ hands-on test of the application is complete, give them a survey to help summarize their impressions. You should collect demographic information for each user, as well as a summary of the usability testing plan for them to review as they answer your questions. These questions should help assess the ease or difficulty-of-use of your application and provide them an opportunity to give suggestions.

Here is an example set of questions on a usability testing survey:

1. How easy or difficult was [EXAMPLE TASK]?
   1. Very easy
   2. Somewhat easy
   3. Somewhat difficult
   4. Very difficult

2. Did you find the amount of time spent to conduct [EXAMPLE TASK] acceptable?

• 1. Very acceptable
  2. Somewhat acceptable
  3. Somewhat unacceptable
  4. Very unacceptable

3. What was easiest or most helpful about setting up, reviewing, and editing [EXAMPLE]?
4. What was difficult or confusing about setting up, reviewing, and editing [EXAMPLE]?
5. What changes would you suggest to make it easier to set up, review, and edit [EXAMPLE]?
6. Do you feel that the form names accurately describe what you can do in each form? If not, what other name would you suggest for the forms?

Usability Testing Report

Once you have completed your user acceptance testing, analyze the results to find patterns of behavior, acknowledging both things that worked well and that should be improved. 

What did users like, and find easy to navigate and understand? Which features confused them, and could be improved in future iterations? It can be helpful to tag and group common themes across users, to help in determining the prevalence of certain behavior and also which areas need the most improvement first.

Sometimes you will have an outlier – a user whose behavior does not match the others. Try to understand from the qualitative interviews why that could be, and either incorporate the feedback to improve everyone’s experience or use your judgment if the feedback is less applicable for other reasons.

In essence, your usability testing report should summarize your methodology, user personas, and testing scenarios used, while sharing observations and recommendations for application improvements based on the feedback collected.

Now Start Testing

Now that you know the five key components of usability testing, you can get started on organizing them for your own mobile app: Who are your users? What will they use your app for? How will you go about testing it? What questions will you want to ask your users? And based on all the information you collect, what changes will you make to your app?

Of course, there are many more considerations you should keep in mind as you organize this process, drawing from learnings around avoiding bias and application design. We’ve compiled a few of them here, along with a short checklist of the tools and people you should include. Happy testing!