Usability Testing: Six Considerations and a Checklist

Usability (or user acceptance) testing is the process of putting a functioning app in front of your end users and seeing what they think. The concept is quite simple, but the execution is a bit more nuanced.

From selection bias to asking leading questions, there are many ways that your approach to collecting feedback can lead your users to give you the answers you want. As tempting as this can be, you need to make sure that the feedback you collect is as unbiased as possible.

Usability testing does not have to be that hard, but you do need to make sure you understand it. For instance, some easy tips are:

• Any user testing is better than no user testing
• An ideal number of people to test with is five
• Consider having two people to run the user test – someone to facilitate and someone to take notes

Here are six considerations to keep in mind when performing usability testing for your mobile data collection app, along with a checklist to make sure you’re ready to go. 

1. Consent is important.

Always give people the option to opt out, and if they do want to participate, get them to sign a simple form that explains the following:

• Reason for research
• Allows you (NGO/ Company) to use the research to improve the app
• Add in any details about if you can use the voice recordings, and pictures, and for what purposes

2. Consider reimbursing participants for their time or travel costs.

You value the research and feedback you are getting, so ensure you adequately compensate the fieldworker for their time, or at least their travel costs. You could provide an airtime voucher, a meal, or something of value to the participant instead of cash if necessary.

3. Focus on the app, not the person’s skills.

Explain you are focusing on what can be changed with the app, and not evaluating their skills. Give an example such as:

There is a bridge, and every time too many people walk on it, another brick falls into the water. Whose fault is that? The people walking on the bridge? Or the bridge engineer? (Answer: the bridge engineer, but we needed to use the people to test the bridge, so now we can fix it)

Remove yourself from the process – claim you are helping to audit another team’s mobile data collection app, or that this tool was given to you broken and you need to fix it – this helps people feel less shy about giving honest feedback to the “person who built the app.” Remember to state that honest feedback is what you are after, and changes can be easily made.

4. Don’t tell the user what to do. Observe while they follow a storyline.

Create a script that enables you to test certain functionality piece by piece. This helps the user know what to do, while you can see how they approach the problem naturally without intervention:

Correct prompts:

1. A child walks into your clinic, very sick. Please register the child
2. Now screen the child for illnesses
3. Say they have a fever for 8 days and no other danger signs
4. You do have the equipment to perform the malaria test

What you don’t want is step by step details that prompt the user too much and doesn’t let them interact naturally with the app. This will just evaluate if they can follow instructions, and not how intuitive your app is to use.

Incorrect prompts:

1. Unlock the phone
2. Press CommCare
3. Now press “Register New Client”
4. Now press next, then select “age 5”, it’s over there by the green button

You want to evaluate how the user reacts naturally, and where they might get stuck. If they need more detail, give story details, not technical steps.

5. Note down any and all observations.

It helps to get people to “think aloud” while they are following the steps. This helps the facilitator reduce assumptions about the participant’s behavior, as it gives them a chance to explain their own reasoning for their actions. For example:

Interviewer: Please register the child, her name is Abigail. While you do this, speak about what you are doing step by step.

User: “Okay, I am looking for CommCare, now I am clicking it, but it’s opening so slowly. Okay, I’m inside, now I’m trying to see. Where should I go? Hmm.. that didn’t work. Okay, found the form, I’m trying to write ‘Abigail’ but there is no capital letter. Ah, there – I found it.”

Any exclamations of frustration, side comments, expressions can be signs that point to something that is not working. Note down when and where that happens.

If you come across any bugs, note them down too, but remember to focus on workflows, ease of usability and any assumptions the users are making, not just the quality of the app.

6. Explore solutions

If you notice something during user testing, start exploring solutions together. For example, during one of our user testing sessions, a community health worker said, “Okay, while I wait for my malaria test I want to carry on with another child but I can’t.”

This led us to ask more questions:

• Why do you need to do this (“The clinic gets busy.”)
• How do you do this on paper currently? (“We just change to another paper and come back later to fill in the test results; the child sits and waits.”)
• How do you think the app could help you? (“I want the app to freeze on this child and open another one without losing information.”)

If you have an idea for the tech workflow, sketch it out on different screens. Show them to the user one by one and see if it helps them understand your proposed solution and if it needs changing.

This is very useful for user-testing reports – sketch out the table of what data they want to see, the headings and columns and where the data is coming from.

Checklist for Usability Testing

• Facilitator and notetaker (they should either understand local languages or be able to effectively communicate within the cultural context)
• Consent forms printed
• Participants identified – try to find some who are in different areas, ie. urban users, rural users, experienced clinicians versus new clinicians, older people versus younger people (to see how the technology is used)
• Scripts for workflows printed
• Notebook and pens
• Voice recorder (if necessary)
• App set up with a demo account so that the user can make mistakes, register test clients, etc.
• If you are user testing in remote areas, make sure the app is synced and ready beforehand
• “Screens” templates printed that you can tweak to add more details – or just more blank paper to draw out workflows

Now that you know how simple user testing can be, it’s time to get out into the field – whether it’s a rural area, a hospital, agricultural co-op, or supply chain warehouse – and test with the people who will use your mobile data collection app on a daily basis. With careful listening and observation, you will find many clues to areas that need improvement in your app – from the icons, the written wording, and most importantly, the workflow. 

---

Read up on more ideas of how to improve your app here, and if you want to consider a pilot as part of your testing process, read up on why piloting is still important.