Early Stage User Testing

Re-blogged from Project Mwana.

It’s always nerve racking showing your software to actual users for the first time, and SMS systems are no exception. Technically there’s plenty to worry about. Did you connect the modem and make sure your server isn’t going to hibernate or crash? Did you bootstrap the database properly with the demo data that you’re going to use? Did that bugfix you made 30 minutes before the demo break anything else? All of these things can, and often do, go wrong.

However the larger issues are not technical, but programmatic. Is your system user-friendly? Is it practical? Is it even solving a real problem? Often times you’ll build what you think is a simple, intuitive, and useful SMS system only to find that it’s far too complicated, confusing, and actually solves the wrong problem entirely. So naturally it was with both excitement and trepidation that the team made our way to Mansa Central Clinic on Friday to get our first round of real user feedback.

At Mansa Central we met with Majory, a sharp and busy nurse. We tried to put her at ease as seven of us of us sat in an imposing circle of chairs, taking notes and hanging on her every word. As we explained the purpose of Results160 (to get early infant diagnosis test results from the central labs back to the clinics faster by sending them straight to the phones of the clinic workers via SMS) we were immediately reassured by her reaction. Yes, the results take a very long time to come back. Yes, it is a big problem, and frustrating for both the health workers and the mothers. Anything we could do to
improve the speed of results delivery would be fantastic. Useful? Check.


Next we talked about phones. She told us that all the workers have personal phones that they keep on them at all times. Service is good where she works (this will unfortunately not always be the case in our clinics) and everyone is quite familiar with SMS. Practical? Check.

Finally it was time to demonstrate the system. Talking through carefully designed training materials, we showed her how the system would work, and explained to her how she could register and receive test results for her clinic. We talked her through an imaginary workflow of being notified that new results were ready, responding with her secret security code, and receiving the results directly to her phone.

Much to our surprise and delight, it went shockingly well. Many times in early-stage user testing, you’re happy if the user has a single back-and-forth with the system successfully. However, Majory got through the entire workflow we’d planned with little guidance from the team. When she got stuck, she fell back to the training materials, and was able to figure out what to do next on her own. She was fast, efficient, and happy at the conclusion. So were we. Usable? Maybe.

The session went so well (and so quickly) that we decided to make a second stop at another nearby health facility: Buntungwa Clinic. There we met with two more clinic workers, Beatrice and Hildah, and went through the system again. Once again the feedback was extremely positive. Yes, they would love to get their test results sooner. Yes, everyone has a phone. Yes, everyone uses SMS. However, when it came to demo the system we had one small problem. Because we had only anticipated making a single stop, we had only setup the database to include one clinic with pending test results, and those results had just been collected by Majory at Mansa Central. We had no data and no way of adding data to the database remotely.

Enter agile user testing. Rather than have them SMS our carefully crafted system (which would not be able to send them results), we had them SMS one of the developer’s phones. While we explained and introduced the system to them, the developer was frantically copying and pasting messages from his own previous interactions with the system in order to fake the responses.

Again, things went off great. They breezed through the session and had almost no issues, even offering insights as to where they thought small things could be improved.

We’re not sure how representative the clinics we visited will be compared to sites in the more rural areas where we will also implement, however, the first round of testing was extremely encouraging for the whole team. There is nothing like seeing your system in use for the first time, and it is that much more exciting to see that you’ve built something that’s actually practical, usable, and may even solve an important problem.



Similar Articles

The World's Most Powerful Mobile Data Collection Platform

Start a FREE 30-day CommCare trial today. No credit card required.

Get Started

Learn More

Get the latest news delivered
straight to your inbox