Tactics in Field Research: Card Sorting

At Dimagi, we work directly with local end users to facilitate collaborative decision-making, rapid prototyping, and user acceptance testing in developing locally-appropriate digital solutions. 

Our mantra is to “design under the mango tree.”

This means that in the process of developing these digital solutions, we first secure the end users’ buy-in to guarantees that the mobile apps we build will support the unique challenges that these frontline workers face. This approach is useful not only for designing applications but investigating user workflows and understanding their unique challenges.

Our largest project, and in fact, the largest mHealth program in the world, was an application called the Integrated Child Development Services-Common Application Software (or ICDS-CAS). This app served as a key pillar of POSHAN Abhiyaan, the Indian Prime Minister’s Overarching Scheme for Holistic Nourishment, and supported more than 600,000 frontline workers known as Aanganwadi workers (AWWs) across 28 states in India. ICDS-CAS helped tackle malnutrition by digitally equipping community health workers with a mobile solution that improved the quality of their care and enabled effective monitoring of new births and malnourished children across the country.

In the past few months, the ICDS-CAS team has adopted a few innovative approaches to connect with Anganwadi workers and offer them channels for feedback. These activities were designed for specific objectives of user research, but they engage the AWWs in an interactive and fun way that – importantly – reduces bias. Their first approach was card sorting.

Card sorting is a method of compiling rapid, initial survey results that are easily incorporated into further evaluation. Participants explain their expectations and understanding of various topics while organizing them into discreet categories.

These categories allow users to think freely about their associations between topics but within a clear context and framework. This makes it much easier to analyze results across sessions and set your team up for further investigation. Data collected in pre- and post-activity questionnaires can also help establish various parameters for future studies.

The ICDS-CAS team used card sorting on four different occasions, each time in a slightly different way, in order to get unbiased feedback from the end users of the application:

Card Sorting Investigation #1:

Their first use of card sorting was to understand growth monitoring knowledge amongst our survey of Angandwadi workers (AWWs) in CAS districts compared to those in districts that used paper registers. 

First, an AWW was given a stack of fruits and vegetables that needed to be sorted based on whether it was a fruit or a vegetable, then based on its shape, and lastly based on its color. At the end of each step, she was asked what information she used to put the cards in different categories. 

This was done for practice to familiarize the AWW with the activity and does not actually affect the test results. 

Next, the AWW was given a stack of 20 growth monitoring charts of different children. She then sorted these based on different categories and parameters (e.g. gender).

Once she compiled the cards by one parameter, she was asked what information she used to do the sorting. The cards were then collected and stacked together and to be sorted again by the next parameter from the list. 

While the AWW did the sorting activity, the investigator simultaneously recorded the data on a CommCare application. 

Card Sorting Investigation #2:

Next, the team used card sorting to confirm and learn more about the different data requirements for the platform that its reporting dashboard users had raised in an initial interview (e.g. what data types did they track, how were they collecting those data, what did they use them for, etc.). Their key objective was to find out what the help desk users did with the available data.

In this case, the team used an online card sorting tool called OptimalSort to ask users to sort different data points related to ICDS program delivery into various predefined categories. 

In this exercise, different reports and indicators from the dashboard were compiled into a list of 18 keys. Cards were then prepared with each one representing a data point related to the ICDS Program (from either the online dashboard or paper reports).

From these, three themes were identified:

1. Frequency of Usage 
2. High to Low Importance
3. Frequency of Data Updates 

While an individual sorted the cards, the investigator made observations about their overall performance and comfort with the online tool. As part of the exercise, investigators would ask about the priority in which a particular card was sorted and why it might have differed from their responses in the interview. This was especially true if a user marked a data point or report as one they “never” needed.

Card Sorting Investigation #3:

In their third exercise, the research team used card sorting to discover the most comfortable communication medium amongst rural women who interacted with a semi-automated conversation tool named Poshan Didi, which was created to support ICDS-CAS. This approach allowed the team to explore how common communication channels were used for different purposes, including communication with Poshan Didi.

In this exercise, they selected four communication channels: SMS, Telegram, Phone Call and WhatsApp. An icon for each was put on individual cards and placed in four different corners of a room. 

All the beneficiaries were asked a series of questions in which they had to choose a particular medium to communicate. To make their choice, they would move to the corner of the room where the icon’s card was placed. 

The investigator recorded the participants’ choices and asked questions, such as why a particular medium was preferred and what their second choice would be.

Card Sorting Investigation #4:

Finally, the research team used a card sorting approach to gauge the trust a beneficiary had in going to different people for different situations related to their healthcare. These situations were read out for a group of beneficiaries, and each of them had to make a choice. If mothers were willing to discuss further, facilitators would as probing questions to understand when they would or would not reach out to Poshan Didi.

The beneficiaries could choose from four sources of healthcare information: AWW, Doctor, Poshan Didi, and Family Member. Each was written on a card, represented by an icon, and placed in front of the beneficiaries.

All beneficiaries were then asked a series of questions and had to choose a particular person they would go to for receiving or sharing a certain kind of information or support.

Like the prior exercise, the investigator recorded their choices and asked follow-up questions, such as why a particular person or resource was preferred and what their second choice would be.

In all of these exercises, you can start to see how a simple activity, like sorting items into different categories, can help your users and beneficiaries uncover powerful insights about the way they offer and receive support. Think about the different aspects of your own program and how the way they are structured in your application might align (or not) with how your users and beneficiaries perceive them. As we have learned with card sorting, if there’s ever any doubt, they are the best people to consult.

You can learn more about these techniques and more in our Guide to Usability Testing.