How Long Do Programs Use CommCare For?

This is post #5 in our Under the Data Tree blog series, where we  share insights from analyzing CommCare data

Do programs that adopt CommCare tend to keep using it? This question is obviously important to Dimagi. We see continued use of CommCare as evidence that it is providing value to programs and so we hope to see high rates of continued use.  Potential users of CommCare will also be interested in understanding whether other programs continue to use CommCare as proof that it is a valuable addition to frontline programs .

We analyzed all 330 programs that deployed CommCare within the 5-year period from January 2010 to December 2014. The number of months in which programs had at least one user submitting  data ranges from 1 to 58 active months, with a median of 9 months in which data was submitted. The 9 month median may seem low, though that is primarily because CommCare has been scaling and so most programs started towards the end of this 4-year window. The following chart shows the distribution of how many months programs have actively used CommCare. The dark blue bars show the number of programs that can be considered ongoing, in that they were still active in the last quarter of the time period we analyzed. The lighter blue bars show all the inactive programs, i.e., ones that stopped or halted using CommCare.

We’re happy to report that many programs continue to use CommCare after adoption, and for a long time. As you can see, many of the programs that have been using CommCare for many (7+) months are programs that are continuing to use CommCare. And it seems that the longer a program uses CommCare, the more likely it is to continue using CommCare. Many of the programs that were inactive in the final quarter of 2014 had only used it for a few months. There are very few examples of programs that had used CommCare for more than 20 months and stopped.

To further explore how likely programs are to continue using CommCare, we analyzed the 126 programs that had submitted CommCare data in January 2014. Some of these programs started in January 2014, but most would have started before. We chose these 126 programs as a representative sample with which to analyze a year of CommCare use.  The first question we asked is how many of these programs were still active at the end of the year. We defined this as having submitted any data in the last quarter of 2014, since some of the smaller programs might not have been active in December but still were generally active. Of the 126 programs, 87 (69%) were still submitting data during at least one month in the last quarter of 2014. The following table breaks this down by program size, showing that programs that had more active users generally had higher rates of continued usage.

This table shows, as expected, there is a higher rate of continued use of CommCare for programs with more users. For all 58 programs that were active in January and ever had more than 20 active users, 84% were still using CommCare by the end of 2014.

One might think that once a program starts using CommCare, it typically has active users every month until it stops entirely, and that the program generally increases or maintains the number of users month to month. This turns out not to be the case at all. The median number of total months on CommCare is 13, while the median active months is only 9.

The following shows different life cycle patterns of CommCare activity from 5 different programs.  Each program is represented as one horizontal line, with the height of the line indicating how many users were active for that program in that month.

The first and fourth graphs show programs that start small and eventually scale, never missing a month. But the second graph shows a program that tries CommCare briefly and then a year later tries it again briefly, and then about a year later starts using it more and eventually scales very quickly. The third graph shows a program that uses CommCare for a long time at a low rate and then scales and maintains the use for a year and half and then gradually phases out. The fifth graph shows a program that quickly scales CommCare and then after using it about a year of sustained use tapers off to just a few active users.

As these examples illustrate, CommCare programs are diverse in terms of the usage patterns and lengths of their deployments. Some programs are active all year round, while others such as surveillance systems or agriculture programs may only be seasonally active. Many programs are designed to run indefinitely, but CommCare is also used to collect data for short-term research studies, while other programs simply run out of funding. Many programs test out CommCare for a month or two and then there is a long gap before they really start using it.

When we looked at these graphs, we were surprised to see see how often a program would re-start using CommCare after stopping:

• 193 of our programs have stopped using CommCare for at least 3 months.
• Of those 193 programs, 51% have restarted and used CommCare again after the 3+ month break.
• The majority of 3-month attrition events take place early in the life of a program, and programs are more likely to recover from these early attrition events compared to those that take place later in the program life cycle.

The chart below shows the restart rate based on the number of active months when the program stopped using CommCare:

These data provide us some insight into the variable shapes that a program’s activity can take. In particular, there is much more stopping and starting of program’s than we previously thought. It is common for programs to stop using CommCare entirely sometimes, especially in the early months, and restart later. There is also a fairly high rate of continued use of CommCare.

The findings here could inform future investigations into program activity patterns. For example, if we know that programs that are exhibiting a particular behavior pattern that we know often precedes a program stopping, is there anything we can do as implementers to encourage continued activity? Or are stoppages just due to external factors like a program ending? Supplementing the activity data with more information about programs could help to answer these questions. In addition we would like to move this investigation into attrition one step down – to the actual user; we will be exploring different patterns of user activity and attrition, and exploring some of the reasons that a worker might stop and start again.

Thanks for checking out Under the Data Tree! Please feel free to comment with any thoughts, or send questions to datatree@dimagi.com.

———-

Here is visualization of all 330 programs. In each graph, we compressed the vertical axis to show groups of 25 users, so each vertical block represents up to 25 users that were actively using CommCare that month.