We’re thrilled to share that Mathematica Policy Research has released the report, “Evaluation of the Information and Communication Technology (ICT) Continuum of Care Services (CSS) Intervention in Bihar,” which presents findings from a randomized controlled trial (RCT) of a program implemented by CARE International in Bihar, one of India’s poorest and most populous states. This is by far the largest and strongest study yet to assess the impact of CommCare, and should help address the often-lamented dearth of rigorous evaluations on the impact of mHealth.
The 82-page report contains a wealth of information, including areas of success, areas for improvement, and insights to further improve the role of mobile technology in community health. From our perspective, the bottom line in helping us understand if CommCare improves health outcomes is shown in tables V.6 through V.10 (pages 58-64). These tables report the health-seeking behaviors of 1500 women surveyed in the Saharsa district in Bihar, India– a district that has suffered persistently low health outcomes.
The 1500 women were sampled from 70 subcenters in Saharsa district, half of which were randomized to be in the intervention group, with the other half in the control group. All 70 subcenters received the benefit of extensive health system strengthening implemented by CARE International in several districts in Bihar. The services included beneficiary mapping, home visit trainings and resources for Frontline Workers (FLWs), reproductive and family health counseling tools, facility-based interventions to improve the quality of deliveries, and even an Interactive Voice Response (IVR) system delivering health messages to households, developed by BBC Media Action and Grameen Foundation. The only difference between the women surveyed in the control and intervention areas was that FLWs in the intervention were supported by a MOTECH Suite implementation, also implemented by CARE, including equipping each FLW with CommCare, including videos also created by BBC Media Action.
There were statistically significant impacts on FLW interactions and all health domains that were considered, except for child immunizations. All of these statistically significant impacts were positive, i.e., improved by the MOTECH SUITE intervention. The following chart shows the statistically significant primary outcomes, which were also highlighted in a short brief released by Mathematica to summarize the longer report. The chart shows the improvement in the intervention group as a percentage increase relative to the control group.
*/**/*** Significantly different from zero at the .1/.05/.01 level
We’re especially impressed that these improvements in antenatal care, delivery, child health, and reproductive health happened on top of such a strong control effort. Many previous studies compare the use of CommCare to providing little or no extra support to FLWs. That is, the control group gets no intervention. In this case, it is hard to separate the beneficial effects of CommCare from the beneficial effects of the extra attention and support that the FLWs receive in the process of training them on the technology. In contrast, this new study from Bihar shows the value-add benefits of ICT in addition to extensive support for the community health system.
The report also suggests several areas for improvement in the implementation of CommCare and MOTECH Suite. The report describes challenges with data connectivity and broken mobile phones, which sometimes resulted in poor coordination between FLWs and supervisors, and an increased workload due to paper documentation also being required by the government. It also highlighted low understanding and usage of the supervisor app, resulting in no substantial increase to FLW supervision from the intervention. We’re encouraged that the above results were achieved even with an imperfect system and these challenges, and excited to continue to drive improvements to CommCare supervision tools. (Our current supervision app is already much different than the one tested in this study.)
At Dimagi, we’ve been hard at work for a long time to deliver mobile applications to frontline programs globally, in hopes of improving health outcomes for the people served by our users. We’ve been encouraged by the steady stream of peer-reviewed studies demonstrating the potential of CommCare over a wide range of geographies and use cases. The addition of this large RCT, that is powered to detect changes in life-saving behaviors and has such a credible control group, is truly a cause for celebration!