Guide to Data-Driven Program Improvements

Let’s make your programs better

You’ve already set up a program.

Your app is working. Your team is trained. Maybe you’ve even set up a regular report.

Nice work! So, what’s next?

Once things are moving along nicely, it’s time to make them even better. Unfortunately, nearly 50% of organizations avoid or delay making updates to their applications, because they think “designing an appropriate organizational structure to support data and analytics activities” is too big of an obstacle.

We hope to open your eyes to a fact that they can’t see: You’ve actually already done it.

Inside the data you’ve already collected are hidden insights that can improve your program—and maybe even save it.

In this guide, we’ll walk you through the process of evaluating your current data practices, whether that’s donor reports, baseline & endline reports, or just monthly report outs to your team. Then, we’ll show you how to quickly create more regular reports of that same data. By reviewing these metrics more regularly, you’ll be able to spot potential areas that can improve not only your application but your program overall – whether it’s related to the resources you have, the workforce you support, or how you make your services available to the communities you serve.

This is an iterative process that will continue for the duration of your program. And if you’re anything like us, there will be things you spot the first time through that will make you want to go right back and do it again to make your program even better.

Let’s get started.

1  Evaluate Current Data Practices

2  Review data more regularly

3  Uncover Hidden Insights

4  Using CommCare Data to Adapt to COVID-19

5  What’s next?

1 Evaluate Current Data Practices

We love this process because it uses the data you already have.

Of course, for it to work, you need to know what that is. And that involves quickly reviewing your current data practices, which can range from baseline inputs and macro-level reporting to day-to-day and micro-level reporting.

Program managers and M&E officers, who are usually responsible for the reporting practices of a program, often focus more on aggregate, program-wide reports or dashboards that look at the state- or district-level results of their programs. They are responsible for the entire program, after all, and what time they have to spend on monitoring and evaluation is often spent on high-level, donor-ready reports.

But think about your field-level staff—the people actually putting your program into action. They’re more often engaged with the community at a micro level, responsible for a small cadre of beneficiaries in the same village, school, or health center. And by looking only at aggregate data, worrying trends in one of these villages or schools might remain hidden.

Each level of data review can have a role in your program. Let’s have a look at each of the main types of data practices we see, starting with the high-level approaches and moving into the more precise, day-to-day readouts.

Baseline & input
reporting

Aggregated
Endline &
outcome reporting

Workforce
monitoring

Micro-level
monitoring

Segmented
Macro-level,
aggregate reporting

Donor
Reporting

Day-to-day
reporting

Baseline & input
reporting

Aggregated
Macro-level,
aggregate reporting

Endline &
outcome reporting

Donor
Reporting

Workforce
monitoring

Day-to-day
reporting

Micro-level
monitoring

Segmented

1. Baseline & input reporting

Baseline & input reporting comes before your program even truly begins, but it can play a key role in measuring the performance of your program after it’s implemented. It can measure things like the number of resources distributed (e.g. hygiene packets, school books, etc.), the number of teachers trained, or the number of community members reached.

Baseline reports also help us determine the priority areas of projects with multiple objectives. Capturing the right information before your intervention begins can show you which aspects of your target population best align to which objectives. In fact, baseline reports or needs assessments are sometimes required by funding organizations or partners in order to ensure the optimal use of their resources.

To learn more about implementing baseline (& endline) reporting, click here.

2. Macro-level, aggregate reporting

Macro-level, aggregate reports look at the program as a whole or by a large region, such as state or district.

While they may look at similar metrics (e.g. crop yields, vaccination rates, etc.), they do so through a much wider lens. In these reports, you can see whether your program is hitting its overall targets, but it doesn’t necessarily uncover a clear path to success if it’s not.

3. Endline & outcome reporting

Endline & outcome reporting may be the least helpful indicator for program improvement, because it comes after the program has finished. These reports can be extremely useful in measuring the impact your program had, but won’t offer much in terms of affecting that impact.

That said, sometimes our partners have received additional funding to restart their programs after an initial grant ran out. In these cases, endline reports can be vital to the success of that program’s second run, ensuring that new funds are both secured and used optimally.

To learn more about how to execute an endline survey, click here.

4. Donor Reporting

In exchange for the funds and resources they offer, donors seek transparency and accountability for how these resources are used. Many donors demand detailed programmatic reports on the activities their funding supports. These reports often include metrics like “total beneficiaries reached,” which can paint a large, heartwarming picture for them.

These are what some might call “vanity metrics.” The total number of people you reached might give you some sense of the overall program, but it can’t help you understand exactly what you’ve done well (or poorly) or how you can do it better. But if you start to break the number down into smaller pieces and segment it—by village, by age, or by gender—that’s when you start seeing a clearer picture of a program’s true impact.

Higher-level reports are where we see most organizations having success. They know how to track their program’s performance overall, and they do it well. It’s worth noting that part of the reason programs do this well is because they know their funding is dependent on it.

Let’s now look at a few data practices that we only see our more successful partners leveraging. These are more precise, micro-level reports that can help you uncover the potential program improvements available to you.

5. Workforce monitoring

What about your frontline workers? Do you monitor them to see who is submitting data, and when? Can you identify workers who might be having trouble completing their work?

Looking at this data regularly will help your team set expectations for how much work your frontline workforce should be doing and alert you when you should check in with certain users. We’ve added several pre-configured worker activity reports in CommCare that can tell you a lot about the work your team is doing on the ground. Don’t forget to applaud those workers who you consistently see doing great work, too!

6. Day-to-day reporting

By aggregating all your data, high-level reports can hide time-based trends, especially those related to holidays, political events, and natural disasters.

By its very nature, day-to-day reporting segments your data by time, allowing you to identify potential issues earlier and take immediate corrective action. You may see certain trends emerge in these reports that you can begin to account for and plan around. For instance, on Friday you may see half as many patients coming to the clinic as any other day, so you may look to reduce staff hours or send more workers into the field.

7. Micro-level monitoring

Instead of looking at district or state-level reporting, could you break your data down and look at the village level? Perhaps view it by small groups of data collectors?

Just as monthly and quarterly reporting can hide day-to-day trends, aggregate high-level data can obscure trends in particular groups of users. In CommCare, you can download your data for a group or an individual user, allowing you to look more granularly at trends within small groups of users. This level of monitoring might allow you to spot users who need additional training or a new type of content to support them in the field.

What you’ll notice about these three data practices is that they actually use the exact same data the high-level, aggregate reports do—they just break it down a bit more and more often.

These types of reports and readouts, if viewed regularly and incorporated into a regular planning and optimization process, can help you to identify opportunities for immediate course corrections that you otherwise would have missed.

2 Review data more regularly

The day-to-day and micro-level report outs come from the same place as your aggregate, macro-level reports, so the only additional work is in setting them up. There are many ways to do this, but the method you choose should be the one that fits best with your team’s habits.

Think about how your team could most easily and organically integrate regular data review into their week. Don’t create a beautiful Tableau dashboard if no one will check it – maybe an automated daily email works better for your team.

If you’re using CommCare, here are a few tips on how to create these reports for your team and find ways to act on what they’re telling you:

Create Custom Exports

Custom Exports in CommCare were created specifically for this routine data monitoring. They allow you to review all of the data already being collected in your application and in the worker monitoring reports to help you find meaningful indicators.

You can then refine your data exports to only report on the indicators your team cares about. You can also filter these reports to run the data for any set of groups or users—for as short or long of a time period as you’d like.

Schedule Data Review

The scheduling of automated data exports can turn a one-time expenditure of effort into daily potential epiphanies for your team. In CommCare, both Worker Monitoring Reports and the Report Builder exports can be scheduled to be sent out via email, so there’s no excuse not to be checking your data every day.

The best part is that the only thing you need to do after setting up these automated reports is make the time to read them. Schedule short blocks in your calendar to review them and even invite your team members to do it together and hold each other accountable. These reviews can be as short as five or ten minutes, but the payoffs could be massive.

Establish Data-Action Triggers

Actually acting on what your reports tell you becomes much easier if you take the time to plan ahead. Set ranges for what “normal” data looks like and agree ahead of time on what actions should be triggered when normal thresholds are crossed.

When discussing the potential actions, you first need to know what you can and cannot change in your program, keeping in mind both the scope and scale of any potential actions:

  • What does your application development time look like? Could you release a new version of the app, if needed?

  • How is your project set up to contact frontline workers and give them assistance if they’re having a hardware or software issue?

  • How flexible are your frontline workers? Could you send them to alternative locations or catchment areas?

  • How flexible is the budget? Do you have extra funds you could allocate to a new activity? Could you reallocate funds from one activity to another?

  • Are there partner organizations working in the same space that could provide add-on services to your beneficiaries, if needed?

Once you establish these data-action triggers, the whole process becomes much simpler: Just check your automated reports every day, ready to take action. And be flexible! You can always change the parameters or actions themselves.

Incorporate More Flexible Practices at Work

Foster the potential for continuous change and improvement within your organization based on what your data is telling you. Set up time either as a new, quick meeting with the core team or carving out time within regular, standing meetings to review your data and discuss whether to take action.

Encourage all staff to speak up when they have ideas for ways to improve your program or even if they just understand the data differently. Change is good when that change is based on data!

The things you learn along the way, from the indicators you choose to review to the data-action triggers you set, can all be adapted and improved based on your program’s needs. You’ll soon see potential for these practices in all your meetings and projects—embrace the opportunities they provide!

3 Uncover Hidden Insights

As you establish your data-action triggers, the biggest question to ask is “what can we change?”

Your data can tell you a lot of things. It’s up to you to decide what you’re willing and able to listen to.

On programs we’ve worked on, we typically see three key categories of potential changes:

Resource Reallocation

Your data might tell you some teams are receiving too many (or more often, too few) resources. How might you be able to shift things around to make sure everyone gets what they need?

Situation #1:

Refugee Camp A and B got allocations of sanitary napkins based on their population of women and girls last month. This week hundreds of new young girls came to Camp A.

View Suggestion

Situation #2:

Each legal help desk across the city has one attorney for case intake. But Desk 1 has gotten twice as many visitors as any other desk the last two weeks, meaning the average wait time for clients is 40 minutes longer than at any other Desk.

View Suggestion

(Re)Training

Sometimes, you’ll see that your team just isn’t using your application quite right. Did they ever go through training? Maybe they need a refresher?

Situation #1:

The pre-test scores for children in School C were 30% higher than any other school. After talking with staff at School C, you realized they were administering the pre-test after reviewing the day’s lesson, instead of before.

View Suggestion

Situation #2:

Users in Factory M are inputting workers’ temperatures in Fahrenheit, but in Factory L, they are inputting temperatures in Celsius.

View Suggestion

Modifying Services

You might find that certain populations are in dire need of the services you provide. Or maybe the people you serve need something slightly different to what you planned for. How prepared are you to adjust your services to give the right people what they need?

Situation #1:

Our project was meant to work with PLHIV ages 15-25, but attendance data from community meetings showed that boys and girls as young as 12 were attending meetings and needed services.

View Suggestion

Situation #2:

At every one-on-one farmer meeting, you asked farmers what their number one challenge was. Overwhelmingly, farmers said they needed access to high-powered machinery.

View Suggestion

4 Using CommCare Data to Adapt to COVID-19

In 2016, MiracleFeet reached out to Dimagi to set up the Clubfoot Administration System (CAST), a universal tool to track treatment and program data for their clinics around the world. Their goal is to reach and treat at least 70% of children born with clubfoot, a common birth defect historically overlooked in many countries that is relatively cost-effective and easy to treat.

The information that their clinics collect with the application feeds into a global dashboard, where administrators around the world can review treatment quality indicators for every clinic in every country in real time.

By late 2019, more than 300 clinics in 29 countries were using CAST to reach patients and track their treatment. So, when the COVID-19 pandemic hit in early 2020, they already had a system in place that allowed them to see which clinics were slowing or shutting down treatment.

But what could they do with this information?

Well, having detailed information about clinics in Tanzania showed them that most were not shutting down and would need PPE for clinic staff and families. In Guatemala on the other hand, clinics were completely closed, and data showed that many families were facing dangerous food insecurity, so the program reallocated funds to provide food and supplies to the most vulnerable patient families. Meanwhile, the SMS functionality of the CAST app, normally focused on appointment notifications, was retooled to share stretching exercises with the families of patients that they could do at home while they waited for clinics to reopen.

MiracleFeet found out that one in five of their patients has had their treatment delayed due to the pandemic, so instead of standing still waiting for it to pass, they acted on the data they have available.

After interaction with a CHW, primary and specialty care visits increased and urgent care, inpatient, and outpatient behavioral health care utilization decreased, resulting in a reduction of monthly uncompensated costs by $14,244.

Chesca colloredo-Mansfeld
Co-Founder & CEO
MiracleFeet

5 What’s next?

The best part about everything we’ve just covered is not that it all comes from data you already collect and processes you already follow—it’s that it can actually make your program better.

Step 1) Review the data practices you already follow

Step 2) Set up automated reports and regular reviews with your team

Step 3) Look at the services you already offer and find the levers you might need to pull in different circumstances.

And remember: Those circumstances are always changing, so the assumptions you made when you first decided what metrics to track or established your data-action triggers may no longer hold.

Try to review your plans and processes regularly (quarterly? semi-annually?) to make sure that you’re able to maximize the effectiveness of your program. After the onset of a global pandemic, we have to realize that it’s not necessarily about planning for every eventuality as much as it’s about fostering a sense of vigilance and flexibility for anything that comes your way.

Watch the entire webinar from our Customer Success team on how to uncover data-driven program improvements or your team

Watch Now

Speak with our team yourself

Get In Touch

Download the Guide

Get all this advice and more in our Guide to Data-Driven Program Improvements. Step-by-step instructions and illustrative examples will get your team ready to make improvements to your own program.

Download Now