Back in 2014, we began work to allow application builders to define graphical visualizations that would display in their CommCare applications. Since CommCare provides a platform for collecting virtually any kind of data, the scope of visualizations that users might want to create was theoretically infinite. To ground this work, we focused on two specific use cases for health care in clinical settings: growth charts, which are commonly used in pediatrics, and the partograph, a graphical record of vital measurements tracking childbirth. These two visualizations would add value to existing projects, and they were complex enough visualizations that supporting them would also mean supporting many simpler use cases.
From the two main use cases – growth charts and partographs – we broke out more granular requirements:
- Line graphs
- Bubble graphs – we chose to replace the partograph’s unique bar graph, which uses differently-colored bars to indicate contraction strength, with a bubble chart, a more standard way to depict three dimensions of data
- Multiple series
- Multiple y axes
- Shading and color to highlight critical areas
- Text annotations to label critical areas
Besides the concrete requirements for the two main use cases, we considered how graphing functionality might evolve over time. In particular, adding interactivity in the future sounded appealing, perhaps a graph that would allow a user to add or correct data and then redraw itself immediately. While this wasn’t a hard requirement, it was a consideration in the early stages of this initiative.
Initial Implementation (Java)
The set of required graphing functionality was too large to justify implementing it from scratch, particularly for a research and development effort. Instead, the implementation was a matter of evaluating third-party libraries and integrating one with CommCare.
This was a fairly quick search. No more than half a dozen Android libraries looked like plausible candidates, and our fairly lengthy list of requirements narrowed the candidates down fast. The most limiting of the concrete requirements turned out to be the need for a bubble graph, a somewhat uncommon chart type. While evaluating the handful of libraries we’d found, AChartEngine emerged as the most fully-featured.
The implementation was intended to be only loosely coupled with AChartEngine, in case we needed to change libraries at some point. So it defined a set of classes to model graphing concepts, and it then integrated these classes with AChartEngine. This implementation covered the hard requirements, but thinking about interactivity was pushed into the future, since we didn’t have a concrete use for it yet and hadn’t found a library that supported it well.
The first implementation held up reasonably well for about a year. We added functionality as requests came in, but none required major code restructuring or, the major risk, switching libraries. Graphing-related changes during this time consisted of:
- Adding time-based data, needed by health projects to show change in vitals or other patient data over time.
- Fixing minor visual issues, which involved us forking AChartEngine.
- Adding bar graphs, which was inspired by a desire to see report data on mobile devices. The HQ server team completed an initiative to make mobile worker performance data available on devices, and the new graphing code allowed applications to create bar graphs showing community health workers their performance relative to their peers. This kind of feedback has the potential to improve mobile worker performance, an exciting area for future work.
In the summer of 2015 we started to seriously consider replacing AChartEngine, largely for visual reasons. In particular, while it had been technically easy to implement bar graphs, they were prone to having text labels that overlapped or spilled into the data area of the graph, making the data difficult to interpret. We decided to revisit graphing on Android and evaluate the current state of the art.
Creating a graphing library that properly handles labels, without long labels overlapping each other or running out of the bounds of the view, is a challenging problem, and not many libraries handle it well. Some deal with this problem by having the calling code to specify exact margins, which would be fine if we were displaying known data, but doesn’t work well with dynamic labels. We could have exposed that configuration to end users, but creating a graph already involved a lot of configuration, without users needing to worry about making pixel-level visual tweaks to their graphs.
There were three major contenders to replace AChartEngine. D3 was reliable, high-quality, and gives the developer complete control over the visual output, but it didn’t offer pre-canned visualizations, so it could be time-consuming to implement. Google Charts handled dynamic labels and margins well, but its license prohibited offline use, which made it a non-starter for CommCare. Vega looked interesting and powerful, but its data model didn’t map quite as well to CommCare’s data models, so it would be a higher level of effort to implement.
D3 worked the best for our needs once we added one more library, C3, which added a layer on top of D3 to provide standard visualizations like line charts and bar charts. C3 took a JSON object of options as input and fed it to D3 to generate reasonable-looking graphs consisting of SVG elements, with the visual details handled pretty well, including calculating appropriate spacing so that text labels didn’t overlap.
Present and Future State
The list of specific improvements added by the re-implementation is fairly short,but makes a non-trivial difference in graphs’ look and feel. The new implementation added opacity, which makes a surprising amount of difference in making something look professional, and indeed handled issues of overlap much better, also a big win for professionalism. As a minor benefit, the new version of graphing supports a little bit of interactivity, in the form of tooltips.
Having direct access to the SVG elements in the graphs allows for hacky but fine-grained control when needed. This enabled us to support a feature where bar graphs highlight a single bar in a series with a unique color – not a typical feature of bar graphs, and not one C3 is likely to support any time soon, but a win for worker performance reports, where it’s beneficial to make the user’s own data visually different from that of their peers.
Unfortunately, performance has indeed degraded: graphs can now take seconds to load, which feels like eternity for users. Investigating this didn’t yield much low-hanging fruit: we minified the web files and added UI elements so it’s apparent to the user that the graph is loading, not broken. We’ve done some experimenting with rendering in canvas instead of SVG and with rendering the graphs in the background as PNGs, but this hasn’t been fruitful. It’s also difficult to share code between graphs when there are several on one screen, since the underlying CommCare code means that each graph is rendered as its own WebView. The next step in improving graphing performance will likely involve non-trivial code restructuring.
The Next Frontier
Beyond performance, the next frontier in graphing in CommCare is likely figuring out how to fit it into a good product experience. Graphing is still part of research and development, available to a limited number of projects, and even if it were in the main product today, configuring a graph while building an application in CommCareHQ is prohibitively complex. In an ideal world, a user could add a growth chart or a worker performance report with minimal configuration, yet an advanced user would still have the power to put together a visualization as complex as the partograph.