Django ORM Memory Leaks in Debug Mode
TL;DR: Django has tremendous memory leaks for any long running task with many SQL operations with debug on.
Solution: Turn debug off.
Story:
So I just spent a long time digging this up. We have a management command that runs a ton of SQL statements and queries/inserts all kinds of objects into the database. I was attempting to run it locally but it kept getting slower and slower over time and the memory was just linearly expanding. I naively first did a really simple profile with heapy. This told me that I had a whole lot of strings taking up a whole lot of memory. My first inclination was that maybe python was doing something dumb by not interning strings properly or by not garbage collecting properly. I added a few intern() functions on what I thought might be candidates for duplication, and i explicitly garbage collected throughout the script. It had zero effect.
Share
Tags
Similar Articles
Dimagi to Launch Monkeypox Case Management Solution to Support US Public Health Departments
Digital template application powered by CommCare for investigating monkeypox cases and tracing contacts to be offered free for 6 months to control outbreaks
Product Updates
September 7, 2022
Mitigating the Impact of COVID-19 on Global Vaccination Efforts
Disruptions in routine immunization due to COVID-19 can lead to secondary health crises, especially in children. That's why we've expanded on the platform used to provide more than 1 million COVID-19 vaccines to include routine immunization, as well.
Product Updates
October 11, 2021
CommCare Precision Tasking Framework: A Prioritized Tasking Solution for Supervisors and Health Workers
Dimagi has created a new Prioritized Tasking Framework that aims to help CHWs focus on service delivery, enable data-driven decision making and provide patients access to better care.
Product Updates
September 6, 2021