Fostering the use of quasi-experimental designs for evaluating public health interventions: insights from an mHealth project in Malawi

The evidence base to support the growing field of mHealth is relatively nascent, with most studies lacking the level of rigor needed to inform scale up of interventions. This paper investigates the impact of a maternal, newborn and child health (MNCH) mHealth project in Malawi, comparing the intention-to-treat (ITT) and the treatment on the treated (TOT) estimates, and discussing the implications for future evaluations. Services offered included a toll-free case management hotline and mobile messaging service for women and children. The evaluation methods included a quasi-experimental pre-test post-test design, consisting of cross-sectional household surveys. A total of 4,230 women were interviewed in the intervention area and 2,463 in the control site. While the intervention did not have any ITT effects of the MNCH outcomes studied, there were large TOT effects. Rigorous evaluation designs can be successfully applied to mHealth pilot projects, helping to understand what works and what does not.