Premium
Challenges of Evaluating Multilevel Interventions
Author(s) -
Nastasi Bonnie K.,
Hitchcock John
Publication year - 2009
Publication title -
american journal of community psychology
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 1.113
H-Index - 112
eISSN - 1573-2770
pISSN - 0091-0562
DOI - 10.1007/s10464-009-9239-7
Subject(s) - psychological intervention , mental health , psychology , community based participatory research , population , multilevel model , program evaluation , institutionalisation , applied psychology , participatory action research , medical education , medicine , sociology , environmental health , computer science , political science , psychiatry , public administration , machine learning , anthropology
This article uses the Comprehensive Mixed‐Methods Participatory Evaluation (CMMPE; Nastasi and Hitchcock Transforming school mental health services: Population‐based approaches to promoting the competency and wellness of children, Thousand Oaks, CA: Corwin Press with National Association of School Psychologists 2008; Nastasi et al. School‐based mental health services: creating comprehensive and culturally specific programs. Washington, DC: American Psychological Association 2004) model as a framework for addressing the multiplicity of evaluation decisions and complex nature of questions related to program success in multilevel interventions. CMMPE defines program success in terms of acceptability, integrity, social or cultural validity, outcomes (impact), sustainability and institutionalization, thus broadening the traditional notions of program outcomes. The authors use CMMPE and an example of a community‐based multilevel sexual risk prevention program with multiple outcomes to discuss challenges of evaluating multilevel interventions. The sexual risk program exemplifies what Schensul and Tricket (this issue) characterize as multilevel intervention–multilevel evaluation (M–M), with both intervention and evaluation at community, health practitioner, and patient levels. The illustration provides the context for considering several challenges related to M–M designs: feasibility of randomized controlled trials within community‐based multilevel intervention; acceptability and social or cultural validity of evaluation procedures; implementer, recipient, and contextual variations in program success; interactions among levels of the intervention; unanticipated changes or conditions; multiple indicators of program success; engaging multiple stakeholders in a participatory process; and evaluating sustainability and institutionalization. The complexity of multilevel intervention and evaluation designs challenges traditional notions of evaluation research and experimental designs. Overcoming these challenges is critical to effective translation of research to practice in psychology and related disciplines.