Rigour in Methods and Evaluation for Community Engagement

This paper is an overview of the important considerations that arise at the outset of a project. There are numerous ways that a work team may decide on which methods should be prioritized among the many tools available for community engagement. As the project comes to grips with the scale and the scope of a 7-year project on Community Engagement, it will be essential to explore how the various evaluative methods: Theory of Change (ToC), Developmental Evaluation, Collective Impact, and Action Research are combined, and how Evaluation scholars have typically approached these subjects in the past. Is it possible to use ‘Theory of Change’ at the same time as other methods? One may answer this question with a resounding “Yes!” In the community sector, there are many versions of a Theory of Change. The term may be applied to both one’s personalized impression of the arrow of change, as well as according to traditional Log Frame models for mapping long term ‘policy change.’ Even if there are dilemmas in coming up with language to describe what is meant by “Theory of Change,” there are many opportunities for ToC to be fused with other methods, and tried and tested over the life of the CFICE project, whatever the original connotations of the researcher or community practitioner may be. Community First: Impacts of Community Engagement (CFICE) Rigour in Methods and Evaluation for Community Engagement 2 Introduction Both methods and evaluation (and sometimes evaluative methods) will come into view with increasing clarity over the first year of the CFICE project. As a means to synthesize some of the thoughts from the initial CFICE meetings, this document is a critical reflection on what constitutes analytical rigour in both the methods and evaluative tools that were mentioned in the initial CFICE meetings (of Steering Committee and Program Committee) and in the initial research proposal. Rigour is not something that can be achieved merely by adhering to a quantitative framework, and nor is it something that requires the adoption of one toolset over another. In-stead, the CFICE project will rely on multiple methods and toolsets, and establish a criteria of rigour on the basis of the quantitative and qualitative research needs of the Hubs, and based on their own approaches to the reporting of community activities. There are various levels at which rigour can be examined according to the various disciplinary backgrounds that make up the CFICE hub teams. On the research methods side, postpositivist theory is a point of departure at the intersection point between policy studies, human geography, social work research and feminist theory. On the evaluation side, the theory of change as well as action research have been used as macro frameworks to bring together (in a semi-rigorous fashion) the theory of change for community actors, and community organizations. Indeed, post-positivism is a common ground from which the research methods and evalua-tion framework could both be elaborated. Another way to view the activities of the Hubs is by subdividing them into micro, meso and macro levels of the project. To use the words of Mark Cabaj, and Mohamad Yunus, this amounts to developing, not only a bird’s eye view, but also a “worm’s eye view” of community actors in integrated networks (Cabaj, 2011: 139). Part of the definition of rigour for the CFICE project may be determining the extent to which qualitative and quantitative rigour is possible at each of these levels. It also means examining researchers and policy actors as elements in a common ecosystem, and looking at the potential outcomes from the perspective both the highest level of aggregation (such as the country or regional policy level) as well as at the grassroots where local changes happen. At the risk of stretching the metaphor too far, one might also want to consider the meso level of aggregation, the “bee’s eye view” of the Hub coleads, where the community actors are networked, and their knowledge is mobilized for social Community First: Impacts of Community Engagement (CFICE) Rigour in Methods and Evaluation for Community Engagement 3 change and difused across space between many different organizational units. As the Hubs of the CFICE pro-ject act as points where information and academic data can be moved beyond the cellular “hive” level, to the flower level (i.e. where the community organizations are the organic starting point of community activity) it is anticipated that there will be a crossfertilization of ideas. Community organizations will benefit from the Hub leads’ crosspollination efforts as well as being the gen-erative force behind them. Where community organizations are already in full bloom as net-worked organizations, there are ways to strengthen the buzz of activity over the course of the CFICE project in multiple ways with the particular interactivity of the hubs. This metaphor for conceptualizing micro, meso and macro interactions has been used be-fore in various other studies, but especially in the social sciences and geography (Rose, 1997; Reed and Peters, 2004). As shown in these previous studies, an ecological metaphor involves a power dynamic which must be taken into account between researcher and research subject. Such relationships of power are rarely one way or easily predictable. As Reed and Peters elaborate, this is only one of many metaphors that have been used in the past. (If adhering to a true ecologi-cal model, in the broadest sense, ecological theory is social theory because it considers humans as part of ‘nature’ and develops theory from interdisciplinary approaches used in both the social and natural sciences) (Reed and Peters, 2004). This brief document on rigour asks a couple of key questions with respect to methods and evaluation, and seeks to define what constitutes rigour for community based research and com-munity service learning initiatives. The approaches to rigour from these sources may be made more specific given our own research questions in the proposal. The condensed version of these research questions include the following: 1. Scale and replication of community-campus engagement (CCE) 2. CBO definition, evaluation and use of CCE 3. CBO control or shared control in design and implementation of CSL, CBR 4. University governance, evaluation, feedback, course design (which maximizes value for CBOs) 5. Capturing community impacts quantitatively and qualitatively with CBOs 6. Ethical issues in community-campus partnerships Community First: Impacts of Community Engagement (CFICE) Rigour in Methods and Evaluation for Community Engagement 4 Of concern here in this document are primarily questions of scale (macro meso and micro) which are addressed by the 1 research question. Capturing impacts quantitatively and qualitatively with CBOs (the 5 research question) is also of interest when ascertaining the appropriate re-search and evaluation methods. Rigour and the Multiple levels of Quantitative and Qualitative Analysis As Stoecker identifies in his book on research methods for community change, there are several polarities and decision points that often appear when doing community engagement and commu-nity based research. These include choices about: Basic vs. applied research Intensive vs. extensive research The project-based research model: diagnosing, prescribing, implementing, evaluating Reasons to do project-based research: reduce waste, compete for funding, win on advoca-cy issues Ways to get research done: staff, volunteers, academics, students The steps in research: choosing the question, designing the methods collecting the data, analyzing the data, reporting the results Definitions of community, organization, and group (Stoecker, 2005) Just as some of these questions may be discussed together in terms of the overall project vision (at the macro level) they may also be made specific on an individual basis as a researcher goes about his or her problem definition or demonstration project (the micro level). In wrapping these ideas around a core of community based design, research problems may be decidedly qualitative or quantitative. They may also depend on the management structure for the project itself. Stoeck-er’s criteria may thus be modified by the addition of two additional factors: Qualitative / Quantitative Centralized / Decentralized Project Management Community First: Impacts of Community Engagement (CFICE) In the first meetings of CFICE, one of the points of departure agreed upon was the idea that rigour is not necessarily quantitative rigour, and project teams are committed to working with a di-versity of methods according to the needs of each of the Hubs and their community activities. This also implies that at each of the levels of the project, a decision may be made as to whether qualitative and quantitative are best suited to the needs of stakeholders (whether within demon-stration projects, Hub teams, Program, or Steering Committee). This goes for the micro (com-munity organizational level), meso (Hub level), and (Program Committee). One of the interesting ways to visualize this connection is on a spectrum in three dimensions, because few, if any, re-search designs are purely quantitative, qualitative, centralized or decentralized. Our configura-tion of methods and evaluations may thus take any of the following forms, which will shape how the overall CFICE project is evaluated: Moderately convergent: