Engagement in Practice: Engaging a Non-Profit to Facilitate Effective Assessment

Whether a community engagement organization is effective depends upon its ability to articulate its impact, or the extent to which its outcomes can be attributed to its activities or outputs. This ability to effectively articulate its impact is predicated upon the collection and analysis of data associate with the organization’s output. A smaller affiliate of a national non-profit undertook an initiative to improve its ability to articulate its impact—specifically its means for collecting data on its activities. The data will be used to measure the affiliate’s outcomes or the effects of the affiliate’s activities or its outputs. It will then attempt to hold static the effect of other influencers to draw conclusions about the affiliate’s impact. Background (including partnership development) and motivation for project. A smaller affiliate of a national non-profit engages volunteers, including students from a local 28,000 student body university, to provide home repairs and modifications at no cost to low-income homeowners. Affiliates also complete community center rehabilitation projects, playground builds, and support energy efficiency, sustainable community garden, volunteer engagement, and homeowner education projects, all in an effort to sustain communities. The volunteer pool is comprised of skilled trades’ people, the semi-skilled, and the unskilled who are looking for ways to make a difference in their community. The affiliate, like other affiliates, must leverage its status to accomplish its mission. Funding is provided, more often than not, on a competitive basis, by entities such as foundations and other philanthropics, local businesses and other donors, and major corporate partners. Donors, however, want their donations to have an effect or make an impact. In an attempt to show results, recipients of funds as is the case with many non-profits, mistakenly report on their outputs rather than on their impact. The focus of this national non-profit and its affiliates is on preserving affordable homeownership and revitalizing neighborhoods. The population of homeowners served are those most at risk for remaining in their homes and are those in need. Most are at risk and are in need because they are subsisting on a fixed income, whose buying power continues to fall over time, and who live at or below the poverty level. Many also have medical bills that consume much of their income, which forces them to put off needed home repairs and modifications. This population generally includes but is not limited to senior citizens, the disabled, families with children, single parent households, and veterans. Using this as a springboard, the national office administers an annual affiliate survey 1 at the end of each calendar year. The purpose of the survey is to collect data from their affiliates for the past calendar year so the national office can measure and articulate its collective impact. While the affiliate conducts surveys and collects other data, it does so because it has to, like so many other non-profits . Their measures frequently are reasonable estimates more than anything because repairs and modifications for homeowners is the priority given their limited resources. This is also due in part to the lack of the skill sets needed to differentiate between terms like evaluation, assessment, outputs, outcomes, and impact and other resources needed to assess. It is also due to the need to collect data that are needed by the national office that the affiliate may not use. Tuan , as an example, cautions nonprofits to be aware of drowning in data or collecting data because it is easy, measuring things that are not important, asking the wrong questions, and the like. The key according to Tuan is asking why the data are being collected and whether something will be done with the data once it is collected. With this in mind, a Public Service Fellow (PSF) was retain through East Carolina University’s Public Service Fellows program, which was supported by the State Employees Credit Union Foundation. All PSF projects are aimed at community-identified priorities. Fellows might be tasked with exploring community and economic development, grant writing, or project implementation involving discipline-based skills learned through academic programs that range from graphic design to engineering. The Fellows Program includes professional and leadership development components. Specifically, all PSF students attend workshops to teach communication (written as well as oral), business etiquette, networking, and professional practice. The Fellows are asked to read a common text that challenges them to think about the role public service will play in their career. They become familiar with types leadership style through discussions about leadership theory and practice. Hence, the PSF program allows students to develop a better understanding of the collaborative nature of public sector issues that impact community sustainability, growth, and well-being while strengthening core competencies gained through academic programs and experiential learning (S. Paynter, personal communication, February 21, 2017). The PSF student selected for this project was a double major—industrial engineering technology and design. They worked with the affiliate’s executive director; its two AmeriCorps members, whose role was to increase the capacity of their assigned affiliate to serve additional low-income homeowners through volunteer recruitment, client outreach, repairs for low-income homeowners, new project implementation, community partnership development, and program expansion; and a host institution faculty member. Project design and execution. The product for this project was a user friendlier means for collecting and compiling data. The outcome envisioned was a collection of data that better characterizes the affiliate’s outputs, which could be used to articulate the affiliate’s outcomes and impact. To achieve the project’s vision, a data collection interface 4, 5, 6, 7 was produced. It was anticipated that this would improve the response rate, increase accuracy, and make the collection and compilation of data less arduous: all of which are current limitations to the affiliate’s ability to collect and compile data. That is, rather than collecting and compiling data by means of direct entry into a spreadsheet, a survey instrument would be used as the means for compiling the affiliate’s output data. Normally, as it is with classroom test construction, the first order of business when developing a survey is the development of a table of specifications (TOS) . The purpose of a TOS in test construction is to ensure there is alignment between the instructional objectives and the items that appear on the test. This helps to ensure the test results tell whether the instructional objectives are being achieved. Thus with a survey too there must be alignment between the data needed to convey a message and the message that needs to be conveyed. Because the national office requires all its affiliates to report summary data annually, the raw material for the TOS and the resulting survey items were readily available, were digitized, and became part of the data collection tool. However, because the local affiliate also collects data— as an example additional home and homeowner data, a decision was made to include all items in the survey by melding it with the annual affiliate survey items. Because of its availability, Qualtrics , a browser-based survey software, was used to construct the survey instrument. A major concern though was whether the affiliate would always have access to Qualtrics. It was anticipated that as long as adequate documentation was archived, the affiliate could transition to an alternative survey software, such as the ubiquitous SurveyMonkey. A portion of the homeowner application is shown in Figure 1. These data are collected and form the basis for reporting the affiliate’s outputs. A portion of the affiliate’s health and safety priorities is shown in Figure 2. These data too are collected and used to report on the affiliate’s health and safety outputs. An item from the annual affiliate survey appears in Figure 3. This item is used to collect data on the affiliate’s programs. For this affiliate, it included its two programs: Green Housing and Safe at Home (aka Health and Safety). It should be noted that the affiliate reported on its outputs rather than its impact in the Program Accomplishments column. Figure 1. Partial image of the homeowner’s application, repairs to be considered—paper copy. Figure 2. Partial image of the health and safety priorities—paper copy. Figure 3. Program inputs to the 2016 annual affiliate survey. Figure 4. Partial image of the health and safety priorities—Qualtrics. Figure 5. Partial image of the homeowner’s application, repairs to be considered—Qualtrics. A partial image of the Health and Safety section of the Qualtrics survey is shown in Figure 4 and a partial image of the homeowner application is shown in Figure 5. Lessons learned through successes or failures. Like most non-profits, the affiliate knows that, properly done, the collection and analysis of data can yield new knowledge, which can be used to improve its processes. The affiliate also knows it has a responsibility to its community engagement stakeholders and it knows the sustainability of the affiliate depends on effectively assessing their efforts and articulating the results. In response, the affiliate undertook an initiative to improve its ability to articulate its impact—specifically its means for collecting and compiling data on its activities. One of the glaring lessons learned during the course of this project was underestimating the time it would take to formulate the online survey items, even though many of the items had been previously formulated. Another lesson learned was the discovery that far more data, duplicate in some cases, were bei