|October 8, 2012||Posted by M. P. under Evaluation, Management, Philanthropy||
Regardless of the outcome of the upcoming election, the nonprofit social services sector – from mental health clinics to food banks – will still be challenged to meet an increased need with fewer resources and limited funding. Savvy nonprofits have already moved toward an evaluation culture, embracing logic models and short-and-long term impact data to illustrate why (and how) their programs work. Organizational innovation and unique program accomplishments are practically prerequisites for making a successful connection with alternate funding sources, including corporate partnerships, yet nonprofits are still struggling to identify and quantify their impact on clients, the community, and the overall condition they work to modify. Performance measurement, logic model and outcomes are not new or faddish terms, so why the hesitation?
The report, Tough Times, Creative Measures: What Will it Take to Help the Social Sector Embrace an Outcomes Culture? from the Urban Institute, came out of a Fall 2011 event that brought together leaders from the government, nonprofit, philanthropy, and business sectors to discuss the issue of data-driven management in social and human services and the challenges related to successfully utilizing a performance management system. Some of the challenges identified included,
The difficulty of turning away from the organization’s immediate needs to plan and implement a measurement system. No matter how small the agency, the demands on the executive director’s time and talent are immense. Writing up an organization-wide evaluation strategy and implementation plan, including models, indicators, instruments, and data collection plans is an enormous amount of work – and I haven’t mentioned the pilot testing, analysis and reporting aspects. The role of director should be to communicate progress and needs with the board as they guide the agency through this kind of culture change, not create every step of the process.
The reality that sometimes the best outcomes may not be rewarded. Conspiracy theories and snarky excuses aside, well-crafted stories, high profile connections and nonprofits with missions or target audiences that are more interesting or appealing than your own may have an easier time selling their effectiveness. That said, incomplete or inaccurate information on program impact won’t help remedy the situation.
Some nonprofits may be waiting for the trends to flip and the tides to turn. Why move heaven and earth within your organization to embrace a culture that may seem like a phase (especially to long-time employees who have seen edicts from funders come and go). Buy-in for outcomes tracking and reporting may be based on acceptance of the hoop-jumping norms, not the real value of performance measurement to the overall health of the organization. It is time for boards and directors to be brave and commit to an organizational culture change - but be prepared to illustrate how it will be beneficial for staff and (more importantly) clients.
In response to these and other impediments, I mean realities, the symposium attendees identified strategic areas that would have the most impact in encouraging and implementing a data-centered culture: human and financial capital – the tenacity and the tab, creative advocacy – sector giants to back this shift, and ready-to-use systems and tools so directors don’t have to start from square one. How can nonprofit leaders better model and manage a measurement culture? Why are some nonprofits hesitant to embrace this shift?
|June 13, 2012||Posted by M. P. under Evaluation, Management, Research||
This spring I’ve been lucky enough to be working with a colleague on a multi-program evaluation project after an extended absence from the world of outcome measurement. It is a bit like riding a bicycle, in that your never forget HOW to do it, but it seems I did forget the pleasure that is found in working with agency staff as they help inform the evaluation plan and models, assist in identifying key indicators and witness the first round of data come in for review. Each project allows me to get up close and personal with a new nonprofit organization as well as to meet exemplary, dedicated nonprofit professionals at all phases of their careers, but there is something about evaluation that really gets to the essence of a nonprofit. I am, indeed, glad to be back in the measurement mix.
My colleague shared this link with me and because there is so much I love about this succinct, on point article, Six Pieces of Advice to Demystify Evaluation by Johanna Morariu, Director of the Innovation Network, I wanted to post on it rather than just send the link off into the tweetosphere.
No matter where your organization is in the evaluation (or for that matter strategic) planning process, start making data collection your friend. Immediately. It’s not going away (ever), there are more tools than ever before to help with it, and even if you hire an outside firm to conduct your evaluation – eventually their contract ends and it falls to your organization to sustain it. Don’t spend a dime on a contract or software until you know you will be able to do so. Not to worry though, a thorough consultant involves you and your staff in each step of the process and will provide the necessary technical assistance during the transition to ensure you will be able to take over the reins.
So, feel free to make eye contact with and extend a hand to that evaluation. Soon, when you are knee deep in useful data for your board, clients, funders and community supporters you won’t be able to remember life without it.
|June 9, 2012||Posted by M. P. under Drug and Alcohol, Evaluation, Policy, Program Model||
Having their start in the late 1980’s and gaining in popularity since, drug courts are court treatment programs that target criminal defendants, juvenile offenders, and/or parents involved in the child welfare system who have alcohol and drug addiction and dependency issues. There are now approximately 2,600 drug courts operating in the United States, with 50 percent of them exclusively for adult offenders.
The December 2011 report, The Multi-Site Adult Drug Court Evaluation, by Shelli B. Rossman, John K. Roman, Janine M. Zweig, Michael Rempel and Christine H. Lindquist presents the findings of an extensive evaluation of the nation’s drug courts. The study examined the successful drug courts in reducing drug use and criminal activity among adult participants while having a positive impact on their lives in other ways. Key findings include,
- Yes, drug courts resulted in statistically significant reductions in relapse by participants. Compared to the group not on the specialized drug court track, participants were significantly less likely report any drug use (76 percent compared to 56 percent) in the past year (at the 1.5 year follow-up point). Also, fewer drug court participants tested positive for illegal drugs (29 versus 46 percent).
- Yes, drug courts resulted in a significant decrease in criminality of participants. Court patrons were significantly less likely than the comparison group to report committing crimes (40 versus 53 percent) in the year prior to the 1.5 year follow up contact. In fact, participants were also significantly less likely to report committing any crime at all at the six- month and the 18-month follow-up. Perhaps due to the nature the intervention, the researchers also found that drug court participation specifically reduced the crimes of drug possession, drug sales offenses, driving while intoxicated, and property related crime.
- Yes, members of the drug court sample did experience some positive personal outcomes outside of both reduced drug/alcohol use and criminality. Data from the 1.5 year follow up interviews indicated that drug court participants were significantly less likely than those in the comparison group to report an employment, education, or financial service need, and reported less family conflict. There were no differences between the groups for self-reported symptoms of depression or homelessness.
With evidence that drug courts are working better than the traditional justice system with these type of offenders, support of their use and expansion makes good academic, practical and fiscal sense. Information about the Allegheny County Drug Court (that has been in operation for nearly 15 years) is available at the Allegheny County Department of Human services website.
|January 31, 2012||Posted by M. P. under Education, Evaluation, Policy||
The latest findings from an ongoing study of the effectiveness of charter schools on student achievement indicate some are having a positive impact on student graduation rate and college enrollment.
The report Charter-School Management Organizations: Diverse Strategies and Diverse Student Impacts, conducted by Mathematica and the Center on Reinventing Public Education (CRPE) -University of Washington looked at the effect of charter school management organizations (CMOs) on test scores, graduation and post-secondary education planning. While the data indicate a positive impact on high school graduation statistics (for schools with that data), the overall impact on student academic achievement by school varied considerably.
The study also examined characteristics of CMO schools in relation to positive educational impacts, finding:
- high levels of teacher coaching were associated with positive impacts on academic achievement;
- performance-based teacher pay structure was not statistically associated with student achievement; and
- class size was not statistically associated with student achievement.
Previous reports from National Study of Charter Management Organization (CMO) Effectiveness are available online (as is this newest report) via the Mathematica website.
|November 13, 2011||Posted by M. P. under Budget, Education, Evaluation||
Though empirically associated with better educational outcomes and considered by many policymakers to be key to academic success, early childhood education is in danger of being diluted or cut competently from budgets as funding becomes scarce.
A new report from the The Center for Public Education should be required reading for school board members, parents of young children and early childhood education professionals as it provides additional evidence of the benefit of pre-Kindergarten (Pre-K) attendance on future academic performance. The study findings suggest:
- Children who attended Pre-K and half-day kindergarten were more likely to have higher third grade reading skills scores than children who attended only full-day kindergarten, without Pre-K.
- The higher the level of reading skill examined (above basic), the larger the likelihood of students who attended Pre-K/half-day kindergarten, as opposed to only full-day kindergarten, reaching that level.
- The impact of the Pre-K/half-day kindergarten combination was significantly greater for some when the sample data was examined by race, ethnicity and family income. Overall, the impact was greatest for Hispanic students, Black students, students below the poverty level and English-learning students.
- The educational attainment of the mother has an impact on the reading level achievement of the student.