Category: Evaluation
Implementing, Modeling and Managing a Measurement Culture
| October 8, 2012 | Posted by M. P. under Evaluation, Management, Philanthropy |
Comments off
|
Regardless of the outcome of the upcoming election, the nonprofit social services sector – from mental health clinics to food banks – will still be challenged to meet an increased need with fewer resources and limited funding. Savvy nonprofits have already moved toward an evaluation culture, embracing logic models and short-and-long term impact data to illustrate why (and how) their programs work. Organizational innovation and unique program accomplishments are practically prerequisites for making a successful connection with alternate funding sources, including corporate partnerships, yet nonprofits are still struggling to identify and quantify their impact on clients, the community, and the overall condition they work to modify. Performance measurement, logic model and outcomes are not new or faddish terms, so why the hesitation?
The report, Tough Times, Creative Measures: What Will it Take to Help the Social Sector Embrace an Outcomes Culture? from the Urban Institute, came out of a Fall 2011 event that brought together leaders from the government, nonprofit, philanthropy, and business sectors to discuss the issue of data-driven management in social and human services and the challenges related to successfully utilizing a performance management system. Some of the challenges identified included,
The difficulty of turning away from the organization’s immediate needs to plan and implement a measurement system. No matter how small the agency, the demands on the executive director’s time and talent are immense. Writing up an organization-wide evaluation strategy and implementation plan, including models, indicators, instruments, and data collection plans is an enormous amount of work – and I haven’t mentioned the pilot testing, analysis and reporting aspects. The role of director should be to communicate progress and needs with the board as they guide the agency through this kind of culture change, not create every step of the process.
The reality that sometimes the best outcomes may not be rewarded. Conspiracy theories and snarky excuses aside, well-crafted stories, high profile connections and nonprofits with missions or target audiences that are more interesting or appealing than your own may have an easier time selling their effectiveness. That said, incomplete or inaccurate information on program impact won’t help remedy the situation.
Some nonprofits may be waiting for the trends to flip and the tides to turn. Why move heaven and earth within your organization to embrace a culture that may seem like a phase (especially to long-time employees who have seen edicts from funders come and go). Buy-in for outcomes tracking and reporting may be based on acceptance of the hoop-jumping norms, not the real value of performance measurement to the overall health of the organization. It is time for boards and directors to be brave and commit to an organizational culture change - but be prepared to illustrate how it will be beneficial for staff and (more importantly) clients.
In response to these and other impediments, I mean realities, the symposium attendees identified strategic areas that would have the most impact in encouraging and implementing a data-centered culture: human and financial capital – the tenacity and the tab, creative advocacy – sector giants to back this shift, and ready-to-use systems and tools so directors don’t have to start from square one. How can nonprofit leaders better model and manage a measurement culture? Why are some nonprofits hesitant to embrace this shift?
Getting Acquainted with Evaluation (Careful, it Smells Fear)
| June 13, 2012 | Posted by M. P. under Evaluation, Management, Research |
Comments off
|
This spring I’ve been lucky enough to be working with a colleague on a multi-program evaluation project after an extended absence from the world of outcome measurement. It is a bit like riding a bicycle, in that your never forget HOW to do it, but it seems I did forget the pleasure that is found in working with agency staff as they help inform the evaluation plan and models, assist in identifying key indicators and witness the first round of data come in for review. Each project allows me to get up close and personal with a new nonprofit organization as well as to meet exemplary, dedicated nonprofit professionals at all phases of their careers, but there is something about evaluation that really gets to the essence of a nonprofit. I am, indeed, glad to be back in the measurement mix.
My colleague shared this link with me and because there is so much I love about this succinct, on point article, Six Pieces of Advice to Demystify Evaluation by Johanna Morariu, Director of the Innovation Network, I wanted to post on it rather than just send the link off into the tweetosphere.
No matter where your organization is in the evaluation (or for that matter strategic) planning process, start making data collection your friend. Immediately. It’s not going away (ever), there are more tools than ever before to help with it, and even if you hire an outside firm to conduct your evaluation – eventually their contract ends and it falls to your organization to sustain it. Don’t spend a dime on a contract or software until you know you will be able to do so. Not to worry though, a thorough consultant involves you and your staff in each step of the process and will provide the necessary technical assistance during the transition to ensure you will be able to take over the reins.
So, feel free to make eye contact with and extend a hand to that evaluation. Soon, when you are knee deep in useful data for your board, clients, funders and community supporters you won’t be able to remember life without it.
Evaluation Shows Effectiveness of Drug Courts in Reducing Drug Use, Criminal Behavior
| June 9, 2012 | Posted by M. P. under Drug and Alcohol, Evaluation, Policy, Program Model |
Comments off
|
Having their start in the late 1980’s and gaining in popularity since, drug courts are court treatment programs that target criminal defendants, juvenile offenders, and/or parents involved in the child welfare system who have alcohol and drug addiction and dependency issues. There are now approximately 2,600 drug courts operating in the United States, with 50 percent of them exclusively for adult offenders.
The December 2011 report, The Multi-Site Adult Drug Court Evaluation, by Shelli B. Rossman, John K. Roman, Janine M. Zweig, Michael Rempel and Christine H. Lindquist presents the findings of an extensive evaluation of the nation’s drug courts. The study examined the successful drug courts in reducing drug use and criminal activity among adult participants while having a positive impact on their lives in other ways. Key findings include,
- Yes, drug courts resulted in statistically significant reductions in relapse by participants. Compared to the group not on the specialized drug court track, participants were significantly less likely report any drug use (76 percent compared to 56 percent) in the past year (at the 1.5 year follow-up point). Also, fewer drug court participants tested positive for illegal drugs (29 versus 46 percent).
- Yes, drug courts resulted in a significant decrease in criminality of participants. Court patrons were significantly less likely than the comparison group to report committing crimes (40 versus 53 percent) in the year prior to the 1.5 year follow up contact. In fact, participants were also significantly less likely to report committing any crime at all at the six- month and the 18-month follow-up. Perhaps due to the nature the intervention, the researchers also found that drug court participation specifically reduced the crimes of drug possession, drug sales offenses, driving while intoxicated, and property related crime.
- Yes, members of the drug court sample did experience some positive personal outcomes outside of both reduced drug/alcohol use and criminality. Data from the 1.5 year follow up interviews indicated that drug court participants were significantly less likely than those in the comparison group to report an employment, education, or financial service need, and reported less family conflict. There were no differences between the groups for self-reported symptoms of depression or homelessness.
The evaluation report, downloadable in 4 parts, is available via the National Institute of Justice website.
With evidence that drug courts are working better than the traditional justice system with these type of offenders, support of their use and expansion makes good academic, practical and fiscal sense. Information about the Allegheny County Drug Court (that has been in operation for nearly 15 years) is available at the Allegheny County Department of Human services website.
Mixed Results from Charter School Impact Research
| January 31, 2012 | Posted by M. P. under Education, Evaluation, Policy |
Comments off
|
The latest findings from an ongoing study of the effectiveness of charter schools on student achievement indicate some are having a positive impact on student graduation rate and college enrollment.
The report Charter-School Management Organizations: Diverse Strategies and Diverse Student Impacts, conducted by Mathematica and the Center on Reinventing Public Education (CRPE) -University of Washington looked at the effect of charter school management organizations (CMOs) on test scores, graduation and post-secondary education planning. While the data indicate a positive impact on high school graduation statistics (for schools with that data), the overall impact on student academic achievement by school varied considerably.
The study also examined characteristics of CMO schools in relation to positive educational impacts, finding:
- high levels of teacher coaching were associated with positive impacts on academic achievement;
- performance-based teacher pay structure was not statistically associated with student achievement; and
- class size was not statistically associated with student achievement.
Previous reports from National Study of Charter Management Organization (CMO) Effectiveness are available online (as is this newest report) via the Mathematica website.
What Should NonprofitsThink About in 2012?
| December 29, 2011 | Posted by M. P. under Budget, Evaluation, Management, Technology |
Although there are glimmers of hope that things will soon “get back to normal,” my thoughts on what nonprofits should watch for in 2012 are based on the premise that no matter how much we wish it to be so, there isn’t any returning to a pre-recession world. The landscape of the sector is changing – has changed already – due to both crisis and innovation. Unfortunately, the bleak funding outlook has muted any excitement over the adoption of technologies that have resulted in more and better data visualization and engagement across sector boundaries.
These times call for nonprofits to stay true to their mission but to let go of the past, and face 2012 eager, nimble and fearless.
Showcasing Your Data
Look for the once cutting-edge trend of infographics to become even more widespread, if not the unofficial standard, in nonprofit marketing and reporting materials, while large, data-rich organizations make use of web-based interactive viewers to share information with stakeholders.
Annual reports will be stripped down and simplified, (hopefully) resulting in lower costs while allowing for a far more striking presentation of data. That said, simplified doesn’t mean simple, as telling a story via numbers, headings and brightly hued graphics may present more of a challenge than the emotive narratives we are so used to cranking out. Save those stories, including videos, to feature on your webpage and YouTube channel, and as content for e-newsletters.
The Social Media Mini-Backlash
I have heard the murmurs among nonprofit professionals that social media isn’t working as promised, or is (still) too nebulous to play a noticeable role in an organization’s strategy. It is a criticism that is due – it is even happening in the for-profit sector. After a few years of listening to experts extol the benefits of social media for nonprofits, executives want to see impact via the bottom line, not the weekly number of followers, likes or re-tweets. So, cue the new meme: perhaps nonprofits shouldn’t put SO many of their eggs in the social media basket and instead concentrate on tried and true traditional methods of communications and fundraising.
Well, before you cancel your Youtube, Twitter, Facebook and Constant Contact accounts and run to the mimeograph machine…
Social media utilization is, no must be, part of the organizational culture, modeled from the top down and integrated across departments/sites/states/etc. Take a pause in early 2012 to revisit your strategy and social media’s place in it. Work in small feedback loops so that you have the ability to experiment with the technology and messaging methods to find what works best for you, but do not, please do not let your accounts go silent.
There is not going to be a hot new trend that people will abandon it for, social media will change and diversify, but it will not go away. Frankly, it should be utilized more and in tandem with email and other communications, marketing and fundraising tools.
Shifting Structures
Fiscal constraints may have finally done what years of cautionary tales of nonprofits inadvertently drifting into the silo mentality could not: the merging of departments, and likewise, goals, strategies and priorities. Since they were already changing the organizational charts, some also tweaked their traditional hierarchies to those with more horizontal communication.
The move toward the non-hierarchical models will continue, particularly in new nonprofits. In preparing this post, I was surprised to find that many service and advocacy organizations highlight their non-traditional organizational structures in copy and hiring pitches. This slow but steady shift is a major managerial trend to watch over the next decade. Will established agencies, leaders and board members accept a team-based linear model featuring open dialogue, collaborative efforts, and transparency?
In 2012 we may also see social media have a larger internal presence in nonprofits (especially large, multi-site organizations) as a way for leadership, staff and volunteers to share information, collaborate on ideas and work as communities or teams connected to the larger hub. The possibilities of such a concept are laid out by Shawn Graham in his piece Using Social Media To Improve Employee Communication, Collaboration, And Even Compensation (at the Fast Company Blog) about Shopify’s UNICORN, their on-site social network.
Everything Evidence-Based
Funder demand for anything deemed evidence-based continues to grow, although the misuse of that term and its direct linkage to program funds have caused it to be viewed with suspicion by some in the sector. With nonprofits already cutting budgets, future government funding in jeopardy, and the need for services on the rise, expect accountability to center on realistic expectations of what can and should be collected, analyzed and reported in a timely manner.
Foundations will continue to the lead the way in improving evaluation capacity in nonprofits, encouraging it to become part of the culture rather than handled by one department or only utilized for grant-funded projects. Luckily, this role also provides them with the opportunity to learn of any misapplication of evidence-based programs – those used in an environment and/or with a target population divergent from that which was intended, or watered down to make service implementation quicker and cheaper. Service agencies and foundations have a responsibility to the community and the sector to guard against this practice.
Sector Overlap and A(nother?) Leadership Gap
Nonprofits will find increased competition for the next generation of passionate, service-focused leaders as the popularity of corporate social responsibility programs and non-government organizations (NGOs), and opportunities for social entrepreneurs continue to grow. We have only scratched the surface in exploring potential collaboration and funding patterns, and future downturns or austerity measures will keep this topic at the forefront of planning discussions. While young talent may have gotten their feet wet in the traditional nonprofit sector through volunteering, internships or entry-level positions, the sectors are blending, and it is no longer the only way to work for social change.
The 2011 Daring to Lead study reported that 58 percent of nonprofit executives felt depleted by interactions with government funders. Would it really be a surprise, then, if up-and-coming nonprofit stars choose against the continuous struggle for government funding and opted for a leadership role at privately backed or hybrid programs?
One last note on what to watch in 2012: keep an eye on the Occupy movement. If their applications for nonprofit status are approved, they become part of the sector, a peer, and potential collaborator or competitor for donor attention and other resources
More Evidence of the Value of Pre-K Programs
| November 13, 2011 | Posted by M. P. under Budget, Education, Evaluation |
Comments off
|
Though empirically associated with better educational outcomes and considered by many policymakers to be key to academic success, early childhood education is in danger of being diluted or cut competently from budgets as funding becomes scarce.
A new report from the The Center for Public Education should be required reading for school board members, parents of young children and early childhood education professionals as it provides additional evidence of the benefit of pre-Kindergarten (Pre-K) attendance on future academic performance. The study findings suggest:
- Children who attended Pre-K and half-day kindergarten were more likely to have higher third grade reading skills scores than children who attended only full-day kindergarten, without Pre-K.
- The higher the level of reading skill examined (above basic), the larger the likelihood of students who attended Pre-K/half-day kindergarten, as opposed to only full-day kindergarten, reaching that level.
- The impact of the Pre-K/half-day kindergarten combination was significantly greater for some when the sample data was examined by race, ethnicity and family income. Overall, the impact was greatest for Hispanic students, Black students, students below the poverty level and English-learning students.
- The educational attainment of the mother has an impact on the reading level achievement of the student.
The report, Starting Out Right by Jim Hull, a Senior Policy Analyst at the Center for Public Education, is available on their website.
Recent Comments