Getting Acquainted with Evaluation (Careful, it Smells Fear)

This spring I’ve been lucky enough to be working with a colleague on a multi-program evaluation project after an extended absence from the world of outcome measurement. It is a bit like riding a bicycle, in that your never forget HOW to do it, but it seems I did forget the pleasure that is found in working with agency staff as they help inform the evaluation plan and models, assist in identifying key indicators and witness the first round of data come in for review. Each project allows me to get up close and personal with a new nonprofit organization  as well as to meet exemplary, dedicated nonprofit professionals at all phases of their careers, but there is something about evaluation that really gets to the essence of a nonprofit.  I am, indeed, glad to be back in the measurement mix.

My colleague shared this link with me and because there is so much I love about this succinct, on point article, Six Pieces of Advice to Demystify Evaluation by Johanna Morariu, Director of the Innovation Network, I wanted to post on it rather than just send the link off into the tweetosphere.

No matter where your organization is in the  evaluation (or for that matter strategic) planning process,  start making data collection your friend.  Immediately. It’s not going away (ever), there are more tools than ever before to help with it,  and even if you hire an outside firm to conduct your evaluation – eventually their contract ends and it falls to your organization to sustain it.  Don’t spend a dime on a contract or software until you know you will be able to do so.  Not to worry though, a thorough consultant involves you and your staff  in each step of the process and will provide the necessary technical assistance during the transition to ensure you will be able to take over the reins.

So, feel free to make eye contact with and extend a hand to that evaluation. Soon, when you are knee deep in useful data for your board, clients, funders and community supporters you won’t be able to remember life without it.

Program Aimed at Dropouts Worth the Investment

A recent study from the  RAND Corporation on the effectiveness of the National Guard Youth ChalleNGe, a residential and mentoring program aimed at 16 and 17 year old high school dropouts,  found the intervention to be both cost effective and beneficial to the youth cadets.

A rigorous evaluation of the program (that operates in over half of the states of the country) found substantial long-term benefits to the program participants including a high level of educational attainment and the related social and financial rewards. A cost-benefit analysis of Youth ChalleNGe estimated that it provided $2.66 worth of social benefits for each $1.00 invested in the program. The report did not identify many long-term benefits to the larger community (such as reduced criminal activity) associated with the program, however, some may be inferred from the significant impact on the individual participants.

The full report, A Cost-Benefit Analysis of the National Guard Youth ChalleNGe Program by Francisco Perez-Arce, Louay Constant, David S. Loughran and Lynn A. Karoly, and a one-page summary of the findings are available at the RAND website.



What Should NonprofitsThink About in 2012?

Although there are glimmers of hope that things will soon “get back to normal,” my thoughts on what nonprofits should watch for in 2012 are based on the premise that no matter how much we wish it to be so, there isn’t any returning to a pre-recession world. The landscape of the sector is changing – has changed already – due to both crisis and innovation. Unfortunately, the bleak funding outlook has muted any excitement over the adoption of technologies that have resulted in more and better data visualization and engagement across sector boundaries.

These times call for nonprofits to stay true to their mission but to let go of the past, and face 2012 eager, nimble and fearless.


Showcasing Your Data

Look for the once cutting-edge trend of infographics to become even more widespread, if not the unofficial standard, in nonprofit marketing and reporting materials, while large, data-rich organizations make use of web-based interactive viewers to share information with stakeholders.

Annual reports will be stripped down and simplified, (hopefully) resulting in lower costs while allowing for a far more striking presentation of data. That said, simplified doesn’t mean simple, as telling a story via numbers, headings and brightly hued graphics may present more of a challenge than the emotive narratives we are so used to cranking out. Save those stories, including videos, to feature on your webpage and YouTube channel, and as content for e-newsletters.


The Social Media Mini-Backlash

I have heard the murmurs among nonprofit professionals that social media isn’t working as promised, or is (still) too nebulous to play a noticeable role in an organization’s strategy. It is a criticism that is due – it is even happening in the for-profit sector. After a few years of listening to experts extol the benefits of social media for nonprofits, executives want to see impact via the bottom line, not the weekly number of followers, likes or re-tweets. So, cue the new meme: perhaps nonprofits shouldn’t put SO many of their eggs in the social media basket and instead concentrate on tried and true traditional methods of communications and fundraising.

Well, before you cancel your Youtube, Twitter, Facebook and Constant Contact accounts and run to the mimeograph machine…

Social media utilization is, no must be, part of the organizational culture, modeled from the top down and integrated across departments/sites/states/etc. Take a pause in early 2012 to revisit your strategy and social media’s place in it. Work in small feedback loops so that you have the ability to experiment with the technology and messaging methods to find what works best for you, but do not, please do not let your accounts go silent.

There is not going to be a hot new trend that people will abandon it for, social media will change and diversify, but it will not go away.  Frankly, it should be utilized more and in tandem with email and other communications, marketing and fundraising tools.


Shifting Structures

Fiscal constraints may have finally done what years of cautionary tales of nonprofits inadvertently drifting into the silo mentality could not: the merging of departments, and likewise, goals, strategies and priorities. Since they were already changing the organizational charts, some also tweaked their traditional hierarchies to those with more horizontal communication.

The move toward the non-hierarchical models will continue, particularly in new nonprofits. In preparing this post, I was surprised to find that many service and advocacy organizations highlight their non-traditional organizational structures in copy and hiring pitches. This slow but steady shift is a major managerial trend to watch over the next decade. Will established agencies, leaders and board members accept a team-based linear model featuring open dialogue, collaborative efforts, and transparency?

In 2012 we may also see social media have a larger internal presence in nonprofits (especially large, multi-site organizations) as a way for leadership, staff and volunteers to share information, collaborate on ideas and work as communities or teams connected to the larger hub.  The possibilities of such a concept are laid out by Shawn Graham in his piece Using Social Media To Improve Employee Communication, Collaboration, And Even Compensation (at the Fast Company Blog) about Shopify’s UNICORN, their on-site social network.


Everything Evidence-Based 

Funder demand for anything deemed evidence-based continues to grow, although the misuse of that term and its direct linkage to program funds have caused it to be viewed with suspicion by some in the sector. With nonprofits already cutting budgets, future government funding in jeopardy, and the need for services on the rise, expect accountability to center on realistic expectations of what can and should be collected, analyzed and reported in a timely manner.

Foundations will continue to the lead the way in improving evaluation capacity in nonprofits, encouraging it to become part of the culture rather than handled by one department or only utilized for grant-funded projects. Luckily, this role also provides them with the opportunity to  learn of any misapplication of evidence-based programs – those used in an environment and/or with a target population divergent from that which was intended, or watered down to make service implementation quicker and cheaper. Service agencies and foundations have a responsibility to the community and the  sector to guard against this practice.


Sector Overlap and A(nother?) Leadership Gap

Nonprofits will find increased competition for the next generation of passionate, service-focused  leaders as the popularity of corporate social responsibility programs and non-government organizations (NGOs), and opportunities for social entrepreneurs continue to grow. We have only scratched the surface in exploring potential collaboration and funding patterns, and future downturns or austerity measures will keep this topic at the forefront of planning discussions. While young talent may have gotten their feet wet in the traditional nonprofit sector through volunteering, internships or entry-level positions, the sectors are blending, and it is no longer the only way to work for social change.

The 2011 Daring to Lead study reported that 58 percent of nonprofit executives felt depleted by interactions with government funders. Would it really be a surprise, then, if up-and-coming nonprofit stars choose against the continuous struggle for government funding and opted for a leadership role at privately backed or hybrid programs?

One last note on what to watch in 2012: keep an eye on the Occupy movement. If their applications for nonprofit status are approved, they become part of the sector, a peer, and potential collaborator or competitor for donor attention and other resources

Individualized Services, Lower Caseloads & Higher Adoption Rates: The Wendy’s Wonderful Kids Program Evaluation Report

The Dave Thomas Foundation for Adoption (DTFA) and its Wendy’s Wonderful Kids initiative strive to increase adoptions of foster children in the United States and some provinces in Canada. A recent evaluation of the program found that it is more successful in attaining the goal of adoption than traditional casework and adoption models.

The Wendy’s Wonderful Kids (WWK) program, funded partially from donations from Wendy’s restaurants, administers grants through DTFA to adoption agencies across the country who then work with local professionals to place children in the foster care system with adoptive families. The process is highly individualized, with much attention paid to getting to know the children and placing them with recruited, well-vetted families according to the history, strengths and unique needs of the youth. In this model, WWK staff carry a caseload of 12 to 15 children.

The evaluation found that overall; the youth in the WWK program were more likely to be adopted (1.5 times more) than youth outside the program and those who had mental health diagnoses were three times more likely to be adopted than their counterparts in the control group. Also, as the age of the child increased, so did the likelihood that they would be adopted compared to youth receiving traditional services; for example, the report states that youth entering the program at age 11 were twice as likely to be adopted and those referred at age 15 were three times as likely to be adopted. Taking this data at face value it appears the model used by the WWK works exceptionally well with youth sometimes considered more challenging to place in adoptive families (those with special needs, teenagers, etc.).

Demographics of the intervention group, a more detailed breakdown of the findings and information on the WWK program model are available in the Evaluation Report Summary and other evaluation materials at the Child Trends website. A fact sheet of the findings is also available.

Citation of Evaluation Report: Malm, K., Vandivere, S., Allen, T., DeVooght, K., Ellis, R., McKlindon, A., Smollar, J., Williams, E. and Zinn, A. (2011). Evaluation Report Summary: The Wendy’s Wonderful Kids Initiative, Child Trends, Washington, D.C.