Process Metrics are not Outcomes

This is an unexpected follow-up to my last post.  I just heard about a nonprofit losing a hefty grant at renewal time due primarily to a lack of reported outcomes.  There was measurement – lots of data on process and organizational performance metrics – but not much to demonstrate the difference the program made in the lives of participants.  This kind of news is disheartening.

My first thought is – how did it get to that point?  Were the grant terms a surprise sprung on an unsuspecting organization at the last moment?  Was any mention of measuring program outcomes waved off by executives who preferred to discuss ways to scale up at the next funding cycle?

Probably not.

That said, I am pretty certain that…

  • the funder/s made their reporting criteria and protocol clear;
  • the program administration and staff were dedicated to their mission and conducted outreach and activities according to their model;
  • people who experienced the program gained something from it;
  • the nonprofit thought that they were collecting data that showed the impact they made on participants and in the community.

So what went wrong in that story I heard?  I’ll never know.  No one accepts a grant award with the expectation of a giant hole in their final report, but if there are questions about program application, geographic distance between sites, and/or irregular communication, measurement can and will get lost in the shuffle. Here are some steps you can take to prevent a similar situation from happening to your organization.

  1. Update your data collection plan. The outcomes listed in a column on a chart in your program materials will not measure themselves.  What are you currently collecting that may also fit as an indicator of your expected results?  Can you create a measure to better capture a specific change expected in the program participants?
  2. Make expectations clear and follow up regularly. Keep staff up-to-date on data collection with a matrix that lays out indicators, data sources, person(s) responsible and timeline.  Have a check-in call monthly to report on progress and address questions and other issues around the collection.
  3. Have patience. It will take a while to get used to a shift from collecting process metrics (still important – don’t stop doing that) to outcomes data,  But, if you have a plan ready to go you can work out any knots early on in the funding period rather than panic at report time.

Collaboration – Everybody’s Doing It

Here’s your one word resolution for 2015 – collaborate. If you already are, do it strategically and more often. If you aren’t, you are missing out on a highly adaptable, relatively low-cost way to increase impact. A December 2014  study from the Bridgespan Group, in conjunction with The Lodestar Foundation, found that collaboration isn’t just a popular topic in the nonprofit sector – it’s actually happening and it’s working well for the majority of players.

Highlights from the study,

Collaboration is happening.  The trend is real.  Over 90 percent of the nonprofit leaders surveyed had participated in one of the forms of collaboration examined by the study (associations, joint programs, shared support functions, and mergers) within the last three years,  with 54 percent participating in at least two forms.  The majority (93 percent) of nonprofit executives expect to become involved in additional collaboration during 2015.

People in the sector like it. The majority (over 70 percent) of nonprofit executives described the collaborations they participated in as successful. Only a small percentage of each type of collaboration did not achieve their intended goals, according to respondent ratings.

People in the sector intend to do more of it. Both nonprofit executives and foundations reported their intention to do more collaboration in the future. Funders want to see more collaboration in the sector, specifically shared support functions (76 percent) and mergers (55 percent).

Additional findings, including the very real challenges facing quality collaboration, are included in the brief Making Sense of Nonprofit Collaborations by Alex Neuhoff, Katie Smith Milway, Reilly Kiernan, and Josh Grehan, available at the Bridgespan Group website.

One note of caution. Before you get the urge to start trimming programs also offered by peer organizations or make merger your “word for 2015,”  check out the article Again, Nonprofit Mergers are no Cure All at Nonprofit Quarterly.  Collaboration takes many forms and not all may be the best fit for your mission, constituency or bottom line. If collaboration is your resolution for 2015 then, as with all resolutions, start slowly, research what will work best for you, and keep at it.

Survey Suggests We’re Not Talking about Domestic Violence and Sexual Abuse

A study commissioned by the Avon Foundation for Women on the experiences and perceptions of domestic violence and sexual abuse found a lack of discussion and action on these issues by both teenagers and adults.

Data from the study, NO MORE Domestic Violence and Sexual Assault, Survey of Attitudes and Experiences of Teens and Adults  indicate that respondents felt these issues were important conceptually, but not much attention was given to them through words or actions, for example,

  • 60 percent of women and 75 percent of men had not discussed the topic of domestic violence with friends
  • 73 percent of parents with children under age 18 had not discussed the topic of sexual assault with their children
  • 15 percent of respondents felt that sexual abuse or domestic violence were problems among their friends
  • The majority of both male and female victims of domestic violence who had told someone about their situation reported that no one helped them

The Avon Foundation for Women plans to use this data to inform a new initiative to better train employers on the signs of domestic or sexual abuse and how to best support those who have experienced it. As the cost of domestic abuse in health care, mental health services and lost productivity amounts to billions of dollars each year, a scalable strategy to connect companies with local professionals to improve response and prevention efforts for families experiencing such crises is a step in the right direction.

Fewer Summer Jobs for Teens Fits Trend of Declining Youth Employment

Summertime employment has traditionally been seen as a rite of passage, a builder of character and a source of funding for teenage frolic, but even a part-time job serving burgers or minding the retail racks isn’t easy to come by these days.  In 2000, the average summer employment rate was nearly 52 percent, dropping to 30 percent last year.  In 2012, just a quarter  of teens reported having a paying job.*

The report, The Dismal State of the Nation’s Teen Summer Job Market, 2008-2012, and the Outlook for the Summer of 2013, from the Center for Labor and Market studies at Northeastern University details the decline in teenage employment rates and the weakness of the current job market and what that means for young adults.  A key take-away from the report is that household income was a better predictor of youth employment than race.  Low income youth were least likely to be working. As family income rose, so did teen employment rates, with 21 percent of youth  from households earning under $20,000  reporting summer employment compared to 38 percent from households earning between $100,000 and $150,000 a year.

The sad state of teen summer employment isn’t surprising considering the decline of the overall youth employment rate.  A policy report from the Annie E. Casey Foundation on the growing number of  teens and young adults both unemployed and not in school –  referred to as disconnected youth in the brief – found that such youth were most likely to be from low income families. Specifically,

  • 21 percent of low income (under $20,000/household) 16-to-19 year old youth were disconnected compared to 8 percent of their counterparts in families with an income over $100,000; and
  •  among 20-to 24-year-olds from low income families, 30 percent were not in school or employed,compared  to 10 percent of those from families earning $100,000 or more.

The Pennsylvania employment rate for young adults (20 to 24 years old) is approximately 62 percent, for teens 16 to 19, 39 percent.

The authors of the report Youth and Work: Restoring Teen and Young Adult Connections to Opportunitychallenge policy makers to find cooperative cross-system approaches to reconnecting and reengaging youth with education and employment opportunities.  An approach that is flexible enough to use the strengths of the community where it operates but based in proven outreach and engagement strategies that go beyond mere job-matching  might have a chance, if the funding survives.

If you are interested in learning more about models of youth employment initiatives check out  Best Practices for Youth Employment Programs:A Synthesis of Current Research from What Works, Wisconsin.




*Source: Andrew Sum, Ishwar Khatiwada, Walter McHugh, and Sheila Palma, The Dismal State of the Nation’s Teen Summer Job Market, 2008-2012, and the Outlook for the Summer of 2013, Center for Labor Market Studies, Northeastern University, May 2013.