Process Metrics are not Outcomes
|August 26, 2016||Posted by M. P. under Evaluation, Management, Program Model||
This is an unexpected follow-up to my last post. I just heard about a nonprofit losing a hefty grant at renewal time due primarily to a lack of reported outcomes. There was measurement – lots of data on process and organizational performance metrics – but not much to demonstrate the difference the program made in the lives of participants. This kind of news is disheartening.
My first thought is – how did it get to that point? Were the grant terms a surprise sprung on an unsuspecting organization at the last moment? Was any mention of measuring program outcomes waved off by executives who preferred to discuss ways to scale up at the next funding cycle?
That said, I am pretty certain that…
- the funder/s made their reporting criteria and protocol clear;
- the program administration and staff were dedicated to their mission and conducted outreach and activities according to their model;
- people who experienced the program gained something from it;
- the nonprofit thought that they were collecting data that showed the impact they made on participants and in the community.
So what went wrong in that story I heard? I’ll never know. No one accepts a grant award with the expectation of a giant hole in their final report, but if there are questions about program application, geographic distance between sites, and/or irregular communication, measurement can and will get lost in the shuffle. Here are some steps you can take to prevent a similar situation from happening to your organization.
- Update your data collection plan. The outcomes listed in a column on a chart in your program materials will not measure themselves. What are you currently collecting that may also fit as an indicator of your expected results? Can you create a measure to better capture a specific change expected in the program participants?
- Make expectations clear and follow up regularly. Keep staff up-to-date on data collection with a matrix that lays out indicators, data sources, person(s) responsible and timeline. Have a check-in call monthly to report on progress and address questions and other issues around the collection.
- Have patience. It will take a while to get used to a shift from collecting process metrics (still important – don’t stop doing that) to outcomes data, But, if you have a plan ready to go you can work out any knots early on in the funding period rather than panic at report time.