Where Should We Start in Using Evaluation as a Tool for Learning?

March 20, 2014

Learning and Evaluation

More in the Playbook:

Many grantmakers agree that an increased focus on evaluation and learning can help us tease out insights on our true impact and how we can continue to do better. However, despite significant investments in evaluation in recent years, philanthropy continues to struggle to measure its work. This piece offers funders a starting point for developing and strengthening our ability to evaluate by focusing on key elements of planning and organizing the work.

In this era of fewer resources and a growing demand for support from the communities and causes we care about, grantmakers need to have a better understanding about what works, what types of support have the greatest impact and what we can do to continue improving our grants and programs.

Evaluation can take many different forms and the term evaluation can refer to a lot of different activities, from data collection and information gathering to research about grantmaker-supported activities. GEO’s emphasis, however, is on evaluation for learning. Viewing evaluation through a learning lens requires grantmakers to foster a mindset that drives our staff and stakeholders to consistently and continuously look beyond specific projects to understand how well we are doing. When we approach evaluation as more than an accountability mechanism, it can yield valuable information to help us make faster and smarter decisions about our work.

“Philanthropy still has a way to go before it can deliver on the true promise of evaluation to drive learning and deliver better results for organizations and the communities they serve. The challenge for grantmakers is to weave evaluation into the fabric of what they do every day, and to shift the focus of this work so it’s about improvement, not just proof.” – Kathleen Enright, president and CEO, GEO

While evaluation can be challenging, funders can focus on a few key elements to help steer and shape the effort to use what we and our grantees are learning to improve our initiatives. Grantmakers can begin the work of using evaluation for learning by: 1) developing a framework to evaluate with a purpose and 2) ensuring we and our grantees have the necessary systems to support the plan.

1. Developing a framework to “evaluate with a purpose.”

Given the enormous challenges that grantmakers are working to address, evaluation has to be about more than producing reports that few people actually read. A far better path is to adopt an approach to evaluation that supports continuous learning and that anchors these activities in the goals of funders, grantees and other key partners. This means asking questions to ensure everyone is gaining the knowledge and understanding needed to make improvements on an ongoing basis. In order to evaluate with a purpose, funders need to:

Ask the right questions. Becoming a learning organization starts with a thoughtful assessment of what exactly we need to know in order to improve our work. Rather than just asking grantees, “What did you learn?” at the end of grant reports, consider what conversations we ought to be having to identify areas for improvement.

Think collaboratively. A grantmaker’s strategy for learning should be about more than what our staff and board want to know. To have real impact, the strategy also should reflect the learning interests and priorities of people outside the organization — grantees, funding partners, community leaders and more. Grantmakers who are acting more collaboratively are increasingly inviting multiple stakeholders to help select the right questions and systems for evaluation.

Questions for discussion. The following questions offer entry points to discuss with foundation staff, board and other partners what you are doing (and could do more of) to ensure you and your grantees are evaluating with a purpose:

What do we want to know and why? What questions do we want to answer?
To what extent is that vision shared with our board, staff, grantees and other partners?
What can we do to ensure our organization has an up-to-date plan that connects evaluation activities to our broader mission and goals?
What are we measuring and why?
How well do the outcomes we track reflect the complexity of the issues our grantees are working on?
How can we engage grantees and other partners to identify more useful outcomes to measure?
How can we ensure that our learning strategy reflects the complexity of the issues we and our grantees are working to address by assessing the full range of factors affecting progress, allow for regular reflection on what we’re learning and facilitate mid-course corrections?

2. Ensuring grantmakers and our grantees have the necessary systems to support the plan.

Successful evaluation is founded on having the necessary people, processes and technology in place for collecting and analyzing evaluation results, as well as systems that allow people to reflect and act on what they’re learning in a timely fashion. In order to build and support the systems that undergird evaluation plans, funders need to:

Know our capacity — and our grantees’ capacity too. The infrastructure and systems that grantmakers employ to advance evaluation for learning will depend on the resources and capacity available to do this work. Capacity, in this case, can mean everything from staff time and in-house evaluation expertise to IT systems and organizational budgets. Grantmakers should take steps to assess (and strengthen) our own and our grantees’ organizational structures that support evaluation for learning — including who is responsible for what, what skills are in place among the staff to support learning, and what outside support might be needed from consultants and other experts.

Build on what we already do. One of the main barriers to becoming a learning organization for many funders and their grantees is the perception that this is an add-on activity and will take too much time and money. Grantmakers should therefore resist the temptation to add new layers of work or technology to what the staff and board are already used to and doing. By identifying systems and processes already in place and exploring how to enhance our role in advancing evaluation and learning, grantmakers can build broader acceptance for this work while laying the groundwork for a stepped-up commitment among board and staff.

Don’t measure what we won’t use. Good evaluation systems depend on clear and reliable indicators. But grantmakers should be careful not to overdo it when it comes to developing metrics to assess our performance and that of our grantees. As Lynn Taliento, Jonathan Law and Laura Callanan of McKinsey and Company wrote in an introduction to a recent book by Mario Morino, this can be difficult for grantmakers: “Funders are notorious for requiring overly rigorous assessments. The result is a misallocation of resources and unnecessary headaches for the nonprofit. We’ve observed that the right level of rigor is the result of an open dialogue between nonprofits and their funders. By getting clarity on a program’s strategic and assessment objectives, they can determine the level of rigor that’s required.”

Find indicators that make sense for everybody. Developing indicators and metrics should not be a closed-door exercise. Given that grantmakers are working in common cause with grantees, other funders and community partners, it only makes sense to engage others in identifying indicators of progress toward shared goals. In particular, grantees’ involvement in the development of metrics is essential in gaining their buy-in and in ensuring that they have the capacity to measure what’s needed.

Questions for Discussion. The following questions offer entry points to discuss with foundation staff, board and other partners the extent to which key stakeholders have the evaluation capacity needed to learn and improve:

What can we do to identify better outcome indicators in ways that do not overburden and are useful for our staff and grantees?
What is the burden we are currently placing on grantees to meet our evaluation requirements? How can we lighten the burden while still getting good information? How can we strengthen grantees’ capacity to do this work?
Who else is an important part of our assessment efforts? What support do they need?
What additional capacity do we need to make better use of evaluation for learning?
What can we do to make better use of our existing systems and infrastructure so they can more effectively support evaluation for learning?
What grant report questions or data collection activities can we eliminate because we don’t use the information collected?

Other Keys to Successful Evaluation for Improvement

In addition to planning for and organizing our evaluation, funders need to pay attention to a couple of other key elements:

Affirming our leadership’s commitment to learning as an everyday priority. Successful evaluation for learning happens in organizations that create a culture where key learning strategies and practices become the norm and are embedded in the day-to-day work of the entire organization. Building a culture that supports and embeds evaluation for learning requires a commitment to evaluation for learning on the part of the grantmaking organization’s board and staff leaders, plus a pledge to create time and space for staff members and grantees to assess and learn from their work.

Collaborating with grantees, grantmaking colleagues and others to ensure that evaluation is producing meaningful results. More and more grantmakers are recognizing the value of collaborating with others in our evaluation and learning activities. By working with stakeholders to identify and track indicators and outcomes, grantmakers will do a better job assessing the full range of factors that affect progress on complex issues. This, in turn, can help to advance learning among a broader group of partners about what’s working and what’s not.

Conclusion

Evaluation in philanthropy should have one goal: To improve the results that grantmakers and our grantees achieve as we seek to impact the communities we serve in a positive way. This is both the promise and the challenge of evaluation work. Evaluation with a focus on learning can help grantmakers and our grantees improve outcomes on the ground. Knowing where to begin in transforming evaluation into efforts that support learning is critical. Two key starting points are working with grantees and other partners to focus on evaluating with a learning purpose and building the necessary systems to support learning.