How Can We Help Our Grantees Strengthen Their Capacity for Evaluation?

There is widespread and growing recognition in the nonprofit sector about the importance of evaluation — not only for measuring impact, but also for improving programs and better serving communities. While grantmakers generally see evaluation as necessary, most are not yet investing enough resources in this area. Indeed, in 2014, nearly three quarters of nonprofits reported that their funders “rarely or never” fund impact measurement costs. This means many nonprofits face an unfunded mandate to provide data that they don’t have the time or resources to produce.

So how can we put grantees in a better position to evaluate their work? One strategy is to fund the evaluation of specific programs. Another is to help grantees incorporate evaluation and evaluative thinking into their basic organizational DNA. This kind of capacity building can help organizations continuously ask fruitful questions; collect, store and analyze relevant data; and develop the staffing, processes – and culture – that foster a routine use of that data to inform decisions. In addition to funding specific evaluation capacities like staffing and data systems, grantmakers need to pay attention to ensuring nonprofits have the continued support needed to keep that capacity alive.

Some grantmakers have been ahead of the curve with their investments in this area. The Bruner Foundation, for example, has been supporting and studying evaluation capacity building for nearly 20 years. Its Rochester Effectiveness Partnership, which ran from 1996 through 2003, provided intensive, hands-on training and coaching about how to plan for, conduct and meaningfully use program evaluation. The initiative was a robust collaboration of local funders, service providers and evaluation experts, producing a plethora of lessons and resources for the field.3 Beth Bruner, the foundation’s director of effectiveness initiatives, says, “Evaluation capacity building is a vastly underfunded, undervalued area of philanthropy. If we want numbers from our grantees, and evidence of effectiveness, we need to give them the resources to do it.” Since 2003, the Bruner Foundation has continued to develop evaluation materials, which they make available to grantmakers and grantees for free on their website.

GEO talked with Bruner as well as a handful of other grantmakers and thought leaders about what it takes to help grantees strengthen their capacity for evaluation. These conversations pointed to the following six principles for successful evaluation capacity building. By applying these principles, grantmakers can make the most of our evaluation investments and help our grantees become more effective and sustainable organizations.

1. Know Thyself

Funders with experience in this area highlight the need to look inward to clarify our goals and where we need to build our own capacity for evaluation, so we can be a knowledgeable partner and resource for grantees. As Rebekah Levin, director of evaluation and learning at the Chicago-based Robert R. McCormick Foundation, explains: “Grantmakers need to take a serious look at themselves first, before they look at grantees. What is it they are using data for? How are they using evaluation to drive their own work? Because it’s way too easy for us to say, ‘you do it,’ to nonprofits without understanding what that entails and how difficult it is to use evaluation in a thoughtful and ongoing way.”

Focusing on our own evaluation capacity enables grantmakers to model the learning and continuous improvement ethos that we often want our grantees to adopt. Michele Guzmán, director of evaluation at the Meadows Mental Health Policy Institute, and formerly assistant director of research and evaluation at the Hogg Foundation for Mental Health in Texas, says: “We need to walk the walk. If you aren’t building logic models, you can’t really ask grantees to do it. Be clear about outcomes. Have learning meetings. Start at home, so evaluation becomes a value for the foundation, not just an activity you do sometimes.”

One benefit of this approach is that it can help funders move from ambitious, high-level goals to specific, measurable objectives for the initiatives we support. According to Deb Vaughn, the arts education coordinator for the Oregon Arts Commission: “Having conversations with evaluators before they started working with funded sites forced me to really define my outcomes for the grant program — what I intended to change and what I needed to measure — which then, in turn, I was able to communicate to grantees.” When funders are clear about what we want to know and how we will use that information, we can be more streamlined in our work with grantees and avoid burdening them with requests for unnecessary data.

How to Start the Conversation

Grantees are unlikely to say “we don’t have the ability to provide the data you just requested.” In most cases, funders will have to start a dialogue with grantees about their existing evaluation capacity and the kinds of support they would find helpful. Approaching these conversations with a spirit of genuine interest, openness and candor is important. Here are some questions you might consider asking:

• What does success look like for your organization, and what information do you need to know if you’re on track to reach those goals?
• What do you hope to learn through evaluation? What are the kinds of questions you would like answers to, and if you had those answers, how would they inform your work?
• What data do you currently collect and how? How do you use the data you collect? What kinds of data do your other funders request?
• What are the biggest challenges you face around collecting and using data?
• What do you see as your strengths in this area?
• Who on your staff “owns” evaluation activities? What are those activities?

2. Encourage Grantee Ownership

For capacity-building efforts to be effective, grantees need to have a sense of ownership over the process. They need to understand the potential benefits of evaluation and the value of devoting time and energy to this work. Empowering grantees to figure out what they need to know for their own programs and stage of development — rather than simply responding to reporting requirements — is essential.

Often, a good first step is to help nonprofits clearly outline what they are trying to achieve. The Hogg Foundation for Mental Health began providing this kind of support after realizing that grantees in one of their large-scale, multi-year initiatives were having trouble reporting outcomes. The foundation started with a training about basic evaluation principles, including the difference between an outcome (how are participants better off thanks to your program?) and output (how many people did you serve? how many events did you hold?).

The Hogg Foundation staff urged grantees to think about what it would look like if their program were a success, how they expected to get there and how they could measure the results. Guzmán said while at the Hogg Foundation, they often ask grantees to develop a logic model, especially in initiatives that have a planning phase: “Using a logic model or other goal-setting tool is a key component to building evaluation capacity, because it focuses people on where they’re going, what the desired outcomes are and what information is needed to understand if they are making progress.”

Of course, funder and grantee goals for evaluation may not automatically align. Grantmakers should be as transparent as possible about our objectives and expectations, starting with our website and grant application materials and extending to our conversations with grantees. As Levin from the McCormick Foundation describes: “We have to say to nonprofits, ‘OK, we need to understand this. You need to understand that. How do we come up with an evaluation design that satisfies both our needs and yours?”

Not surprisingly, there can be tension between using data for accountability and using it for learning. Being clear about what information we seek for each purpose is key, as is a spirit of trust and collaboration. Vaughn of the Oregon Arts Commission says that “frank conversations with grantees — about what we will want to achieve, what we don’t know, what we’re excited about learning — can remove some of those hierarchical funder-grantee relationships, and convey that we’re all in this together.”

3. Consider a Combination of Group Learning and Individualized Support

Some grantmakers use a combination of group-based evaluation capacity building and one-on-one training and technical support. When the Blue Cross Blue Shield of Massachusetts Foundation revamped the reporting structure for its Connecting Consumers with Care grant area, they discovered that some grantees had very little experience with the evaluative thinking the foundation wanted to promote. Jessica Larochelle, the foundations director of evaluation and strategic initiatives, explains, “Grantees were very good at doing the work to serve the community. But defining measures, collecting data and then using that data for actual program improvement and learning was a completely new world for a lot of grantees.”

Larochelle and her colleague, Senior Program Officer Jennifer Lee, organized a series of group training sessions, as well as customized technical assistance for individual grantees, both of which proved useful. “We found that peer-topeer learning worked well in a group setting,” says Larochelle. “Asking a grantee with strong evaluation capacity to share approaches and lessons learned with their peers was much more effective than the foundation just providing theoretical examples.” At the same time, the one-on-one technical assistance provided an opportunity to discuss grantees’ specific challenges and solutions.

There are distinct advantages of both group learning and individualized support. Bringing grantees together to develop and implement evaluation plans, and to reflect on lessons that emerge, can be powerful. These convenings not only create a space for peer-to-peer learning, they also foster a sense of community among grantees and encourage collaboration, which can be particularly valuable for nonprofits working in the same geographic area. The first group training is likely to focus largely on definitions — that is, making sure everyone understands evaluation jargon. While this is a critical starting point, it doesn’t do much to help grantees incorporate evaluation into their daily work. Thus, group learning shouldn’t be conceived as a “one-shot deal,” but rather as ongoing, structured, interactive sessions. For more information on how grantmakers can support successful learning communities, see GEO’s publication, Learn and Let Learn.

Because grantees bring a wide range of experience and skills to the table, customized support is also important. One-on-one training provides an opportunity to meet individual needs and build on individual strengths. Vaughn from the Oregon Arts Commission advises, “Meet grantees where they live. There are lots of different levels of understanding of and comfort. Customized help allows everyone to raise the bar together.”

Whether building capacity through group learning or individualized technical assistance, opportunities to practice evaluation skills in a hands-on way are essential. Learning about these skills in the abstract is unlikely to significantly change how organizations function. The Hartford Foundation for Public Giving’s Building Evaluation Capacity (BEC) program, modeled on the Bruner Foundation’s approach, offers 18 months of training and coaching designed to increase both evaluation capacity and organization-wide use of evaluative thinking. Annemarie Riemer, Director of the Hartford Foundation’s Nonprofit Support Program, says, “Conducting an actual evaluation is a crucial element of BEC. Though it accomplishes both, the focus of the program is more on continued learning within an organization than it is on an attempt to provide outcomes to an outside funder.”

4. Invest in Organization-Level Change

While developing the knowledge and skills of staff members at grantee organizations is important, the reality is that these staff won’t stay in their jobs forever. Some grantmakers look for ways to build capacity at the organizational (rather than individual) level. Johanna Morariu, Director of Innovation Network, a nonprofit evaluation firm, sees this as a particularly important point: “Whether it’s a new data system, or cleaning up or upgrading an existing system, or trying to institutionalize data and evaluation within management and programmatic decision-making — things at that level have more stickiness.”

This was certainly true for the Latin American Youth Center, a multi-service organization serving low-income youth in an around Washington, D.C. LAYC has had an active evaluation department since 2005. A few years ago, Tony Fujs, the organization’s Director of Learning and Evaluation, realized that staff were spending large amounts of time manually processing data in response to requests from different grantmakers. LAYC solicited support from a long-time funder to help build the technical infrastructure for automated reporting. The investment has resulted in greatly improved productivity. Fujs says, “We can now produce in two minutes what took us hours before.”

Fujs describes a number of options for funders who want to build evaluation capacity at the organizational level. In addition to funding, he says technical support to help select a data system, from the sea of available choices, can be beneficial. He also feels strongly that nonprofits need a dedicated staff personto drive evaluation work internally. Fujs explains, “Sometimes, nonprofits invest in a database, thinking that it will be the silver bullet that will solve their data challenges. But you need someone who understands evaluation and database management to make the most of the system.” Finding the right person for this role — someone with technical skills and the ability to develop an internal culture of learning and continuous improvement — can be challenging. Nonprofits may value guidance (e.g., from an evaluation expert) as they create a job description, search for qualified candidates and decide who to hire.

5. Start Small and Manage Expectations

Developing grantee capacity to use and benefit from evaluation is a significant undertaking. Experienced grantmakers describe several ways to avoid “biting off more than you can chew.” Bruner advises funders who are newer to this work to start small — by selecting a grantee with whom you have a strong relationship, trying out an approach, requesting feedback, and learning from the experience.

It is important to be realistic about what a capacity-building effort can achieve. Grantees’ progress may be incremental. Guzmán, formerly of the Hogg Foundation, explains, “You’re not going to take someone from A to Z. Maybe you’ll go from A to G, or A to N. But not everyone is going to reach the same bar. Provide opportunities for growth, and remember how swamped grantees can be.”

Grantees should also look for ways to build on the capacity that nonprofits already have in place. Morariu from Innovation Network says, “There are probably accountability functions already going on — sharing data with board members or funders, for example. Help grantees figure out how to use these functions to support learning.” Setting modest, achievable goals for improvements to evaluation capacity can increase the odds of a “win,” which funders and grantees can then build on.

6. Think Long Term

Evaluation capacity-building works best when it happens over an extended timeframe, within the context of a trusting, long-term relationship. The typical one-year grantmaking cycle may not be ideal for this type of work. Vaughn says, “People have to get comfortable with the extended timeline that this process takes. We all want to see change happen right now, but there are lots of players, it’s a slow process, and there needs to be a longer-term commitment.”

Part of this longer-term vision may involve grappling with field-wide issues that impact grantees’ ability to use evaluation effectively. One of these issues is access to data needed to track participants’ long-term outcomes. Getting information about educational outcomes or employment, for example, would be extremely useful for many nonprofits, but it is nearly impossible to do in many locales. Another challenge stems from the varied reporting requirements of different funders (both public and private). Fujs from LAYC explains, “A nonprofit building a performance measurement system will pick a set of metrics that matters to the organization. Funders also come up with various metrics that matter to them. If you look at each organization independently, it makes complete sense. Now, if you take a step back and look at the ecosystem of nonprofit organization, and a nonprofit organization is generally funded by more than one funder. You end up with nonprofits being asked to track so many different metrics, or similar metrics defined in different ways that it can be counterproductive.” Grantmakers who invest in this work for the long haul may be able to advocate for better collaboration across institutions and systems, which can help address thorny field-wide issues.

Conclusion

Building evaluation capacity provides grantees with tools they can use for years to come. Guzmán, formerly of the Hogg Foundation, argues, “This is a way to leave something behind — putting grantees in position to keep improving, to communicate their impact, tell their story well and get more funding. This is about a long-term investment in grantees.” By taking stock of our own goals and capacity, by helping grantees identify the questions and data that matter most to them, by providing high-quality training and investing in new systems and processes, and by engaging in long-term trusting partnerships with our grantees, we can significantly enhance their ability to use evaluation — which makes for more effective and more sustainable organizations. As Jennifer Lee from the Blue Cross Blue Shield of Massachusetts Foundation states, “Organizations that have evaluation capacity, and can leverage it, are the strongest. They can be nimble, strategic, and more effective as they work to achieve their mission. They can generate resources to continue doing their work, and can do it better over time.”