Eight Tough Questions for Scaling Up Decisions

Yesterday I wrote about scaling and sustainability. And the next thing I knew, I received an email about the new report A Funder’s Guide to Using Evidence of Program Effectiveness in Scale-Up Decisions by MDRC and the Social Impact Exchange at Growth Philanthropy Network. 

The report highlights eight questions that funders should be asking throughout the scale-up process. I love that they put the question about local conditions at the top of the list. However, I really wish that the authors had been aligned with My Brother’s Keeper and asked a question about how the evidence can ensure that boys and young men of color benefit from the intervention (assuming that it is an intervention for males).

 It’s up to all of us to be a “Keeper,” and that means we need to explicitly challenge ourselves to make sure programs and interventions work for young men of color – and if they don’t, figure out what needs to happen to make them work! I think of being a “Keeper” as being a sneaking, snooping detective – following the clues to expose patterns of institutional racism wherever we can. 

The big eight questions are challenging and will help us stay honest when we want to start scaling up programs and interventions. They’ll also help non-profits think hard about their own scaling process. They are:

1. What can be learned from existing research and the local context? At the outset, a careful review and synthesis of local conditions and relevant lessons from evaluations of similar programs should feed into development of a sound theory of change.

2.What are the characteristics of the people being served? A clear understanding of their characteristics and how they were selected helps in interpreting results. A “funnel analysis” can provide important insights.

3. What is the content of the program, and how well is it implemented? It is important to understand how the theory of change is implemented in practice.

4.How much does the program improve key outcomes compared with a reliable counterfactual? A reliable counterfactual (that is, a benchmark of how participants would have fared in the absence of the program) is crucial to understanding a program’s true added value.

5.What factors most influence the results? The program elements that contributed most to the positive results should be preserved during scale-up.

6. How much does the program cost? Consider program costs from a number of perspectives and how the costs might change during scale-up.

7. How relevant is the evidence to the funder’s specific scale-up decisions? Consider how much the accumulated evidence relates to programs operating under the conditions that will be in effect during scale-up.

8. How can reliable evidence be produced at reasonable cost? Pay attention to the quality of data and how the data are analyzed. Recognize the need to support the real costs of quality evaluations.