Takeaways & next steps
Embracing “the new science of impact” isn’t a trivial undertaking. It requires new practices, sensibilities and partnerships among mission-based organizations, academic researchers, funders and the communities they collectively serve.
Overcoming the Barriers
Robust impact analysis can often be perceived as costly, slow and difficult, and beyond the expertise of even exceptionally gifted social-change organizations. Given tight budgets, such efforts can be framed as either an unaffordable luxury or glorified accounting. Without steadfast support, many organizations will succumb to strong pressures to pour otherwise scarce resources into the intervention itself.
Complexity is also a factor. Many social innovators work with difficult-to-reach populations under challenging circumstances, which makes isolating cause and effect especially daunting. Frequently, these organizations are only able to do their work as the result of carefully cultivated relationships with the community they serve. If not handled carefully, these relationships can be upset by the imposition of external measurement processes — particularly tools like randomized controlled trials with their attendant complex ethical considerations.
Then there is the elephant in the room: the political and financial risk of organizations undertaking such measurements. Many mission-driven organizations thrive on anecdotal evidence of their own impact, and many risk-averse funders would prefer not to look too deeply under the hood. Who wants to mess with the appearance of success?
And yet, as pressing global concerns mount and resources remain flat, a new model is clearly needed — one in which robust impact assessment is an integrated, transparent (and funded) component of all social innovation efforts. This requires mission-led organizations and their funders to adjust existing methods, to adopt an increased tolerance for risk, and to become more responsive to feedback. It requires valuing shared learning (even if its learning what not to do) as a critical part of delivering the mission at hand.
We believe that for those willing to embark on this journey, the rewards can be extraordinary: greater impact, better insight, more honest relationships, new innovation opportunities and improved pathways for scale. Here are some pointers to get started:
- Start with a plain-English hypothesis and identify confirming and disconfirming measures around a given approach to a problem. For example:
“We believe that by doing [Intervention A], [Benefit B] will occur, more than it might otherwise occur naturally.”
“Here is the evidence that might confirm this hypothesis, and evidence that might disconfirm it.”
“Here are factors that might also independently cause [Benefit B] and how we will control for them”
“Here are factors that might also mask or suppress [Benefit B] and how we will control for them”
- Embrace approaches that anticipate and embed data collection in as an intrinsic part of core operations of the intervention from the beginning, not simply as a separate “add-on” activity.
- Bring in a data scientist and, if possible, a social scientist to review the project or approach in advance for methodological blind spots and other confounding factors.
- Since data collection can be expensive, where possible, harness ambient infrastructure (such as mobile devices, GPS, sensor networks, etc.) to collect information passively.
- But don’t rely solely on passive data; use a mix of methods —combine in-person surveys and mobile data, for example.
- Give yourself permission to fail by designing iterative cycles into the intervention, so that analysis has a chance to shape the work, not simply evaluate it.
- Whenever possible, publish open datasets of your raw results so that others can analyze them. This includes data that:
- have been carefully scrubbed of personally identifiable information;
- are published in open standard formats and are not restrictively licensed;
- use common data schemes and vocabularies;
- are made accessible on the Web at a persistent, human-readable address;
- are kept current and up-to-date;
Taken together, these strategies won’t just produce better and more lasting positive impact in the world — they will also encourage a deep and positive cultural shift in how organizations understand themselves, their mission and their relationships with the communities they serve.