New Look, Same Mission
Learn About Our Rebrand

The Playbook on Impact Measurement

November 13, 2024

For-profits chase revenue like there’s no tomorrow. Nonprofits, on the other hand, chase impact. Impact is your who, what, and why. It’s your reason for being. So you better be sure you’ve defined exactly who your helpful solution is impacting (and how!), because the way you design your impact measurement will define your organization’s architecture.

Landing on an ethical stance to guide you in deciding who you’re going to serve – also known as distribution ethics – will aid you in deciding who your product should be for. Don’t worry if you haven’t figured it all out yet – this playbook provides frameworks for thinking through these big questions.

Plus, we made you this handy worksheet to help you land on an impact framework and think through the distribution ethics of the helpful solution you want to build. Use it as you read through this playbook. Trust us, it will be worth the investment of time 10X over.

“For tech nonprofits, impact is the true north. How you define the kind of impact you want to have and your plan to measure it will determine your organization’s structure and roadmap. Impact measurement requires a thoughtful interrogation of why your nonprofit should exist. Don’t take this decision lightly.”

– Shannon Farley, Co-founder & Executive Director, Fast Forward

Goals for Impact Measurement

Defining impact is critical. It’s how you align your tech nonprofit’s work with the value that you want to create for the world. And, it’s how you can raise support to keep doing your work. Potential funders want to make sure they’re not just giving, but giving smartly. Accordingly, a tech nonprofit’s ability to effectively quantify its impact can have a huge influence on its ability to attract funding.

Defining impact metrics that align your goals with company values enable you to:

  • Regularly capture data that reveals your impact performance.

  • Measure, monitor, and maximize progress so you regularly improve.

  • Compare your work to a benchmark.

  • Report your impact to stakeholders.

  • Increase your capacity to do more of your good work.

Definition

Theory of Change

A theory of change (aka ToC) asks: How is your good thing going to change the world? It connects what the activities of a tech nonprofit are with the change it wants to effect. You will be asked about your theory of change so it’s important to know it by heart. Note: Your theory of change can always evolve.

Challenges of Impact Reporting

We’ll start with the hard stuff. Defining success in the nonprofit sector is difficult. Unlike its for-profit counterpart, one cannot simply look to financial returns as a measure of success. Rather, the aims of nonprofits are varied and unique. As such, it is usually not possible to create industry-wide standards for performance. This makes it very difficult to compare the degree of social impact across different organizations and projects.

There are some major challenges to impact measurement in the tech nonprofit sector:

  • Lack of industry standards. There are buzzwords abound that mark universal standards for measuring success in the for-profit sector. It’s why you can compare the metrics of a company like Lyft against a wildly different company like Airbnb. While universal industry standards don’t exist in the social sector, the imperative to measure impact does. Read: there’s no simple calculation.

  • Qualitative focus. Your work will have qualitative impact; some things can’t be quantified. Said differently, stories about the particular ways your product improved the lives of individuals may often be the most powerful way you can communicate impact. But that’s not enough. More and more, funders expect you to be able to provide quantitative evidence for how you know the helpful thing you’re building works.

  • Impact means different things to different people. To complicate matters, different funders look to different types of impact reporting as funding criteria. Therefore, you’ll need to come up with a few different metrics and stories to demonstrate impact.

  • Output does not equal impact. We’ll get into that later in the playbook, but the basic idea here is that just because you’re getting people to use your helpful solution doesn’t mean it’s having the impact you want it to.

Pro Tip

Harness AI for Impact

AI can be your game-changer in real-time impact monitoring. By harnessing tools like Power BI or Tableau with machine learning, you can uncover patterns in service usage and link them directly to outcomes. For example, if you provide educational resources to underserved communities, AI could reveal that students who frequently engage with a specific feature see improved results, empowering you to fine-tune and enhance your offerings for maximum impact.

Distribution Ethics

Josh Nesbit, Founder and Former CEO of Medic Mobile, has shared a framework and set of thought experiments that helped him and other tech nonprofit leaders. Here are the basics.

Distribution ethics is a framework that helps you decide what to build and for whom. Whether or not we discuss them openly, moral decisions are at the center of what we do and how we do it. Everyone and every organization has a moral stance – either you picked it explicitly, you intuited it, you inherited it, or someone picked it for you. But being thoughtful about defining your ethical stance will serve as a guiding principle for decision-making, which is one of the things you’ll do most as a founder.

The Classic Nonprofit Model

Idea → Mission → Delivery Model → Business Model → Org Design → Staffing

Distribution ethics inserts a moral stance into the classic nonprofit model. It asks the hard questions like: Who deserves access? Is depth or breadth more important? And based on those answers, what should we work on? The messy problems that most merit our attention usually involve a moral stance.

Model Emphasizing How Moral Stance Affects Distribution

Idea → Mission → Moral Stance → Delivery Model → Business Model → Org Design → Staffing

Classic Ethical Frameworks

Frameworks for thinking about who gets your helpful solution.

DimensionUtilitarianEgalitarianLibertarian

Goal

The most good for the most people. Utility reigns.

Preferential options and just outcomes. Rights reign.

Respect individual liberty and dignity. Resources reign.

Guiding Principle

Maximize area under the curve where utility = Scale X Net Benefit.

Help those who are worst off and most disadvantaged by society. 

Deliver your helpful thing to those who commit resources.

Codewords

“scale” “low hanging fruit” “public health” “cost-effectiveness”

“human rights” “social justice” “last mile” “universal health coverage” “equitable outcomes”

“sustainability” “market forces” “step out of the way”

The implications of your moral stance will impact everything, including:

  • Your Idea: Intervention, product, or service.

  • Mission: What you do, for whom, and why.

  • Delivery Model: How your organization delivers its helpful solution to people.

  • Organization Design: The way your organization is designed to carry out the delivery model.

  • Staffing: Who to hire and what they’ll work on.

  • Goals: Both your organizational goals and fundraising targets.

  • Storytelling: The way you communicate what you do and why you do it.

  • End game: Your decision around when it’s acceptable to stop serving.

Now to the big question… which of these ethical frameworks resonates with you?

Case Study

Medic

The Playbook on Impact Measurement – Fast Forward

While in rural Malawi, Josh Nesbit witnessed community health workers travel for days to deliver information to the nearest hospital. At the same time, Nesbit experienced stronger cell phone reception there than in his apartment in California. Recognizing that there was only one doctor per every 250,000 people where he was staying, Nesbit was inspired to found Medic Mobile, a tech nonprofit that today builds world-class, open-source software that supports health workers delivering equitable care that reaches everyone.

But it wasn’t always this way. When Nesbit first launched Medic Mobile, the organization was implementing tools in partnership with on-the-ground organizations. In doing so, he was operating from a libertarian stance, providing tools to local partners who already had some human and financial capital.

At a certain point, Nesbit realized that the libertarian framework wasn’t reflective of his moral stance. Philosophically, he wanted to help those who were worst off; he wanted everyone to have the medical care they needed. With that realization and a lot of reflection on how Medic Mobile’s model was dictating who was impacted, he shifted to operate from an egalitarian framework.

This led Medic Mobile down a new path, refocusing as an organization and redefining its product vision. As a result, it began – and continues to build – open-source tools that enable a decentralized network of community health workers to deliver effective last mile health care. And it’s made a huge impact. Last year, Medic Mobile supported a global network of 27,477 health workers, which provided 2,292,004 child health assessments and more than 11 million services for their neighbors.

Deciding Who to Serve

Distribution ethics looks at the types of people you could serve. The spectrum of distribution ethics considers many dimensions, ranging from who you serve and how easy or hard it is to reach them, to how deeply you serve them, to the quantity of individuals you serve, and beyond.

Imagine a circle containing all the possible people who could benefit from your idea, product or service:

In this circle, some people will be easier to reach, and some people will be harder to reach. By harder, we mean that they have less resources; time, money, and social capital. Or, they’re farther away and harder to find, so it costs more to reach and serve them, and it’s riskier to deploy your solution.

You can also identify the people who are easiest to reach. These people still stand to benefit from your solution, but perhaps not as much.

Generally, those who are the very hardest to reach and subsequently serve could benefit from additional resources the most. In some cases, there will be a tension between what the most people need and what the most disadvantaged people need. Serving the most people may lead to great utility. Serving the most needy may lead to progress on specific human rights.

Chain of Causality, Defined

Inputs: The things that go in.

Activities: The things that happen.

Outputs: The immediate results of the activities.

Outcomes: The long-term results of the activities being effective.

Impact: The effect that the intervention has on the root cause.

Chain of Causality

Now that you’ve landed on a moral stance that will guide your decision making, it’s time to think about the chain of causality, or the connection between inputs, outputs, outcomes, and impact. The chain of causality is an important element of impact measurement. The main thing to understand is that outputs and outcomes are very different things. It’s one thing to provide a certain number of families with a service (or output) but another to claim that use of the service created the change you set out to effect (or outcome).

Chain of Causality

Below are simplified examples of the chain of causality for two tech nonprofits, Hikma Health and JustFix.

Tech NonprofitsInputsActivitiesOutputsOutcomesImpact

Hikma Health

  • $$$
  • Staff time
  • Tech
  • Builds and distributes tech that enables health data collection in low-resource settings like refugee camps
  • Physicians can easily collect and access health information in the field to support under-resourced patients
  • NGOs/clinics can use the data to support clinical and operational decision making


  • Patients adhere to prescribed treatments
  • The efficiency of physician-patient interaction improves, as does the operational decision-making capability


  • Refugee health outcomes improve, as evidenced by reduced morbidity and mortality of chronic and communicable diseases
  • Operational costs per patient decrease, and pipelining forecasts for clinics improve


JustFix.nyc

  • $$$
  • Staff time
  • Tech
  • Builds and distributes tech tools that aggregate case data to identify trends and connect tenants and advocates with resources to improve collective outcomes
  • Coordination among housing advocates and organizers
  • Tenants gain information and tools to fight for their right to fair housing


  • Housing advocates and organizers are better equipped to support individual tenants, as well as effect policy change
  • Improves individual tenant outcomes through individual action, collective action, and policy change


  • Individual tenants or a group of tenants working together see tangible improvements in their living situations that would not have occurred had they not used one of Justfix.nyc's tools


Pro Tip

Measure What Matters

The KPIs you’ll track to measure usage of your product are different from the impact measurement you’ll share with your supporters and funders. It’s the difference between seeing if your product works – which you obviously need to track – versus seeing that your organization is moving the needle on the problem.

Proxy Impact Indicators

Oftentimes, you won’t be able to track the exact metrics you’d like to track right away because gathering the data will take years (or decades!). Instead, lean on research and proxy impact indicators.

For example, it took TalkingPoints, the multilingual messaging platform for teachers and parents, years to show that its product improves student outcomes. In the meantime, the organization leaned on studies showing that parental engagement does, in fact, improve student outcomes. So, if the product improved parental engagement, TalkingPoints could extrapolate that the product improved student outcomes. If you’re in a similar boat, it’s okay to do ample research, measure the output, and extrapolate outcomes from there.

Additionally, proxy indicators are more powerful than ever with AI integration. AI can enhance these indicators by incorporating external research and data trends, improving their reliability over time. For instance, if your tech nonprofit aims to improve healthcare access in rural areas, you might track telemedicine consultations as a proxy for health outcomes. Using an AI tool like IBM Watson Health, you can analyze broader healthcare data from similar populations and cross-reference it with your consultation figures. This could reveal that increased telemedicine consultations correlate strongly with reduced emergency room visits. Armed with these AI-driven insights, you can confidently use consultation counts as a proxy for improved health outcomes, while continually refining this indicator as new data emerges.

How to Manage Impact Measurement

Measuring impact is not a one and done thing. It’s an iterative process that requires regular attention. In addition to collecting data on a regular basis, you can (and should!) continually return to the metrics you’ve decided to track and reassess if they’re still the metrics that demonstrate impact.

Here’s the general process for defining and keeping track of your impact metrics:

Pick the impact metrics you want to measure from output to outcome. Keep in mind the important questions like:

  • What’s my theory of change?

  • Do I want to help those who are easiest or hardest to serve?

  • What are the key metrics that will demonstrate that I’m succeeding in my mission?

  • What processes will I use to keep track of my impact metrics over time?

How frequently should I be tracking? Daily, weekly, monthly?

  • What targets will give me confidence that I’m on the right track?

  • Who collects this data? Is it me? Someone on my team?

  • Where is the data stored? This could be as simple as a spreadsheet or a fancy dashboard.

  • How is it audited and how is compliance ensured?

  • Are there any AI tools that can help me track these metrics or store them all in a centralized place?

  • Do I need an impact measurement tool to collect and analyze impact in real time?

Go back and make sure you’re still measuring the right thing. And ask yourself the same questions you started with:

  • Has my theory of change…changed?

  • Am I still serving the same group of people?

  • Are these still the key metrics that demonstrate I’m succeeding in my mission?

Examples of Impact Measurements for Tech Nonprofits

As we’ve explained, every tech nonprofit uses different metrics to demonstrate impact. Here are a few examples of top line impact metrics for some tech nonprofits (Upsolve, CommonLit, DonorsChoose, and TalkingPoints) you’ve met through The Playbook:

  • Upsolve

  • Debt relieved
  • Cases filed
  • CommonLit

  • Students served
  • Schools served
  • Monthly active users
  • Rate of registered low-income schools
  • DonorsChoose

  • Dollars raised
  • Projects funded
  • Students helped
  • Schools participating
  • Teachers funded
  • Supporters
  • TalkingPoints

  • Total number of families served
  • Number of conversations facilitated
  • Percent of underserved schools served, percent of non-English speaking families served
  • Rate of teachers who see improvements in student outcomes after TalkingPoints
  • Rate of low-income, non-English speaking families who are more engaged
  • Number of families who had more conversations about school at home with their children

Definition

Randomized Control Trials (RCTs)

RCTs are a research tool used to validate impact outcomes. But they can be extremely detrimental to early-stage organizations. They’re expensive, they’re time-consuming, and depending on their design, you may have to halt product iteration. Most of the time, these are death sentences for startups. If a funder asks for a RCT, you can point to external research and proxy impact indicators as viable alternatives.

Re-Think Impact

It might be the case that you’ve already built something and can’t change all of your plans tomorrow. That doesn’t mean it’s too late to intentionally choose a moral stance and let that guide how you move into the future. Josh Nesbit provides a helpful thought exercise for doing just that:

  1. Draw your circle of people and write “Here’s where we are” over it. Highlight the group of people – either easiest, easier, or hardest – that you’re serving today.
  2. Draw a new circle of people and write “Here’s where we’d like to be” over it. Highlight the group of people you’d like to serve in the future, based on your moral stance.
  3. In the middle is the delta that you need to solve for. You’re not alone in solving for this delta. The delta represents an opportunity that so many people around you want to help you solve for. By defining an ethical framework and taking a moral stance, you will find resources to help you solve for the delta.

Be the tech nonprofit that is solving for the delta. There is opportunity for transformational impact in the line from where you are today to where you want to be. In the case of Medic Mobile, the organization ended up building an entirely new version of its tools after going through these exercises. Then, it had to do it again and again and again. A moral stance is like a map – it helps you gut check if you’re on the right path with every new decision.

When you’re a tech nonprofit, ethical frameworks matter deeply. Not just in our operations, but to the very core of why and how we do what we do. Tech enables us to scale to any number of people, and its impacts can be far-reaching. It’s by defining a moral stance and letting that guide our decision making that we make smart choices.

There isn’t a right or wrong answer to which ethical stance your organization takes. Each stance has moral value. But you do have to choose to ensure that every decision you make is in alignment. In closing, ethics are complicated, and we can’t do this work alone. We must lean on our peers and community to make sure our moral frameworks are sound. Social impact isn’t a solo sport – we’re in this together.


Additional Resources

Impact Measurement Checklist

  • Use Fast Forward’s Impact Measurement Worksheet to determine and outline your ethical framework.

  • Define your theory of change. What is the change you seek to effect? How does this helpful solution help you get there? Keep it simple.

  • Decide who gets your helpful thing, recognizing that there are any number of people your solution could help.

  • Define your moral stance.

  • Think through your chain of causality. What inputs, outputs, outcomes, and ultimate impact are involved in your helpful solution?

  • Create a simple system for measuring impact. Return to the How to Manage Impact Measurement section and ensure you’ve checked all of the boxes to develop a sound system.

  • Keep refining. The process never ends.

The Playbooks

Want best-in-class tech nonprofit knowledge?

A textural image A textural image