10 Questions for the “Measurement Guy”

Jason Saul calls himself a measurement guy. For virtually all his professional life, he has measured the effectiveness of governments and charities all over the world.

In 2017, he was shocked and disheartened to find a Pew Research study showing that only 18 percent of Americans trusted their government; that in fact, trust in government has been eroding generally since the early 1960s. Worldwide, Saul notes, only about 14 percent of people trust their governments.

He’s setting out to change that. And his work is gaining momentum.

Founder and executive director of Harris’ Center for Impact Sciences, Saul’s mission is to provide policymakers with greater scientific rigor on which to base their decisions. That objective starts with better data harvested from a comprehensive analysis of research on programs then using the data to work up predictive analytics. The goal, he explains, is to find what works in social programs before creating and investing in new ones.

In this edited conversation, Saul explains why government and others in the social impact sector cannot keep “spraying and praying,” as he calls it, and what the Center for Impact Sciences is doing to create a smarter approach.

How did we find ourselves in this moment when trust in government is so low?

Two main theories are out there. One: it’s the fault of the political party leading our government at any particular time. But regardless of what party’s in the White House, trust has declined, except for one brief period after the terrorist attacks of Sept. 11, 2001. Others say we’re not spending enough, but we spend 16 percent of our GDP on social programs—more than $3 trillion a year on social outcomes.

My theory is this: The reason we don’t trust our government is not because we hate it or it’s evil, but because it simply doesn’t work. We’re spending more and yet so many of the important social issues—poverty, food insecurity, childhood obesity, to name a few—are getting worse.

You started in this field a little more than 25 years ago. Why were you drawn to it?

I’ve pretty much always been passionate about public policy and I believe in the power of government to do good. When I was 25 years old, in 1994, I helped start the Center for What Works at Harvard’s Kennedy School of Government. We focused on figuring out how we could help government be more effective. All these people at Harvard Business School were talking about benchmarking, best practices, total quality management, things like that, and I thought, ‘Why can’t we do that for government? Where’s our benchmarks?’ Everyone told me, ‘Oh, that’s impossible. You can’t measure government. It’s too big and complicated.’ But I thought we could. Our goal was to apply innovative business practices to government and social programs. The center morphed into a book I wrote in 2004, Benchmarking for Nonprofits, and then the creation about a decade later of the Impact Genome Project, a platform that standardizes impact data and makes it actionable.

All my work then culminated with the creation of CIS in March of 2020.

Why a Center for Impact Sciences?

At the time, 2020 was the year of highest federal government spending in history, $3 trillion. I thought, ‘What are we doing? This is like spray and pray.’  Government is the only sector of the economy that measures only after they invest. We can’t just keep spending money and then hoping we get positive outcomes.’  That’s not how it works everywhere else. Nobody gives you a mortgage and then checks your credit score.

We have to be able to predict impact.  We can’t just keep guessing.  We have to optimize for ROI - get twice the impact for half the cost.  It’s not about just doing ‘what works’; it’s about doing ‘what works best.’ None of these capabilities are possible without predictive data, without turning policymaking into a true science.  I dream of doing for social science what econometrics did for economics.  That was the genesis of the Center for Impact Sciences.

How did it land at Harris?

I set up a meeting with James Heckman. All my advisors warned me against it. They said he does not suffer fools. I walked in there and we had the best conversation ever. He printed out a segment of his Nobel speech and he’s like, ‘Jason, you’re talking about what I’ve always dreamed of doing.’ He was really helpful early on. Then I had a conversation with John List, then-chairman of the Economics Department at UChicago, who became the flame to Jim Heckman’s spark, and helped me intellectually and administratively realize my dream. Now, he’s the academic director at CIS. I also met with Dr. David Meltzer (co-director of the Health Lab) and (McCormick Foundation Professor at Harris) Bruce Meyer—both of whom have been incredibly supportive.

From my first interaction, I felt that I had an intellectual home at University of Chicago. And then specific to Harris, it’s where, as they say, they take social impact down to a science. It was a natural fit.

If you were at a cocktail party, how would you explain what the Center does?

It’s kind of like inventing the periodic table in chemistry, but for social impact. In order to predict, we must understand the underlying components or “DNA” of a social intervention (i.e. does it use mentoring, how much parental involvement, are there financial subsidies, does it incorporate hands-on learning, is there a focus on social-emotional skills, etc.). These components can be standardized and weighted and systematically studied to understand how those variables interact (like chemistry) and combine to produce desired outcomes. It’s just science.  The dream is to use data science to evaluate and design programs ex-ante, so that we can produce a much better bang for the buck, and step-change the effectiveness of governments all over the world.

Can you provide an example of the work in action?

One great example is the Impact Genome Project, which I created with Dr. Nolan Gasser a Stanford musicologist and the founder of Pandora Radio’s Music Genome Project. We started reading studies on social programs (those that worked and those that didn’t) and we began to see patterns or certain program components or ‘impact genes’ that popped out in the research. We developed comprehensive ontologies, or interlinked taxonomies, and coded every article we read by those same impact genes.  We could rank the genes from ‘incidental’ to ‘dominant’, depending on how crucial they were to the success of the program.  That allowed us to create algorithms to predict the impact of a program merely based on its design, or genomic composition, just like we do with human DNA.

We went on to build genomes for education, domestic violence, youth development, microfinance and dozens more. Then we started working with foundations giving grants, asking them to use this data: instead of funding what you think might work, let’s use data to estimate the likelihood of success of each grant applicant, and then fund the ones with the highest potential. The results were astonishing. Impact rose by 20-30 percent from those funders that used our evidence and science to design programs versus those that merely guessed.

What do you say to people who contend that what you’re trying to do can’t be done in government?

Policymaking is really, really complicated, no doubt. But we have cracked the code on really complicated things before. We cracked the code on human biology with the Human Genome Project. What’s more complex than that? We found a way to predict health outcomes by looking at the common genes among people. We found a way to predict financial behavior, merely by looking at a credit score. We’ve even cracked the code on music, with Pandora’s Music Genome Project, which lets us compare songs by the same core elements in order to predict what songs you’d like.

We have some of the most powerful academic, scientific minds working on this. I know we can get there.

What’s the response been to your work so far?

Pretty powerful. After our Impact Genome work with nonprofits, we received a call from an office of the Canadian government called the Privy Council of Canada— the core policymaking unit that’s part of the prime minister’s cabinet. They asked us to help them use data to optimize  programs in youth employment, anti-racism strategy and global affairs.  We are continuing to roll out the approach across different government agencies.

In the fall of 2021, CIS received a grant from the U.S. Department of Health and Human Services to lead an inter-agency government working group to explore how we can use those core components to improve evidence-based policymaking. In other words, how do we harden the science of evidence-based policymaking and how can we start to make it more predictive? It’s exciting, pioneering work.

Other really exciting developments include a grant we received from the MasterCard Center for Inclusive Growth, and a gift from Splunk for Good. Both support our work on using data science to predict social impact. That work is starting to explore how do we innovate better decision-making tools? How do we raise the visibility of this issue? How do we engage key stakeholders around this problem of improving the data analytics capabilities of the social sector?

The response from government has been incredibly positive, from people at the Office of Evaluation Sciences at the General Services Administration, AmeriCorps, Center for Medicare and Medicaid Innovation, and the Office of Management and Budget, to name a few. The demand for this type of evidence goes all the way up to the Oval Office.

How can students participate in CIS work?

First, we’re just getting started on two student fellowship opportunities—one undergrad and one in the MPP program at Harris to support research activities. Also, we’re going to set up a small fund to support students who have ideas for pioneering new impact science methods.

From there, I’m hoping to produce many fellows working on different projects so that an impact science fellowship is like a rite of passage at Harris. The programs would include partnering with different governments for fellowships and internships where students could apply their skills.

People at many of those agencies I mentioned earlier want first dibs on CIS fellows, when we produce them. These agencies want that capability. That’s one of my dreams: to have students endowed with ‘impact science backpacks’ who can walk in these federal agencies and supercharge new thinking and capabilities.

Ultimately, I really want students to look at the Center as a source of innovation, a place to come up with new ideas, a science incubator, even outside formal fellowships.

What do you hope CIS looks like in five years?

First of all, I want to explore how the Center can become woven more deeply and broadly into the value proposition of Harris—the social impact down to a science—and work across interested centers, labs, and other programs at the school. I also could envision a new dimension of curriculum at Harris, around an expanded set of capabilities that would help students and policymakers predict and measure impact.

I see us having convenings of all the leading policymakers in the world to talk about the science, and why what works and what doesn’t.

I have three fundamental beliefs about this work: That we can start predicting instead of guessing. That we might someday create twice the impact for half the cost and finally, that someday we can bend the arc of trust and restore faith in our democracy.