Skip to main content
The University of Chicago

Utility

  • Log in
  • Directory
    • Hire Harris Overview
    • Find Talent
    • Find Work
    • Info For Overview
    • Alumni
    • Admitted Students
    • Current Students
    • EMP Students
    • Faculty and Staff
    • Media
    • Make a Gift Overview
    • Ways to Give
    • Annual Fund
    • The Harris Society
    • Meet Our Team
Sort

Main navigation

  • About Overview
    • Who We Are Who We Are Overview
      • Who We Are Overview
      • Career Outcomes
      • Harris By The Numbers
      • The Keller Center
      • Awards
    • Leadership Leadership Overview
      • Leadership Overview
      • Alumni Council
      • Dean's Office
      • Harris Council
      • Meet The Dean
    • History History Overview
      • History Overview
      • History of Harris
      • Legacy of Irving B. Harris

    Considering Applying?

    Get a jump start now by creating an account with our simple online tool.

    Get Started

    Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

  • Academics Overview
    • Degree Programs Degree Programs Overview
      • Degree Programs Overview
      • Master of Public Policy (MPP)
      • MS in Computational Analysis and Public Policy
      • MA in Public Policy with Certificate in Research Methods
      • Evening Master's Program (Part-time MA)
      • MA in Public Policy
      • PhD
      • Joint Degree Programs
    • Design Your Path Design Your Path Overview
      • Design Your Path Overview
      • Core Curriculum
      • Practical Experience
      • Specializations
      • Workshops
      • University of Chicago Obama Foundation Scholars
    • Non-Degree Programs Non-Degree Programs Overview
      • Non-Degree Programs Overview
      • Civic Leadership Academy
      • Credential Programs
    • Undergraduate Program Undergraduate Program Overview
      • Undergraduate Program Overview
      • Overview
      • BA Thesis Process
      • Capstone Projects
      • Community Highlights
      • Team
    • Courses Courses Overview
      • Courses Overview
      • Non-Harris Student Registration
    • Academic Calendar

    Considering Applying?

    Get a jump start now by creating an account with our simple online tool.

    Get Started

    Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

  • Admissions Overview
    • Why Public Policy?
    • Applying to Harris Applying to Harris Overview
      • Applying to Harris Overview
      • International Students
      • FAQ
      • Admissions Resources
      • Harris Scholars Program
    • Requirements & Deadlines
    • Meet with Harris Meet with Harris Overview
      • Meet with Harris Overview
      • Connect with Our Students
      • Masterclass Series
      • Meet Our Staff
      • Schedule a Visit
      • Travel Reimbursement
    • Financing Your Degree Financing Your Degree Overview
      • Financing Your Degree Overview
      • Scholarships & Fellowships
      • Loans & Work-Study
      • Third-Party Funding
      • Tuition & Expenses
    • Admissions Blog
    • Request Information

    Considering Applying?

    Get a jump start now by creating an account with our simple online tool.

    Get Started

    Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

  • Student Life Overview
    • What to Expect What to Expect Overview
      • What to Expect Overview
      • From Orientation to Graduation
      • Connect with Your Community
    • Student Engagement & Belonging Student Engagement & Belonging Overview
      • Student Engagement & Belonging Overview
      • City of Chicago
      • Meet the Student Engagement & Belonging Team
      • Mentor Program
      • Our Campus Community
      • Student Organizations
    • Career Development Career Development Overview
      • Career Development Overview
      • Career Resources
      • Alumni Career Support
      • Internship Program
      • Career Development FAQ
      • Meet the CDO Team
    • Dean of Students Office Dean of Students Office Overview
      • Dean of Students Office Overview
      • Academic Advising
      • Harris Writing Workshop
      • Academic Support Programs
      • Meet the Dean of Students Office
      • Student Emergency Resources
      • Policies

    Considering Applying?

    Get a jump start now by creating an account with our simple online tool.

    Get Started

    Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

  • Research & Impact Overview
    • Centers & Institutes Centers & Institutes Overview
      • Centers & Institutes Overview
      • Center for Economic Policy
      • Center for Effective Government
      • Center for Health Policy
      • Center for Human Potential and Public Policy (CHPPP)
      • Center for Impact Sciences
      • Center for Municipal Finance
      • Center for Survey Methodology
      • Center for the Economics of Human Development
      • Energy Policy Institute at the University of Chicago (EPIC)
      • International Innovation Corps
      • Stone Center for Research on Wealth Inequality and Mobility
      • The Pearson Institute
      • Urban Labs
      • Affiliated Centers
    • Faculty Impact Faculty Impact Overview
      • Faculty Impact Overview
      • Nobel Prize Winning Faculty
      • Faculty Directory
      • Expert Guide
    • Initiatives & Partnerships Initiatives & Partnerships Overview
      • Initiatives & Partnerships Overview
      • Cyber Policy Initiative
      • Technology and Society Initiative ↗

    Considering Applying?

    Get a jump start now by creating an account with our simple online tool.

    Get Started

    Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

  • News & Events Overview
    • News News Overview
      • News Overview
      • News
      • Profile
      • Research
      • In the News
      • Admissions Blog
    • Events Events Overview
      • Events Overview
      • Academic Conferences
      • Admissions Events
      • Alumni Events
      • Student Events
      • Special Events
      • Visiting Speakers
      • Past Events
    • Podcasts Podcasts Overview
      • Podcasts Overview
      • Not Another Politics Podcast
      • The Inequality Podcast
      • Public Money Pod
    • Newsletters Newsletters Overview
      • Newsletters Overview
      • Harris Connect: Alumni Newsletter
      • Not Another Politics Newsletter
      • Pointed: Policy Research Newsletter
      • Public Money Policy Newsletter

    Considering Applying?

    Get a jump start now by creating an account with our simple online tool.

    Get Started

    Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

Utility

  • Log in
  • Directory
    • Hire Harris Overview
    • Find Talent
    • Find Work
    • Info For Overview
    • Alumni
    • Admitted Students
    • Current Students
    • EMP Students
    • Faculty and Staff
    • Media
    • Make a Gift Overview
    • Ways to Give
    • Annual Fund
    • The Harris Society
    • Meet Our Team

The Fall 2025 Graduate Program Application is open! Apply by the final deadline on June 12, 2025.

How Can We Evaluate Government Programs If We Can't Trust the Data?

Representative Paul Ryan and Senator Patty Murray, co-sponsors of the Evidence-Based Policymaking Commission Act, November 13, 2013
April 01, 2016
Research
Aimee Levitt
Bruce Meyer found the social safety net is twice as effective as we thought. Now he's helping Congress realize the value of evidence-based policy.

Bruce Meyer, the McCormick Foundation Professor at Chicago Harris, may not have succeeded in forging eternal peace between Republicans and Democrats in Congress and the White House, but his research has contributed to something both sides can agree on: HR 1831, otherwise known as the Evidence-Based Policymaking Commission Act of 2015.

The bipartisan bill, proposed last spring by House Speaker Paul Ryan (then chair of the House Budget Committee) and Senator Patty Murray, will establish a commission to investigate ways in which survey data and administrative data can be used to better evaluate federal programs. The bill passed in the Senate on March 16. First passed in the House of Representatives last July, it now returns to the House for final approval before heading to the president’s desk. The origin of the bill goes back to discussions between Ryan and Meyer’s student James Sullivan. Sullivan, a former visiting assistant professor at Harris, now an associate professor of economics at Notre Dame and director of the university’s Lab for Economic Opportunities, met Ryan through their work with Catholic Charities and began discussing ways to eliminate poverty in America. When the conversation turned to evaluating anti-poverty programs, Meyer joined in the discussion and, eventually, the writing of the bill with Ryan’s staff and other academics.

Meyer’s recent research has emphasized how data that a commission might make available could be instrumental in evaluating whether public welfare programs work – or don’t. This work has been brought to the attention of lawmakers and the general public thanks to coverage in the Wall Street Journal, theWashington Post and Vox.

Traditionally, federal and state governments have used data from household surveys, notably the Current Population Survey, to understand social welfare and insurance programs. In theory, these surveys (administered by phone, mail or in-person visits) are supposed to provide information on income distribution and poverty rates. They are also supposed to provide feedback on how well programs intended to alleviate poverty are doing. But Meyer and his colleagues who study government safety net programs have suspected for a while now that the survey data is not as accurate as it should be.

“There have been little bits and pieces of evidence for a long time,” Meyer says, “suggesting that surveys are not doing a good job at capturing what programs people receive and how much money they get.”

It’s not that people are lying, Meyer says. He suspects they’re just sick of taking surveys. “Any time you buy something online, call someone to complain about something, go to the doctor, get your car repaired, stay at a hotel, you’re asked to fill out a survey,” Meyer explains. “It used to be thrilling to have someone ask you about your life. Now if you need that thrill, you post on Facebook. People don’t quite feel obligated to respond to surveys anymore.”

And so, even when people are legally obligated to fill out a survey, as they are with the U.S. Census and the American Community Survey (formerly the Census long form), they’re not as committed to filling in the data accurately – their exact income, for example, or their program enrollment. Yet this information becomes the data the government uses to calculate poverty statistics, evaluate whether the social programs designed to alleviate poverty are working and determine who is missed by these programs.

“It’s unfortunate,” says Meyer,“that the most important source of information about those at the bottom of the income distribution is household surveys that are increasingly unreliable and inaccurate. The broadest evidence of this problem is that if you count up the dollars people report in surveys and scale them up to represent the entire country, there are many fewer dollars than the government has paid out.”

Meyer believed it was possible to measure the inaccuracy of the household survey data. He would do this by comparing the survey responses of households to the records of government payments to those same individuals. The only problem was, the government data was not readily available to citizens, even social science researchers. Fortunately, he had an idea.

Early in the 2000s, he’d met with Robert Doar, who was at the time the New York State Commissioner of Temporary and Disability Assistance. Meyer had advised him on issues concerning the state’s welfare policy. Doar, who now studies poverty programs at the American Enterprise Institute, was impressed. “He’s a great scholar,” he says of Meyer, “the real deal. He comes at it straight and calls ’em as he sees ’em.” Along with Doar, Meyer met other people in the state bureaucracy – notably George Falco, an advisor to the head of the state welfare office – who were sympathetic to his project and helped him get access to New York’s data for food stamps, Temporary Assistance for Needy Families (TANF), General Assistance and subsidized housing from 2007 through 2012.

This was not an easy process. Meyer estimates it took six or seven years to navigate the state and federal bureaucracy. He had to submit a proposal to describe what he planned to do with the data. He had to undergo a high-level security clearance. He was fingerprinted three times. When he finally had approval to access the data at the Chicago Federal Reserve Bank, he had to go through three sets of locked doors. He couldn’t remove anything from the office. He had to submit the results of his analysis to a review board.

Once he finally had access to the administrative data, Meyer compared them to the Current Population Survey (CPS) results for the same years using social security numbers that had been encrypted in order to protect the recipients’ personal information. Then he analyzed the information to find variation between the survey and administrative data.What he found was astonishing.

The CPS missed more than one-third of housing assistance recipients in New York State, 40 percent of food stamp recipients, and 60 percent of TANF and General Assistance recipients. The survey data underreported the incomes of the poorest households; government transfer payments from all four programs were under- reported by more than $1,400 per person. Most significantly, the administrative data showed that the four programs reduced the poverty rate among recipients by 5.3 percent, not the 2.8 percent reported by CPS.

“Our much-maligned government programs are more effective than even Census Bureau reports indicate,” says Meyer.

In October, the American Enterprise Institute published a paper coauthored by Meyer and Nikolas Mittag, an assistant professor at CERGE-EI (Center for Economic Research and Graduate Education- Economics Institute) at Charles University in Prague, that described the research in exhaustive detail. In the press coverage that followed, journalists interpreted it positively.

“No matter the reason,the fact that we depend on an increasingly faulty system to define things like poverty and judge programs that aim to alleviate it, is disconcerting,” Roberto A. Ferdman wrote in the Washington Post’s Wonkblog. “It can have a profound effect on policy discussions concerning the two. On the one hand, it makes it look like the poor are doing much worse than they are.The official poverty rate now is higher than it was three decades ago, but by almost any measure the poor are better off than they were then.”

Dylan Matthews went even further in Vox: “The emerging, positive view of transfer programs is an indication that we know how to tackle this problem, and that even greater transfers could conceivably end poverty in America altogether.”

Politically, Meyer’s findings, as Doar noted in an op-ed piece in the Wall Street Journal, are as bipartisan as can be. They’re good news for Democrats and other progressives who favor expanding the social safety net: they show that these programs are, indeed, working. They are also good news for Republicans and others who favor greater accountability and less spending for government programs: they show that these programs can keep going without the investment of additional resources. It would make sense for the government to do a similar comparative study on a national scale. The only problem is, although Meyer feels that New York is reasonably representative of the rest of the nation, administrative datasets that encompass social welfare programs for all 50 states don’t exist. Although program guidance comes from the Federal government, much of the data is at the state level.

“If I wanted to evaluate the impact of programs at the local level,” explains James Sullivan, “I would have to negotiate data-share agreements with individual states and counties, and that would lead to a stalemate.”

Adds Meyer, “If administrators can have an excuse not to make data available to the public, they may use that excuse to prevent careful scrutiny of the job they’re doing.”

This is where the Evidence-Based Policymaking Commission Ace comes in. “Sullivan suggested to Congressman Ryan that one thing that’s helpful in making better policy is to have data to evaluate current programs and possible alternatives to what we’re doing,” says Meyer. “That’s how the idea of a commission to encourage the assembly of data for evidence-based policymaking got started.”

Both Meyer and Sullivan gave feedback on early drafts of the bill. “It was a lot of fun,” Meyer says. “I’d never seen how a rough idea can end up in legislative language and be refined over time.” The experience has made him appreciate the importance of writing policy memos that make statistics understandable to policymakers (a skill, he says, that has served him well in presenting his research to legislators).

Ryan introduced the bill last April; it cleared the House and moved onto the Senate, where its sponsor is a Democrat, Patty Murray of Washington, at the end of July. Meyer is uncertain when it will come up for a vote. If it passes and becomes law, he’s also uncertain whether he’ll be asked to serve on the commission.

“The important thing is that the commission be established. Federal agencies aren’t doing many things that are vital to figuring out which programs work and which don’t. Even showing how useful existing administrative and survey data can be in designing policy is a valuable thing for the commission to do, even if the laws on data access aren’t changed.”

Related stories

War Games

April 01, 2015
Hilarie Koplow-McAdams, AM'87

Questions for Hilarie Koplow-McAdams, AM'87

April 01, 2015
Brett Goldstein

Urban Legend

April 01, 2016
  • Prospective Students
    • How to Apply
    • Why Public Policy
    • Financial Aid
    • Events
  • Current Students
    • Academic Advising
    • Policies
    • Courses
    • Disability Resources
  • Faculty & Staff
    • Room Reservations
    • Academic Calendar
    • Faculty Resources
    • Faculty Access
    • Human Resources
  • Alumni
    • News and Events
    • Alumni Directory
    • Get Involved
    • Donate Now
Back to Top
The University of Chicago

Connect with Us

  • Facebook
  • Twitter
  • LinkedIn
  • Youtube
  • Instagram

Questions?

Contact Us

Footer menu

  • Apply
  • Careers at Harris
  • Courses
  • Directions
  • Privacy Notice
  • Accessibility
  • Refer a Student