Skip to main content

Utility

  • Search
  • Log in
  • Directory
  • Hire Harris
    • Find Talent
    • Find Work
  • Info For
    • Prospective Students
    • Admitted Students (Full-Time)
    • New EMP Students
    • Current Students
    • Faculty and Staff
    • Alumni
    • Media
  • Support Harris
    • Ways to Give
    • Harris Society
Sort
Page 1

The University of Chicago Harris School of Public Policy

  • About

    Considering Applying?

    Get a jump start now by creating an account with our simple online tool.

    Get Started

    Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

    • Who We Are

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • Our Principles
      • Harris By The Numbers
      • Career Outcomes Report
    • The Keller Center

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • The Future of Policy
      • Your Impact
      • Design & Sustainability
      • Press Room
    • Leadership

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • Meet Our Dean
      • Dean's Office
      • Harris Council
      • Alumni Council
      • Diversity & Inclusion Advisory Board
    • History

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • History of Harris
      • Legacy of Irving B. Harris
    • Diversity & Inclusion

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • Our Commitment
      • Roadmap & Progress
      • D&I Community
      • Resources
      • News & Events
  • Academics

    Considering Applying?

    Get a jump start now by creating an account with our simple online tool.

    Get Started

    Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

    • Degrees

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • Master of Public Policy (MPP)
      • Evening Master’s Program
      • MS in Computational Analysis and Public Policy
      • MA in Public Policy with Certificate in Research Methods
      • MA in Public Policy
      • MA in International Development and Policy
      • PhD
    • Non-Degree Programs

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • Credential Programs
      • Executive Education
      • Civic Leadership Academy
    • Design Your Path

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • Courses
      • Core Curriculum
      • Practical Experience
      • Certificates
      • Policy Areas
      • Expert Guide
      • Degree Comparison
      • Workshops
      • Academic Calendar
    • Faculty Directory
    • Undergraduate Program

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • Overview
      • Thesis
      • Students
      • Joint Degrees
      • Team
    • Study Abroad in Barcelona
  • Admissions

    Considering Applying?

    Get a jump start now by creating an account with our simple online tool.

    Get Started

    Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

    • Why Public Policy?
    • Applying to Harris

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • UChicago College Graduating Class
      • International Students
      • FAQ
      • Admissions Resources
      • Harris Scholars Program
    • Requirements & Deadlines
    • Meet with Harris

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • Connect with Our Students
      • Schedule a Visit
      • Meet Our Staff
      • Diversity Day
    • Financing Your Degree

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • Scholarships & Fellowships
      • Loans and Work-Study
      • Third-Party Funding
      • Tuition & Expenses
    • Admissions Blog
    • Request Information
  • Student Life

    Considering Applying?

    Get a jump start now by creating an account with our simple online tool.

    Get Started

    Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

    • What to Expect

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • From Orientation to Graduation
      • Connect with Your Community
    • The Harris Experience

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • Our Campus Community
      • City of Chicago
      • Academic Support Programs
      • Mentor Program
      • Student Organizations
      • Leadership Credential
    • Career Development

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • Considering Harris?
      • Current Student Resources
      • Alumni Career Support
      • Internships & Funding
      • International Students
      • Career Development FAQ
      • Meet the CDO Team
    • Dean of Students Office

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

    • Research & Impact

      Considering Applying?

      Get a jump start now by creating an account with our simple online tool.

      Get Started

      Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

      • Centers & Institutes

        Considering Applying?

        Get a jump start now by creating an account with our simple online tool.

        Get Started

        Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

        • Center for Economic Policy
        • Center for Health Policy
        • Center for Human Potential and Public Policy (CHPPP)
        • Center for Impact Sciences
        • Center for Municipal Finance
        • Center for Survey Methodology
        • Cyber Policy Initiative
        • Family Self-Sufficiency Data Center
        • Affiliated Centers
      • Urban Labs
      • Initiatives & Partnerships

        Considering Applying?

        Get a jump start now by creating an account with our simple online tool.

        Get Started

        Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

        • National Partnerships
        • International Partnerships
      • Faculty Impact

        Considering Applying?

        Get a jump start now by creating an account with our simple online tool.

        Get Started

        Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

        • Alumni Impact
      • News & Events

        Considering Applying?

        Get a jump start now by creating an account with our simple online tool.

        Get Started

        Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

        • News

          Considering Applying?

          Get a jump start now by creating an account with our simple online tool.

          Get Started

          Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

          • News
          • Profile
          • Research
          • In the News
          • Confronting COVID-19
          • Black History Month
        • Events

          Considering Applying?

          Get a jump start now by creating an account with our simple online tool.

          Get Started

          Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

          • Academic Conferences
          • Admissions Events
          • Alumni Events
          • Student Events
          • Special Events
          • Visiting Speakers
          • Past Events
          • International Policy and Development: Faculty Insight Series
        • Not Another Politics Podcast

          Considering Applying?

          Get a jump start now by creating an account with our simple online tool.

          Get Started

          Still have questions? Check out Admissions for details on requirements, deadlines, and financial aid.

          • Podcasts Archive: Radio Harris and Day One (2016–19)
        • 20ish Questions
        • Admissions Blog

      Utility

      • Search
      • Log in
      • Directory
      • Hire Harris
        • Find Talent
        • Find Work
      • Info For
        • Prospective Students
        • Admitted Students (Full-Time)
        • New EMP Students
        • Current Students
        • Faculty and Staff
        • Alumni
        • Media
      • Support Harris
        • Ways to Give
        • Harris Society
      Sort

      How Can We Evaluate Government Programs If We Can't Trust the Data?

      Representative Paul Ryan and Senator Patty Murray, co-sponsors of the Evidence-Based Policymaking Commission Act, November 13, 2013
      April 01, 2016
      Research
      Aimee Levitt
      Bruce Meyer found the social safety net is twice as effective as we thought. Now he's helping Congress realize the value of evidence-based policy.

      Bruce Meyer, the McCormick Foundation Professor at Chicago Harris, may not have succeeded in forging eternal peace between Republicans and Democrats in Congress and the White House, but his research has contributed to something both sides can agree on: HR 1831, otherwise known as the Evidence-Based Policymaking Commission Act of 2015.

      The bipartisan bill, proposed last spring by House Speaker Paul Ryan (then chair of the House Budget Committee) and Senator Patty Murray, will establish a commission to investigate ways in which survey data and administrative data can be used to better evaluate federal programs. The bill passed in the Senate on March 16. First passed in the House of Representatives last July, it now returns to the House for final approval before heading to the president’s desk. The origin of the bill goes back to discussions between Ryan and Meyer’s student James Sullivan. Sullivan, a former visiting assistant professor at Harris, now an associate professor of economics at Notre Dame and director of the university’s Lab for Economic Opportunities, met Ryan through their work with Catholic Charities and began discussing ways to eliminate poverty in America. When the conversation turned to evaluating anti-poverty programs, Meyer joined in the discussion and, eventually, the writing of the bill with Ryan’s staff and other academics.

      Meyer’s recent research has emphasized how data that a commission might make available could be instrumental in evaluating whether public welfare programs work – or don’t. This work has been brought to the attention of lawmakers and the general public thanks to coverage in the Wall Street Journal, theWashington Post and Vox.

      Traditionally, federal and state governments have used data from household surveys, notably the Current Population Survey, to understand social welfare and insurance programs. In theory, these surveys (administered by phone, mail or in-person visits) are supposed to provide information on income distribution and poverty rates. They are also supposed to provide feedback on how well programs intended to alleviate poverty are doing. But Meyer and his colleagues who study government safety net programs have suspected for a while now that the survey data is not as accurate as it should be.

      “There have been little bits and pieces of evidence for a long time,” Meyer says, “suggesting that surveys are not doing a good job at capturing what programs people receive and how much money they get.”

      It’s not that people are lying, Meyer says. He suspects they’re just sick of taking surveys. “Any time you buy something online, call someone to complain about something, go to the doctor, get your car repaired, stay at a hotel, you’re asked to fill out a survey,” Meyer explains. “It used to be thrilling to have someone ask you about your life. Now if you need that thrill, you post on Facebook. People don’t quite feel obligated to respond to surveys anymore.”

      And so, even when people are legally obligated to fill out a survey, as they are with the U.S. Census and the American Community Survey (formerly the Census long form), they’re not as committed to filling in the data accurately – their exact income, for example, or their program enrollment. Yet this information becomes the data the government uses to calculate poverty statistics, evaluate whether the social programs designed to alleviate poverty are working and determine who is missed by these programs.

      “It’s unfortunate,” says Meyer,“that the most important source of information about those at the bottom of the income distribution is household surveys that are increasingly unreliable and inaccurate. The broadest evidence of this problem is that if you count up the dollars people report in surveys and scale them up to represent the entire country, there are many fewer dollars than the government has paid out.”

      Meyer believed it was possible to measure the inaccuracy of the household survey data. He would do this by comparing the survey responses of households to the records of government payments to those same individuals. The only problem was, the government data was not readily available to citizens, even social science researchers. Fortunately, he had an idea.

      Early in the 2000s, he’d met with Robert Doar, who was at the time the New York State Commissioner of Temporary and Disability Assistance. Meyer had advised him on issues concerning the state’s welfare policy. Doar, who now studies poverty programs at the American Enterprise Institute, was impressed. “He’s a great scholar,” he says of Meyer, “the real deal. He comes at it straight and calls ’em as he sees ’em.” Along with Doar, Meyer met other people in the state bureaucracy – notably George Falco, an advisor to the head of the state welfare office – who were sympathetic to his project and helped him get access to New York’s data for food stamps, Temporary Assistance for Needy Families (TANF), General Assistance and subsidized housing from 2007 through 2012.

      This was not an easy process. Meyer estimates it took six or seven years to navigate the state and federal bureaucracy. He had to submit a proposal to describe what he planned to do with the data. He had to undergo a high-level security clearance. He was fingerprinted three times. When he finally had approval to access the data at the Chicago Federal Reserve Bank, he had to go through three sets of locked doors. He couldn’t remove anything from the office. He had to submit the results of his analysis to a review board.

      Once he finally had access to the administrative data, Meyer compared them to the Current Population Survey (CPS) results for the same years using social security numbers that had been encrypted in order to protect the recipients’ personal information. Then he analyzed the information to find variation between the survey and administrative data.What he found was astonishing.

      The CPS missed more than one-third of housing assistance recipients in New York State, 40 percent of food stamp recipients, and 60 percent of TANF and General Assistance recipients. The survey data underreported the incomes of the poorest households; government transfer payments from all four programs were under- reported by more than $1,400 per person. Most significantly, the administrative data showed that the four programs reduced the poverty rate among recipients by 5.3 percent, not the 2.8 percent reported by CPS.

      “Our much-maligned government programs are more effective than even Census Bureau reports indicate,” says Meyer.

      In October, the American Enterprise Institute published a paper coauthored by Meyer and Nikolas Mittag, an assistant professor at CERGE-EI (Center for Economic Research and Graduate Education- Economics Institute) at Charles University in Prague, that described the research in exhaustive detail. In the press coverage that followed, journalists interpreted it positively.

      “No matter the reason,the fact that we depend on an increasingly faulty system to define things like poverty and judge programs that aim to alleviate it, is disconcerting,” Roberto A. Ferdman wrote in the Washington Post’s Wonkblog. “It can have a profound effect on policy discussions concerning the two. On the one hand, it makes it look like the poor are doing much worse than they are.The official poverty rate now is higher than it was three decades ago, but by almost any measure the poor are better off than they were then.”

      Dylan Matthews went even further in Vox: “The emerging, positive view of transfer programs is an indication that we know how to tackle this problem, and that even greater transfers could conceivably end poverty in America altogether.”

      Politically, Meyer’s findings, as Doar noted in an op-ed piece in the Wall Street Journal, are as bipartisan as can be. They’re good news for Democrats and other progressives who favor expanding the social safety net: they show that these programs are, indeed, working. They are also good news for Republicans and others who favor greater accountability and less spending for government programs: they show that these programs can keep going without the investment of additional resources. It would make sense for the government to do a similar comparative study on a national scale. The only problem is, although Meyer feels that New York is reasonably representative of the rest of the nation, administrative datasets that encompass social welfare programs for all 50 states don’t exist. Although program guidance comes from the Federal government, much of the data is at the state level.

      “If I wanted to evaluate the impact of programs at the local level,” explains James Sullivan, “I would have to negotiate data-share agreements with individual states and counties, and that would lead to a stalemate.”

      Adds Meyer, “If administrators can have an excuse not to make data available to the public, they may use that excuse to prevent careful scrutiny of the job they’re doing.”

      This is where the Evidence-Based Policymaking Commission Ace comes in. “Sullivan suggested to Congressman Ryan that one thing that’s helpful in making better policy is to have data to evaluate current programs and possible alternatives to what we’re doing,” says Meyer. “That’s how the idea of a commission to encourage the assembly of data for evidence-based policymaking got started.”

      Both Meyer and Sullivan gave feedback on early drafts of the bill. “It was a lot of fun,” Meyer says. “I’d never seen how a rough idea can end up in legislative language and be refined over time.” The experience has made him appreciate the importance of writing policy memos that make statistics understandable to policymakers (a skill, he says, that has served him well in presenting his research to legislators).

      Ryan introduced the bill last April; it cleared the House and moved onto the Senate, where its sponsor is a Democrat, Patty Murray of Washington, at the end of July. Meyer is uncertain when it will come up for a vote. If it passes and becomes law, he’s also uncertain whether he’ll be asked to serve on the commission.

      “The important thing is that the commission be established. Federal agencies aren’t doing many things that are vital to figuring out which programs work and which don’t. Even showing how useful existing administrative and survey data can be in designing policy is a valuable thing for the commission to do, even if the laws on data access aren’t changed.”

      Related stories

      War Games

      April 01, 2015
      Hilarie Koplow-McAdams, AM'87

      Questions for Hilarie Koplow-McAdams, AM'87

      April 01, 2015
      Brett Goldstein

      Urban Legend

      April 01, 2016
      • Prospective Students
        • How to Apply
        • Why Public Policy
        • Financial Aid
        • Events
      • Current Students
        • Academic Advising & Support
        • Financial Aid for Students
        • Courses
        • Course Evaluations
        • Find a Job or Internship
        • Disability Resources
      • Faculty & Staff
        • Room Reservations
        • Academic Calendar
        • Faculty Resources
        • Faculty Access
        • Human Resources
      • Alumni
        • News and Events
        • Alumni Directory
        • Get Involved
        • Donate Now
      Back to Top

      Footer menu

      • The University of Chicago
      • Apply
      • Careers at Harris
      • Courses
      • Directions
      • Privacy Notice

      Connect with Us

      • Facebook
      • Twitter
      • LinkedIn
      • Youtube

      Questions?

      Contact Us