How Can We Evaluate Government Programs If We Can't Trust the Data?

Bruce Meyer, the McCormick Foundation Professor at Chicago Harris, may not have succeeded in forging eternal peace between Republicans and Democrats in Congress and the White House, but his research has contributed to something both sides can agree on: HR 1831, otherwise known as the Evidence-Based Policymaking Commission Act of 2015.

The bipartisan bill, proposed last spring by House Speaker Paul Ryan (then chair of the House Budget Committee) and Senator Patty Murray, will establish a commission to investigate ways in which survey data and administrative data can be used to better evaluate federal programs. The bill passed in the Senate on March 16. First passed in the House of Representatives last July, it now returns to the House for final approval before heading to the president’s desk. The origin of the bill goes back to discussions between Ryan and Meyer’s student James Sullivan. Sullivan, a former visiting assistant professor at Harris, now an associate professor of economics at Notre Dame and director of the university’s Lab for Economic Opportunities, met Ryan through their work with Catholic Charities and began discussing ways to eliminate poverty in America. When the conversation turned to evaluating anti-poverty programs, Meyer joined in the discussion and, eventually, the writing of the bill with Ryan’s staff and other academics.

Meyer’s recent research has emphasized how data that a commission might make available could be instrumental in evaluating whether public welfare programs work – or don’t. This work has been brought to the attention of lawmakers and the general public thanks to coverage in the Wall Street Journal, theWashington Post and Vox.

Traditionally, federal and state governments have used data from household surveys, notably the Current Population Survey, to understand social welfare and insurance programs. In theory, these surveys (administered by phone, mail or in-person visits) are supposed to provide information on income distribution and poverty rates. They are also supposed to provide feedback on how well programs intended to alleviate poverty are doing. But Meyer and his colleagues who study government safety net programs have suspected for a while now that the survey data is not as accurate as it should be.

“There have been little bits and pieces of evidence for a long time,” Meyer says, “suggesting that surveys are not doing a good job at capturing what programs people receive and how much money they get.”

It’s not that people are lying, Meyer says. He suspects they’re just sick of taking surveys. “Any time you buy something online, call someone to complain about something, go to the doctor, get your car repaired, stay at a hotel, you’re asked to fill out a survey,” Meyer explains. “It used to be thrilling to have someone ask you about your life. Now if you need that thrill, you post on Facebook. People don’t quite feel obligated to respond to surveys anymore.”

And so, even when people are legally obligated to fill out a survey, as they are with the U.S. Census and the American Community Survey (formerly the Census long form), they’re not as committed to filling in the data accurately – their exact income, for example, or their program enrollment. Yet this information becomes the data the government uses to calculate poverty statistics, evaluate whether the social programs designed to alleviate poverty are working and determine who is missed by these programs.

“It’s unfortunate,” says Meyer,“that the most important source of information about those at the bottom of the income distribution is household surveys that are increasingly unreliable and inaccurate. The broadest evidence of this problem is that if you count up the dollars people report in surveys and scale them up to represent the entire country, there are many fewer dollars than the government has paid out.”

Meyer believed it was possible to measure the inaccuracy of the household survey data. He would do this by comparing the survey responses of households to the records of government payments to those same individuals. The only problem was, the government data was not readily available to citizens, even social science researchers. Fortunately, he had an idea.

Early in the 2000s, he’d met with Robert Doar, who was at the time the New York State Commissioner of Temporary and Disability Assistance. Meyer had advised him on issues concerning the state’s welfare policy. Doar, who now studies poverty programs at the American Enterprise Institute, was impressed. “He’s a great scholar,” he says of Meyer, “the real deal. He comes at it straight and calls ’em as he sees ’em.” Along with Doar, Meyer met other people in the state bureaucracy – notably George Falco, an advisor to the head of the state welfare office – who were sympathetic to his project and helped him get access to New York’s data for food stamps, Temporary Assistance for Needy Families (TANF), General Assistance and subsidized housing from 2007 through 2012.

This was not an easy process. Meyer estimates it took six or seven years to navigate the state and federal bureaucracy. He had to submit a proposal to describe what he planned to do with the data. He had to undergo a high-level security clearance. He was fingerprinted three times. When he finally had approval to access the data at the Chicago Federal Reserve Bank, he had to go through three sets of locked doors. He couldn’t remove anything from the office. He had to submit the results of his analysis to a review board.

Once he finally had access to the administrative data, Meyer compared them to the Current Population Survey (CPS) results for the same years using social security numbers that had been encrypted in order to protect the recipients’ personal information. Then he analyzed the information to find variation between the survey and administrative data.What he found was astonishing.

The CPS missed more than one-third of housing assistance recipients in New York State, 40 percent of food stamp recipients, and 60 percent of TANF and General Assistance recipients. The survey data underreported the incomes of the poorest households; government transfer payments from all four programs were under- reported by more than $1,400 per person. Most significantly, the administrative data showed that the four programs reduced the poverty rate among recipients by 5.3 percent, not the 2.8 percent reported by CPS.

“Our much-maligned government programs are more effective than even Census Bureau reports indicate,” says Meyer.

In October, the American Enterprise Institute published a paper coauthored by Meyer and Nikolas Mittag, an assistant professor at CERGE-EI (Center for Economic Research and Graduate Education- Economics Institute) at Charles University in Prague, that described the research in exhaustive detail. In the press coverage that followed, journalists interpreted it positively.

“No matter the reason,the fact that we depend on an increasingly faulty system to define things like poverty and judge programs that aim to alleviate it, is disconcerting,” Roberto A. Ferdman wrote in the Washington Post’s Wonkblog. “It can have a profound effect on policy discussions concerning the two. On the one hand, it makes it look like the poor are doing much worse than they are.The official poverty rate now is higher than it was three decades ago, but by almost any measure the poor are better off than they were then.”

Dylan Matthews went even further in Vox: “The emerging, positive view of transfer programs is an indication that we know how to tackle this problem, and that even greater transfers could conceivably end poverty in America altogether.”

Politically, Meyer’s findings, as Doar noted in an op-ed piece in the Wall Street Journal, are as bipartisan as can be. They’re good news for Democrats and other progressives who favor expanding the social safety net: they show that these programs are, indeed, working. They are also good news for Republicans and others who favor greater accountability and less spending for government programs: they show that these programs can keep going without the investment of additional resources. It would make sense for the government to do a similar comparative study on a national scale. The only problem is, although Meyer feels that New York is reasonably representative of the rest of the nation, administrative datasets that encompass social welfare programs for all 50 states don’t exist. Although program guidance comes from the Federal government, much of the data is at the state level.

“If I wanted to evaluate the impact of programs at the local level,” explains James Sullivan, “I would have to negotiate data-share agreements with individual states and counties, and that would lead to a stalemate.”

Adds Meyer, “If administrators can have an excuse not to make data available to the public, they may use that excuse to prevent careful scrutiny of the job they’re doing.”

This is where the Evidence-Based Policymaking Commission Ace comes in. “Sullivan suggested to Congressman Ryan that one thing that’s helpful in making better policy is to have data to evaluate current programs and possible alternatives to what we’re doing,” says Meyer. “That’s how the idea of a commission to encourage the assembly of data for evidence-based policymaking got started.”

Both Meyer and Sullivan gave feedback on early drafts of the bill. “It was a lot of fun,” Meyer says. “I’d never seen how a rough idea can end up in legislative language and be refined over time.” The experience has made him appreciate the importance of writing policy memos that make statistics understandable to policymakers (a skill, he says, that has served him well in presenting his research to legislators).

Ryan introduced the bill last April; it cleared the House and moved onto the Senate, where its sponsor is a Democrat, Patty Murray of Washington, at the end of July. Meyer is uncertain when it will come up for a vote. If it passes and becomes law, he’s also uncertain whether he’ll be asked to serve on the commission.

“The important thing is that the commission be established. Federal agencies aren’t doing many things that are vital to figuring out which programs work and which don’t. Even showing how useful existing administrative and survey data can be in designing policy is a valuable thing for the commission to do, even if the laws on data access aren’t changed.”