Commission on Evidence-Based Policymaking the Promise of Evidence-Based Policymaking Report of the Commission on Evidence-Based Policymaking

THE PROMISE OF EVIDENCE-BASED POLICYMAKING Report of the Commission on Evidence-Based Policymaking The Promise of Evidence-Based Policymaking Report of the Commission on Evidence-Based Policymaking

September 2017

The Promise of Evidence-Based Policymaking i

COMMISSIONERS KATHARINE G. ABRAHAM SHERRY GLIED ROBERT M. GROVES CHAIR ROBERT HAHN HILARY HOYNES JEFFREY B. LIEBMAN BRUCE D. MEYER RON HASKINS PAUL OHM CO-CHAIR NANCY POTOK KATHLEEN RICE MOSIER

Commission on Evidence-Based Policymaking ROBERT J. SHEA LATANYA SWEENEY 4600 SILVER HILL ROAD WASHINGTON, DC 20233 KENNETH R. TROSKE WASHINGTON, DC 20233 KIM R. WALLIN

September 7, 2017

The President of the United States The Speaker of the House The President of the Senate

Dear Sirs:

On behalf of the Commission on Evidence-Based Policymaking, and pursuant to Public Law 114- 140, we are pleased to submit the following report detailing our findings and recommendations. We believe that adoption of the recommendations contained in our report will prove transformational, improving both the privacy protections afforded to the American public and the availability of rigorous evidence to inform policymaking.

Over the course of its work, the Commission heard from numerous expert witnesses and members of the public, received extensive written public comment, and fielded a survey of Federal agencies. This fact-finding phase was followed by several months of deliberations. The recommendations contained in the report have been agreed to unanimously by the 15 Members of the Commission.

The Commission envisions a future in which rigorous evidence is created efficiently, as a routine part of government operations, and used to construct effective public policy. Advances in technology and statistical methodology, coupled with a modern legal framework and a commitment to transparency, make it possible to do this while simultaneously providing stronger protections for the privacy and confidentiality of the people, businesses, and organizations from which the government collects information.

We call on the Congress and the President to act to ensure that this vision becomes a reality.

Respectfully yours,

Katharine G. Abraham, Chair Ron Haskins, Co-Chair

Sherry Glied Robert M. Groves Robert Hahn Hilary Hoynes Jeffrey B. Liebman

Bruce D. Meyer Paul Ohm Nancy Potok Kathleen Rice Mosier Robert J. Shea

Latanya Sweeney Kenneth R. Troske Kim R. Wallin

The Promise of Evidence-Based Policymaking iii

Contents

Executive Summary: The Promise of Evidence-Based Policymaking...... 1 Commission on Evidence-Based Policymaking Recommendations...... 5

Chapter 1. Introduction: Vision for Evidence-Based Policymaking...... 7 Defining Evidence-Based Policymaking...... 8 Evolution of Evidence Building in the United States...... 12 Commission’s Charge and Process...... 16 Realizing the Promise of Evidence-Based Policymaking...... 17

Chapter 2. Secure, Restricted Access to Confidential Data...... 23 Findings...... 24 Recommendations...... 40

Chapter 3. Enhancing Privacy Protections for Federal Evidence Building...... 47 Findings...... 50 Recommendations...... 60

Chapter 4. Modernizing America’s Data Infrastructure for Accountability and Privacy...... 65 Findings ...... 68 Recommendations...... 80

Chapter 5. Strengthening the Evidence-Building Capacity within the Federal Government...... 87 Findings...... 88 Recommendations...... 101

Chapter 6. Conclusion: Possibilities with More and Better Evidence...... 105

Acronyms ...... 107 Appendix A: Evidence-Based Policymaking Commission Act of 2016...... 109 Appendix B: Commissioner Biographies...... 115 Appendix C: Commission’s Fact-Finding and Deliberative Processes...... 119 Appendix D: Examples of Data Productive for Evidence Building...... 125

The Promise of Evidence-Based Policymaking 1

Executive Summary: The Promise of Evidence-Based Policymaking

he American people want a government that and statistical methods, combined with trans- T functions efficiently and responsibly ad- parency and a strong legal framework, create dresses the problems that face this country. Pol- the opportunity to use data for evidence build- icymakers must have good information on which ing in ways that were not possible in the past. to base their decisions about improving the viabil- This report describes the Commission’s findings ity and effectiveness of government programs and and presents recommendations for fundamental policies. Today, too little evidence is produced to improvements to the Federal government’s evi- meet this need. dence-building systems and capabilities. Specif- The Commission on Evidence-Based Policy- ically, the Commission’s report includes recom- making (the “Commission”) envisions a future in mendations on (1) how the Federal government which rigorous evidence is created efficiently, as a can provide the infrastructure for secure access to routine part of government operations, and used data, (2) the mechanisms to improve privacy pro- to construct effective public policy. The Federal tections and transparency about the uses of data government has already taken important steps for evidence building, and (3) the institutional towards accomplishing this vision, but much capacity to support evidence building. work remains. The growing interest in producing more and higher-quality evidence to support de- Recommendations for cision-making led the Congress and the President to enact the Evidence-Based Policymaking Com- Improving Secure, Private, and mission Act of 2016, creating the Commission. Confidential Data Access The Commission was provided just over a year to study and develop a strategy for strengthening There are many barriers to the effective use of gov- government’s evidence-building and policymak- ernment data to generate evidence. Better access ing efforts. During the Commission’s fact-finding to these data holds the potential for substantial phase, numerous experts, researchers, govern- gains for society. The Commission’s recommenda- ment leaders, public and private organizations, tions recognize that the country’s laws and prac- and members of the public offered their perspec- tices are not currently optimized to support the tives on the Commission’s charge. use of data for evidence building, nor in a manner Based on this collective input, the Commission that best protects privacy. To correct these prob- determined that greater use of existing data is now lems, the Commission makes the following rec- possible in conjunction with stronger privacy and ommendations: legal protections, as well as increased transparency and accountability. The Commission believes • Establish a National Secure Data Service to that improved access to data under more priva- facilitate access to data for evidence building cy-protective conditions can lead to an increase while ensuring privacy and transparency in in both the quantity and the quality of evidence to how those data are used. As a state-of-the-art inform important program and policy decisions. resource for improving government’s capacity Traditionally, increasing access to confidential to use the data it already collects, the National data presumed significantly increasing privacy Secure Data Service will be able to temporarily risk. The Commission rejects that idea. The Com- link existing data and provide secure access to mission believes there are steps that can be tak- those data for exclusively statistical purposes en to improve data security and privacy protec- in connection with approved projects. The Na- tions beyond what exists today, while increasing tional Secure Data Service will do this without the production of evidence. Modern technology creating a data clearinghouse or warehouse. 2 The Promise of Evidence-Based Policymaking

• Require stringent privacy qualifications for de-identified confidential data intended for acquiring and combining data for statistical public release to improve how data are protected purposes at the National Secure Data Service and risk is managed. to ensure that data continue to be effectively protected while improving the government’s • Adopt modern privacy-enhancing technologies ability to understand the impacts of programs for confidential data used for evidence build- on a wider range of outcomes. At the same ing to ensure that government’s capabilities to time, consider additional statutory changes keep data secure and protect confidentiality to enable ongoing statistical production that, are constantly improving. under the same stringent privacy qualifications, may make use of combined data. • Assign senior officials the responsibility for stewarding data within government depart- • Review and, where needed, revise laws autho- ments. Agencies should improve leadership, rizing Federal data collection and use to ensure coordination, and collaboration when imple- that limited access to administrative and survey menting protections for the use of confidential data is possible to return benefits to the public data. through improved programs and policies, but only under strict privacy controls. • Codify policies for maintaining integrity and objectivity in Federal statistics to promote con- • Ensure state-collected quarterly earnings data tinued public trust in the accuracy of informa- are available for statistical purposes, including tion being used to guide government decision- to support the many evidence-building activities making. for which earnings are an important outcome.

• Make additional state-collected data about Recommendations for Federal programs available for evidence build- Implementing the National ing. Where appropriate, states that administer Secure Data Service programs with substantial Federal investment should in return provide the data necessary for The Commission’s recommendations for improved evidence building. data access and strong privacy protections rely heavily on the establishment of the National • Develop a uniform process for external research- Secure Data Service. Being able to combine data ers to apply and qualify for secure access to con- within a secure environment will be an increas- fidential government data for evidence-building ingly vital aspect of the evidence-building com- purposes while protecting privacy by carefully munity’s capacity to meet future demand from restricting data access to qualified and approved policymakers. Increased transparency will enable researchers. the public to be informed about how data are being used to improve their government, even as data are being stringently protected. The Commission Recommendations for envisions that the National Secure Data Service Modernizing Privacy Protections will operate an effective and efficient service that for Evidence Building can be held accountable by policymakers and the American public. The Commission’s recommenda- Enhancements to privacy, coupled with improved tions to implement the National Secure Data Ser- methods for secure data access, will revolutionize vice include: how government uses and protects the data it collects. Among the Commission’s recommendations • Build on the infrastructure and expertise al- to achieve this vision are: ready developed in government, including at the U.S. Census Bureau, to ensure that data • Require comprehensive risk assessments on linkages and access to confidential data for The Promise of Evidence-Based Policymaking 3

statistical purposes are conducted in the most • Develop learning agendas in Federal depart- secure manner possible. ments to support the generation and use of evidence to address the range of policymakers’ • Require public input, guidance, and partici- questions. pation in the policies and procedures for data linkage activities through public and stake- • Improve coordination of government-wide evi- holder representation on the National Secure dence building by directing the Office of Man- Data Service’s steering committee. agement and Budget to facilitate cross-government coordination, and consider how a greater • Establish a new transparency and accountabil- commitment to foundational information pol- ity portal for evidence-building activities to icy responsibilities can be achieved, including ensure the public is notified about how con- through any consolidation or reorganization at fidential data are used for evidence building the Office of Management and Budget that may and to document routine audits for compliance be necessary. with rules governing privacy, confidentiality, and data access. • Align administrative processes with evidence- building activities, including those relating to • Innovate continuously on privacy-protective the approval of information collections and the data access approaches with sufficient admin- procurement of services for evidence building. istrative flexibilities to ensure government can adjust as technology advances. • Ensure that sufficient resources to support evidence-building activities are available, includ- • Increase efforts to make information available ing resources to support implementation of the about the government’s current data invento- Commission’s recommendations. ries and supply related data documentation to help researchers inside and outside govern- Conclusion ment know which data they need to evaluate programs and policies. Generating and using evidence to inform government policymaking and program administration is not a partisan issue. The strategy described in this Recommendations for report offers a non-partisan approach to improv- Strengthening Federal ing how government staff, private researchers, Evidence-Building Capacity foundations, non-profits, the business community, and the public interact to make sure govern- More privacy protective approaches and improved ment delivers on its promises. access to data alone will not improve the volume The Commission’s recommendations represent and quality of evidence. The evidence-building a comprehensive strategy for tackling the greatest community also needs sufficient capacity, admin- problems facing evidence building today—data istrative flexibilities, and appropriate program de- access is limited, privacy-protecting practices are sign to enable a strong and effective evidence-gen- inadequate, and the capacity to generate the evi- eration system to operate. To strengthen the dence needed to support policy decisions is insuf- evidence-building capacity within the Federal ficient. The Congress, the President, and the Amer- government, the Commission makes the follow- ican people are ill-served by this state of affairs. ing recommendations: Government must do what it takes to increase the quantity and quality of evidence building. • Identify or establish a Chief Evaluation Officer The strategy outlined in the Commission’s report in each department to coordinate evaluation simultaneously improves privacy protections and and policy research and to collaborate with makes better use of data the government already other evidence-building functions within Fed- collects to support policymaking. Together with eral departments. leadership from the President and the Congress 4 The Promise of Evidence-Based Policymaking

in calling for credible evidence to support policy how to improve processes for efficiently providing decisions throughout government, implementa- services, evidence should play an important role tion of the Commission’s recommendations is an in key decisions made by government officials. important step in providing the country with an The Commission proposes modernizing the coun- effective government. try’s evidence-building capacity to make sure our Whether deciding on funding allocations, as- government’s decision-making process is among sessing proposed regulations, or understanding the best in the world, now and in the future. ■ The Promise of Evidence-Based Policymaking 5

Commission on Evidence-Based Policymaking Recommendations

Note: Recommendations in this report are numbered REC. 2-6: The Congress and the President should sequentially to align with the discussion. For example, enact statutory or other changes to ensure that 2-1 refers to the first recommendation in Chapter 2. state-collected administrative data on quarterly earnings are available for solely statistical purposes. The data should be available through a single Improving Secure, Private, and Federal source for solely statistical purposes. Confidential Data Access REC. 2-7: The President should direct Federal de- REC. 2-1: The Congress and the President should partments that acquire state-collected adminis- enact legislation establishing the National Secure trative data to make them available for statisti- Data Service (NSDS) to facilitate data access for cal purposes. Where there is substantial Federal evidence building while ensuring transparency investment in a program, Federal departments and privacy. The NSDS should model best prac- should, consistent with applicable law, direct tices for secure record linkage and drive the im- states to provide the data necessary to support ev- plementation of innovative privacy-enhancing idence building, such as complete administrative technologies. data when samples are already provided.

REC. 2-2: The NSDS should be a service, not a data REC. 2-8: The Office of Management and Budget clearinghouse or warehouse. The NSDS should fa- should promulgate a single, streamlined process cilitate temporary data linkages in support of dis- for researchers external to the government to ap- tinct authorized projects. ply, become qualified, and gain approval to access government data that are not publicly available. REC. 2-3: In establishing the NSDS, the Congress Approval would remain subject to any restrictions and the President should amend the Privacy Act appropriate to the data in question. and the Confidential Information Protection and Statistical Efficiency Act (CIPSEA) to require new stringent privacy qualifications as a precondition Modernizing Privacy Protections for the NSDS to acquire and combine survey and for Evidence Building administrative data for solely statistical purposes. At the same time, the Congress should consider REC. 3-1: The Congress and the President should additional statutory changes to enable ongoing amend the Privacy Act and the Confidential In- statistical production. formation Protection and Statistical Efficiency Act (CIPSEA) to require Federal departments to REC. 2-4: The Congress and the President should conduct a comprehensive risk assessment on review and amend, as appropriate, statutes such de-identified confidential data intended for public as Title 13 of the U.S. Code to allow statistical release. De-identified confidential data subject to uses of survey and administrative data for evi- the Privacy Act and CIPSEA should only be made dence building within the CIPSEA secure envi- available after a disclosure review board (1) ap- ronment. proves the release and (2) publicly provides the risk assessment and a description of steps taken REC. 2-5: The Congress and the President should to mitigate risk. consider repealing current bans and limiting future bans on the collection and use of data for ev- REC. 3-2: The President should direct Federal de- idence building. partments, in coordination with the National 6 The Promise of Evidence-Based Policymaking

Secure Data Service, to adopt state-of-the-art da- REC. 4-4: The NSDS should have specific adminis- tabase, cryptography, privacy-preserving, and pri- trative and implementation flexibilities including vacy-enhancing technologies for confidential data the ability to leverage public-private partnerships used for evidence building. and to collect and retain user fees.

REC. 3-3: The President should direct Federal de- REC. 4-5: The Office of Management and Budget partments to assign a senior official the responsi- should increase efforts to make information avail- bility for coordinating access to and stewardship able on existing Federal datasets including data of the department’s data resources for evidence inventories, metadata, and data documentation in building in collaboration with senior department a searchable format. information technology, privacy, and other leaders. A Principal Statistical Agency head, or other appropriately qualified senior official, should Strengthening Federal serve this function. Evidence-Building Capacity

REC. 3-4: The Congress and the President should REC. 5-1: The President should direct Federal enact legislation to codify relevant portions of Of- departments to increase capacity for evidence fice of Management and Budget Statistical Policy building through the identification or establish- Directive #1 to protect public trust by ensuring ment of a Chief Evaluation Officer, in addition to that data acquired under a pledge of confidential- needed authorities to build a high performing evi- ity are kept confidential and used exclusively for dence-building workforce. statistical purposes. REC. 5-2: The Congress and the President should direct Federal departments to develop multi-year Implementing the National learning agendas that support the generation and Secure Data Service use of evidence.

REC. 4-1: The National Secure Data Service (NSDS) REC. 5-3: The Congress and the President should should be established as a separate entity in the direct the Office of Management and Budget Department of Commerce that builds upon and (OMB) to coordinate the Federal government’s enhances existing expertise and infrastructure in evidence-building activities across departments, the Federal government, especially at the Census including through any reorganization or consoli- Bureau, to ensure sufficient capacity in secure re- dation within OMB that may be necessary and by cord linkage and data access for evidence building. bolstering the visibility and role of interagency councils. REC. 4-2: The NSDS should establish a Steering Committee that includes representatives of the REC. 5-4: The Congress and the President should public, Federal departments, state agencies, and align administrative processes to support evi- academia. dence building, in particular by streamlining the approval processes for new data collections and REC. 4-3: To ensure exemplary transparency and using existing flexibilities in procurement policy. accountability for the Federal government’s use of data for evidence building, the NSDS should REC. 5-5: The Congress and the President should maintain a searchable inventory of approved proj- ensure sufficient resources to support evi- ects using confidential data and undergo regular dence-building activities about Federal govern- auditing of compliance with rules governing pri- ment programs and policies. ■ vacy, confidentiality, and access. The Promise of Evidence-Based Policymaking 7

Introduction: Vision for Evidence-Based Policymaking

ith the passage and signing of the Evi- in the country, it is notable that this legislation Wdence-Based Policymaking Commission was embraced by legislators on both sides of the Act in the spring of 2016, elected leaders issued aisle and enacted without dissent. U.S. House of a bipartisan call to improve the evidence avail- Representatives Speaker Paul Ryan, a co-spon- able for making decisions about government sor of the Act, described the potential for evi- programs and policies.1 (See the box “Charge to dence-based policymaking as a “sea change in the Commission on Evidence-Based Policymak- how we solve problems.” Likewise, co-sponsor ing.”) In an environment of growing partisanship Senator Patty Murray said: “Whether you think we need more government, or less government—

1. Evidence-Based Policymaking Commission Act of 2016 (Public you should agree that we should at least have Law 114–140, March 30, 2016). better government.”

Charge to the Commission on Evidence-Based Policymaking

In the Evidence-Based Policymaking Commis- and related statistical data series may be sion Act of 2016 (see Appendix A), the Congress integrated and made available for evidence and the President prescribed a number of duties building while protecting privacy and to the Commission, including the following: confidentiality.

• Study the data inventory, data • Make recommendations on how best to infrastructure, database security, and incorporate evidence building into program statistical protocols related to Federal design. policymaking. Make recommendations on how data infrastructure, database security, • Consider whether a “clearinghouse” and statistical protocols should be modified. for program and survey data should be established and how to create such a • Determine the optimal arrangement for “clearinghouse.” which administrative data, survey data, 8 The Promise of Evidence-Based Policymaking

The American people want a government that ican public’s ability to hold the government ac- solves problems. This requires that decision mak- countable. Adhering to the highest possible stan- ers have good information to guide their choic- dards with respect to privacy and accountability es about how current programs and policies are is an important part of earning the public’s trust. working and how they can be improved. While the The improvements to privacy and accountability Federal government has already taken steps to- that the Commission envisions can occur simulta- wards developing an “evidence culture,” much re- neously with providing policymakers the tools to mains to be done. A particularly important barrier deliver more effective government services. to government’s further progress is lack of access The Commission envisions a future in which by researchers outside of government and by indi- rigorous evidence is created efficiently, as a rou- viduals within government to the data necessary tine part of government operations, and used to for evidence building, even when those data have construct effective public policy. While this may already been collected. sound like a daunting task, the Commission’s vi- While collecting taxes, determining eligibility sion for the future of evidence-based policymaking for government benefits, engaging in econom- in the United States is well within reach (see the ic development, and running programs, govern- box “Examples of the Promise of Evidence-Based ment necessarily collects a considerable amount Policymaking”). This vision requires that new laws of information. In 2017, the American public will and policies be put into place. When implemented, spend nearly 12 billion hours responding to more the Commission is confident that the approaches than 100 billion individual requests for informa- proposed in this report will greatly improve both tion from the Federal government.2 Even though the ability to produce evidence in support of bet- the direct costs of collecting these data are funded ter policies and privacy protections for individuals by taxpayers, these data are not generally available and organizations. for producing evidence. Addressing barriers to the use of already collected data is a path to unlocking Defining Evidence-Based important insights for addressing society’s greatest challenges. Policymaking As the use of existing government data to support policymaking grows, the American public “Evidence” can be defined broadly as information will be concerned about exactly how those data that aids the generation of a conclusion. Through- are being used and whether the privacy and con- out this report, the Commission uses the term in a fidentiality of individuals and organizations are more specific way—this report uses the shorthand being protected. Today, data are protected, in part, “evidence” to refer to information produced by through pledges of confidentiality, privacy laws, “statistical activities” with a “statistical purpose” and legal and policy limitations on how they are that is potentially useful when evaluating govern- used, but the government’s approach to data pro- ment programs and policies. Following U.S. Office tection has not kept pace with important changes of Management and Budget (OMB) Statistical Pol- in technology. icy Directive #1, which in turn follows the Con- Capabilities now exist to improve privacy pro- fidential Information Protection and Statistical tections while making better use of already collect- Efficiency Act of 2002 (CIPSEA), we define “sta- ed administrative data, including recent advanc- tistical activities” as “the collection, compilation, es in statistical methodology, computer science, processing, analysis, or dissemination of data for and computational capacity. Growing experience the purpose of describing or making estimates with successful legal models for data stewardship concerning the whole, or relevant groups or com- points in the same direction. Government also ponents within, the economy, society, or the nat- can dramatically improve transparency about its ural environment, including the development of collection and use of data, improving the Amer- methods or resources that support those activities, such as measurement of methods, statistical clas- sifications, or sampling frames.” A “statistical pur- 2. Office of Information and Regulatory Affairs, Office of Man- pose” is defined as “the description, estimation, or agement and Budget, Inventory of Currently Approved Infor- mation Collections; https://www.reginfo.gov/public/do/PRAReport? analysis of the characteristics of groups, without operation=11 (accessed August 10, 2017). identifying the individuals or organizations that The Promise of Evidence-Based Policymaking 9

Examples of the Promise of Evidence-Based Policymaking

Through transactions with the American pub- safety net.4 These studies were carried out us- lic, governments collect a considerable amount ing a combination of survey and administra- of data. These administrative data, collected tive data, including administrative data from in the first instance to serve routine program locally operated Homeless Management Infor- operation purposes, also can be used to assess mation Systems. how well programs are achieving their intend- As a direct result of this growing body of ev- ed goals.1 Below are examples where admin- idence, in recent years, the U.S. Department of istrative data were used to generate evidence Housing and Urban Development (HUD) has that informed government policies. encouraged and incentivized communities to Permanent Supportive Housing. There increase their supply of permanent supportive is a growing body of research on the impact housing for chronically homeless individuals of providing permanent supportive housing over the past several years.5 Notably, there has to chronically homeless individuals.2 This re- been a 27 percent reduction in chronic home- search demonstrates that an intervention lessness nationally between the years 2010 combining long-term housing assistance and 2016.6 with supportive services can help chronically Substance Abuse Education. The Drug homeless individuals maintain stable housing Abuse Resistance Education program (DARE), and achieve other positive outcomes, such as created in 1983, originally aimed to prevent improved health outcomes and reduced use drug use and gang membership for kindergarten of crisis services, including costly emergen- through 12th grade students in Los Angeles. In cy room visits or stays in a homeless shelter.3 partnership with local law enforcement officers, Cost-effectiveness studies of the intervention DARE grew into a national program focused pri- also suggest that offering permanent support- marily on drug prevention that at its peak was ive housing to chronically homeless individu- in over 75 percent of the schools in the United als with the highest service needs can reduce States and in more than 50 countries.7 taxpayer costs for other components of the More than 30 rigorous evaluations conducted throughout the 1990s and 2000s suggested that the original DARE program did not produce —continues

1. The Evidence-Based Policymaking Commission Act of 2016 (Public Law 114–140, March 30, 2016) defines “administrative 4. S. R. Poulin, M. Maguire, S. Metraux, and D. P. Culhane. “Ser- data” as data “(1) held by an agency or contractor or grantee of vice Use and Costs for Persons Experiencing Chronic Home- an agency (including a State or unit of local government); and lessness in Philadelphia: A Population-Based Study,” Psychiat- (2) collected for other than statistical purposes.” ric Services 61, no. 11 (2010): 1093–1098; M.E. Larimer, D.K, 2. The U.S. Department of Housing and Urban Development Malone, M.D. Garner and others, “Health Care and Public Ser- adopted the Federal definition, which defines a chronically vice Use and Costs Before and After Provision of Housing for homeless person as “either (1) an unaccompanied homeless Chronically Homeless Persons With Severe Alcohol Problems.” individual with a disabling condition who has been continu- Journal of the American Medical Association, 301, no. 13 (April ously homeless for a year or more, or (2) an unaccompanied 1, 2009): 1349–1357. individual with a disabling condition who has had at least 5. See HUD’s Fiscal Year 2018 Congressional Justification for four episodes of homelessness in the past three years.” (See the Homeless Assistance Grant Program for more information; 11 Code of Federal Regulations 91 and 578, 2015.) https://portal.hud.gov/hudportal/documents/huddoc?id=22- 3. Dennis P. Culhane and Thomas Byrne,Ending Chronic HomelessAGrants.pdf (accessed August 10, 2017). Homelessness: Cost-Effective Opportunities for Interagency Col- 6. HUD, The 2016 Annual Homeless Assessment Report (AHAR) to laboration. Federal Strategic Plan Supplemental Document Congress; Part I: Point-in-Time Estimates of Homelessness (Wash- No. 19 (Washington, D.C.: U.S. Interagency Council on Home- ington, D.C.” U.S. Government Printing Office, 2016). lessness, 2010); https://www.usich.gov/resources/uploads/asset _library/DennisCulhane_EndingChronicHomelessness.pdf 7. For information about DARE America, go to http://www. (accessed August 10, 2017). dare.org/about-d-a-r-e (accessed August 10, 2017). 10 The Promise of Evidence-Based Policymaking

Examples of the Promise of Evidence-Based Policymaking— continued

substantial reductions in teenage substance participants can realize improved earnings abuse over the long-term.8 One study carried and employment outcomes, though the ev- out in a suburban setting even found that the idence is mixed on specific strategies.14 The intervention could contribute to increases in U.S. Department of Labor’s Adult Program as- drug use.9 In 2001, the Surgeon General sum- sists people who are economically disadvan- marized the available research and designated taged facing barriers to employment. The Dis- DARE as an “ineffective primary prevention located Workers Program assists workers who program” but also stated “its popularity persists have been laid off or who have been notified despite numerous well-designed evaluations that they will be terminated or laid off. Both and meta-analyses that consistently show little programs provide a range of training and sup- or no deterrent effects on substance abuse.”10 portive services. In a study using administra- The DARE program partnered with Penn- tive data in a non-experimental program eval- sylvania State University to adopt a new ele- uation, researchers found that participants in mentary and middle school curriculum called the adult program experienced an increase in “keepin it REAL.”11 Today, the DARE program quarterly earnings relative to a comparison focuses on a broader vision of empowering group, while participants in the dislocated students to respect others and choose to lead workers program actually saw reduced earn- lives free from violence, substance abuse, and ings in several quarters.15 When the workforce other dangerous behaviors.12 Preliminary ev- investment programs were reauthorized in idence from the revised curriculum suggests 2014 through the Workforce Innovation and more promising effectiveness at achieving the Opportunity Act (WIOA), the new law provid- stated goals related to decision-making.13 ed states additional flexibility to shift fund- Workforce Investment. A large portfolio ing between the adult and dislocated worker of evidence about workforce investments and aspects of the program to better target local job training programs suggests that program needs. WIOA included numerous other evidence-informed strategies based on the exist- 8. Greg Berman and Aubrey Fox, Lessons from the Battle Over ing portfolio of evidence. DARE (Center for Court Innovation and Bureau of Justice As- Implementation of the permanent support- sistance of the U.S. Department of Justice, 2009); http://www. ive housing, DARE, and workforce investment courtinnovation.org/sites/default/files/DARE.pdf (accessed Au- gust 10, 2017). programs each were influenced by evidence developed to inform the implementation of 9. Dennis P. Rosenbaum and Gordon S. Hanson, “Assessing the Effects of School-Based Drug Education: A Six Year Multi-Level Federal policies. With more evidence to inform Analysis of Project DARE,” Journal of Research in Crime and De- a range of policy interests and questions, poli- linquency 35, no. 4 (1998): 381–412. cymakers will have a stronger basis for making 10. U.S. Department of Health and Human Services, Youth Vi- decisions in the future. olence: A Report of the Surgeon General (Washington, D.C: De- partment of Health and Human Services, 2001): 110. 14. Office of Management and Budget (OMB), “Using Admin- 11. DARE America, “Keepin it Real Elementary School istrative and Survey Data to Build Evidence,” white paper for Curriculum;” https://www.dare.org/keepin-it-real-elementary- the Commission on Evidence-Based Policymaking (Wash- school-curriculum (accessed August 10, 2017). ington, D.C.: OMB, Executive Office of the President, 2016); 12. DARE America, “D.A.R.E.’s keepin’ it REAL Elementary https://obamawhitehouse.archives.gov/sites/default/files/omb/ and Middle School Curriculums Adhere to Lessons From mgmt-gpra/using_administrative_and_survey_data_to_build_ Prevention Research Principles;” https://www.dare.org/d-a-r- evidence_0.pdf (accessed August 10, 2017). e-s-keepin-it-real-elementary-and-middle-school-curriculums- 15. Caroline J. Heinrich, Peter R. Mueser, Kenneth R. Troske, adhere-to-lessons-from-prevention-research-principles (accessed Kyung-Seong Jeon, and Daver C. Kahvecioglu, “Do Public August 10, 2017). Employment and Training Programs Work?” IZA Journal of La- 13. Randy Borum and David Allan Verhaagen, Assessing and bor Economics 2, no. 6 (2013); https://izajole.springeropen.com/ Managing Violence Risk in Juvenile (New York: Guilford, 2006). articles/10.1186/2193-8997-2-6 (accessed August 10, 2017). The Promise of Evidence-Based Policymaking 11

comprise such groups; and includes the develop- purposes such as the use of individual-level in- ment, implementation, or maintenance of meth- formation to determine benefit amounts, enforce ods, technical or administrative procedures, or in- laws, or otherwise affect the rights or privileges formation resources that support such purposes.”3 of an individual. The identification of a single in- We return to these definitions in Chapter 2 in the dividual means the information is not being used discussion of CIPSEA. The essence of a “statistical for statistical activities and thus, would not be activity” with a “statistical purpose” is that the re- termed “evidence” in the Commission’s defini- sult summarizes information about a group rather tion. Throughout this report, the Commission has than a single individual or organization. For exam- been mindful that, consistent with applicable law, ple, a statistical activity could include analyzing efforts to make data available specifically for sta- a “unit,” such as a state or a grantee, in order to tistical purposes might also inadvertently put in- generate average values for all of the individuals formation about individuals at increased risk for included within that unit, such as residents, cli- use in other ways. The Commission’s proposals at- ents, or firms. tempt to ensure strict structural and institutional Data can be used for many purposes other than separation between statistical and non-statistical evidence building. These include non-statistical uses of data. The Commission’s evidence-building reforms are engineered to make data difficult to repurpose for non-statistical uses. 3. Statistical Policy Directive No. 1, Federal Register 79 (Decem- ber 2, 2014): 71609–71616; https://www.gpo.gov/fdsys/pkg/ The Commission defines evidence-based poli- FR-2014-12-02/pdf/FR-2014-12-02.pdf. cymaking as the application of evidence to inform

Questions to Answer with More and Better Evidence

The Commission’s fact-finding process pro- • Do Farm Service Agency programs ease duced numerous examples of important ques- credit constraints for farmers? tions that individuals who provided input to the Commission reported cannot currently • What impacts do Federal economic devel- be adequately addressed because of difficulty opment efforts, such as the Appalachian accessing the right data: Regional Commission’s grants, have on the communities they are trying to help? • What effects does the Supplemental Nu- trition Assistance Program have on health • What enforcement approaches are most outcomes? effective in improving clean air regulation compliance? • Are the earnings of veterans improved by training received while in the military? Each of these questions could be studied using administrative data that the government • Which transition-to-adulthood experiences already collects. Too often, however, the capac- make students with disabilities less likely ity and infrastructure to study pressing ques- to rely on the Supplemental Security In- tions faced by decision-makers are lacking. The come program? Commission’s vision for evidence-based policymaking would enable each of these import- • To what extent do eligible active-duty mil- ant policy questions to be addressed with ap- itary households participate in antipoverty propriate information analyzed in a secure and programs and how does this participation privacy-protected environment, and then used affect their economic self-sufficiency? to improve government policies and programs. 12 The Promise of Evidence-Based Policymaking

decisions in government. For evidence-based pol- Evolution of Evidence Building in icymaking to occur, a supply of evidence must first exist. Thus, the Commission recognizes that the United States evidence-based policymaking requires the gen- The nation’s founders recognized the importance eration of evidence, which relies on access to of information for governance, requiring in the data. As the evidence base becomes stronger, the U.S. Constitution a census of the population (see American public should expect that policymakers Figure 1).5 James Madison argued that collecting increasingly will incorporate new and better ev- more data about the populace could guide con- idence into their decisions about the operation gressional decisions about government actions as of government programs and funding for govern- the young country grew.6 Early censuses gathered ment services. information about industry, agriculture, and the Different types of evidence are relevant for population. Census questions changed over time, policymaking and may involve a variety of meth- reflecting important societal and governmental ods.4 Descriptive statistics provide insights about information needs of the day. trends and context. Performance metrics support By the mid-nineteenth century, the Congress monitoring of policy outputs and efficiency. Im- had established several permanent units to pro- plementation and process studies can identify duce national statistics in specific policy areas. how well the application of programs and policies More statistical units were added in subsequent aligns with their intended design and goals. Im- years and formed the basic information infra- pact evaluations provide insights about wheth- structure of the nation’s first two centuries. Over er desired outcomes are achieved. Each of these the years, the statistics on population size, edu- types of evidence and others are relevant for ev- cation, employment, gross domestic product, and idence-based policymaking, and the appropriate others became a routine dashboard on what was approach depends on the policymakers’ question happening in society. (see the box “Questions to Answer with More and Today, the evidence-building community oper- Better Evidence”). ates under a range of laws, regulations, and poli- This report uses the term “evidence-building cies that evolved over time, a state of affairs that community,” which is meant to describe the col- has contributed to a lack of coordination and collective set of individuals located both inside and laboration across the community. Numerous com- outside the Federal government who fulfill a set missions or committees were convened during of roles key to generating evidence for use in pol- the 20th century to recommend improvements icymaking. The evidence-building community infor the country’s evidence-building system (see cludes individuals situated across government and online Appendix H). These included recommen- in the business, non-profit, and academic sectors. dations for greater coordination of activities and The community includes individuals who perform for enhancements to the protection of privacy.7 statistical activities, such as collecting data to pro- Information policy setting and coordination duce national indicators relevant to the country. across government began in earnest in 1939 when The community includes researchers who study President Franklin D. Roosevelt issued an Execu- ways to improve government’s programs and pol- tive Order directing the Bureau of the Budget “to icies and evaluators who assess whether those plan and promote the improvement, development programs and policies are achieving their intend- and coordination of Federal and other statistical ed goals. The community also includes individuals who support program administrators with analysis to achieve targeted improvements to their programs and policies. 5. U.S. Constitution Article I, Section 2. 6. Annals of Congress, House of Representatives, 1st Congress, 2nd Session “Remarks by James Madison on the Bill for the 1790 Census” (1790): 1145. 4. OMB, “Using Administrative and Survey Data to Build Evi- 7. Janet Norwood, Organizing to Count: Change in the Federal Statis- dence,” 2016. tical System (Washington, D.C.: Urban Institute Press, 1995). The Promise of Evidence-Based Policymaking 13

services.”8 The Congress subsequently provided Figure 1. Key Milestones in the additional authority to conduct central reviews U.S. Evidence-Building System for new data collections in government with the Federal Records Act of 1942.9 1787 U.S. Constitution Signed, 1960s and 1970s: Expansion of Evidence Calls for Decennial Census Building and Privacy Protections 1903 Commission Appointed by As more data were collected and used by govern- Secretary of the U.S. Treasury to ment for implementing programs and for statis- Investigate Accounting Methods tical activities, the need for privacy protections became increasingly apparent. In 1973, the U.S. 1942 Federal Reports Act Department of Health, Education, and Welfare developed the Fair Information Practice Principles Federal Records Act (FRA) (FIPPs), recommending that the Congress enact 1950 them into what eventually became the Privacy Act Budget and Accounting 10 of 1974. The principles include transparency, in- Procedures Act dividual participation, purpose specification, data minimization, use limitation, data quality and integrity, security, and accountability and auditing. Dept. of Health, Education, and These principles strive to balance the need for in- 1973 Welfare Develops Fair Information formation with privacy protections for the benefit Practice Principles (FIPPs) 1974 of the American public. The Privacy Act also codified some U.S. infor- Privacy Act mation practices, establishing common require- 1977 ments related to collecting, maintaining, using, Privacy Protection Study Commission and disseminating government records about individuals. The Privacy Act articulated basic trans- 1980 Paperwork Reduction Act (PRA) parency requirements and limitations on how data collected by the government may be disclosed. In 1977, the Privacy Protection Study Commission created by the Privacy Act conveyed two central 1993 Government Performance and tenets for evidence building. First, research and Results Act (GPRA) statistical uses of data about individuals must ex- 1995 PRA Reauthorized clude any result that would directly affect an individual’s rights, privileges, or benefits. Second, government statistical and non-statistical uses of 2000 Information Quality Act data should be separated by a bright line, a prin- Conﬁdential Information 11 2002 ciple referred to as “functional separation.” These Protection and Statistical Eciency Act (CIPSEA)

8. Executive Order 824: Establishing the Divisions of the Exec- E-Government Act utive Office of the President and Defining Their Functions and 2010 Duties, 3 C.F.R. (September 8, 1939); https://www.archives.gov/ GPRA Modernization Act federal-register/codification/executive-order/08248.html (accessed August 10, 2017). 9. Records Management by Federal Agencies, 44 USC § 3101 et. seq. 2016 Commission on Evidence-Based 10. U.S. Department of Health, Education, and Welfare, Records, Policymaking Created Computers, and the Rights of Citizens: Report of the Secretary’s Adviso- ry Committee on Automated Personal Data Systems (1973). Legend Laws Commissions 11. Personal Privacy in an Information Society, The Report of the Pri- vacy Protection Study Commission, July 1977. *Timeline not to scale 14 The Promise of Evidence-Based Policymaking

two core concepts carry forward into the nation’s form the Federal Statistical System.15 The PSAs privacy protection and statistical laws, and remain benefit from explicit statutory authorities and key to this Commission’s work. an established culture that supports secure and Amidst discussions about improving privacy, confidential data collection and stewardship. Ac- approaches to generating and using evidence ex- cording to the American Statistical Association, panded. The U.S. Department of Defense intro- “Federal statistical agencies will play a vital role in duced cost-benefit analysis as part of budgeting evidence-based policymaking and…the historical activities and the technique gained further trac- autonomy of these agencies [is needed] in order tion in the regulatory infrastructure.12 Similarly, in to ensure the integrity of their work.”16 The Com- the 1960s and 1970s, the Department of Health, mission agrees with this assessment. Coordina- Education, and Welfare pioneered many of the ap- tion for the statistical system is supported by the proaches for executing government’s evaluation statutory Interagency Council on Statistical Policy function in partnership with contractors, grant- and the Federal Committee on Statistical Method- ees, and academics, to meet the need for better ology, and led by the Chief Statistician in OMB.17 information about the impacts of Great Society OMB also issues standards and directives to assure programs. Many of these efforts created the basic the quality and integrity of activities conducted infrastructure and resource mechanisms to sup- within the system. port evaluation of education, human services, and With the Government Performance and Results health programs. Act of 1993 (GPRA), the Congress also established an expectation for continuous improvement in 1980s and 1990s: Improved Evidence government by creating government-wide re- Coordination to Meet Demand quirements for departments to set goals, measure results, and report progress.18 As evidence-building activities became more common across government, gaps in coordina- 2000s: Strengthened Protections for tion mechanisms became more apparent. The Pa- Statistical Activities perwork Reduction Act (PRA) of 1980 specifically recognized the role of the OMB in coordinating In recognition of the need to better protect data government-wide information and statistical pol- collected by the Federal Statistical System, CIPSEA icy, which the Congress reaffirmed during a 1995 established common legal authorities for statisti- reauthorization. The PRA specifically encourages cal activities. Consistent with the recommenda- data sharing for statistical activities as govern- tions of the President’s Reorganization Project for ment policy, both to improve the efficiency and the Federal Statistical System twenty years earlier, the quality of statistical series and of evidence CIPSEA provides statutory protections that apply more broadly.13 to all 13 PSAs and other designated units for data Within the decentralized Federal evidence- collected under a pledge of confidentiality for sta- building apparatus, 13 Principal Statistical Agen- tistical activities.19 cies (PSAs) in 11 Federal departments provide the core infrastructure to support government’s main 15. OMB, Statistical Programs of the United States Government statistical activities and provide access to statis- (Washington, D.C.: 2017); https://obamawhitehouse.archives.gov/ tical datasets for research (see Figure 2).14 These sites/default/files/omb/assets/information_and_regulatory_affairs/ 13 agencies, along with nearly 100 smaller units statistical-programs-2017.pdf (accessed August 10, 2017). embedded within program agencies, collectively 16. Clyde Tucker, American Statistical Association, Commission Public Hearing, Washington, D.C., October 21, 2016. 17. The Chief Statistician role and the ICSP are established under the Paperwork Reduction Act, 44 USC § 3504(e) (1995). 18. P. G. Joyce, “The Obama Administration and PBB: Building on the Legacy of Federal Performance‐Informed Budgeting?” Public 12. Edward P. Fuchs and James E. Anderson, “The Institutionaliza- Administration Review, 71, no. 3 (2011): 356–367. tion of Cost-Benefit Analysis,”Public Productivity Review 10, no. 4 (1987): 25–33. 19. James T. Bonnen, Theodore G. Clemence, Ivan P. Fellegi, Thom- as B. Jabine, Ronald E. Kutscher, Larry K. Roberson, Charles A. 13. Paperwork Reduction Act, 44 USC § 3506 (1995). Waite, “Improving the Federal Statistical System: Report of the 14. Throughout this report, the term “departments” is generally President’s Reorganization Project for the Federal Statistical Sys- used to refer to the 24 Federal agencies subject to the Chief Finan- tem,” American Statistician, 35, no. 4 (November 1981, reprint): cial Officers Act of 1990 (Public Law 101–576). 184–196. The Promise of Evidence-Based Policymaking 15

The Privacy Act, CIPSEA, and the PRA offer the Figure 2. Federal Principal legal foundation for coordinating information pol- Statistical Agencies (PSAs) icy and protecting data used in evidence building, but much work remains to realize the promis- U.S. Department of Agriculture es embodied in these laws. For example, Federal Economic Research Service evaluation and policy research offices are more nascent in their development than the statistical National Agricultural Statistics Service agencies. As such, none of them are recognized under CIPSEA to employ the strong protections U.S. Department of Commerce that statistical agencies can provide for confidential data. Bureau of Economic Analysis Census Bureau 2010s: Increased Demand for Evidence

Reauthorizations of programs by the Congress U.S. Department of Education during the 2010s provided numerous examples of the increased demand for evidence to inform pol- National Center for Education Statistics icy. The GPRA Modernization Act of 2010 reiterat- ed the Congress’s interest in better use of evidence U.S. Department of Energy for continuous improvement in government.20 The Workforce Innovation and Opportunity Act in Energy Information Administration 2014 authorized numerous evaluation and mul- tisite projects. The Agricultural Act of 2014 called U.S. Department of Health and Human Services for testing of innovative approaches to helping individuals gain and retain employment that leads National Center for Health Statistics to self-sufficiency.21 The Bipartisan Budget Act of 2015 reauthorized the Social Security Disability U.S. Department of Justice Insurance program’s demonstration authority to test policy improvements.22 The Every Student Bureau of Justice Statistics Succeeds Act in 2015 reauthorized education programs and expanded the use of evidence-based U.S. Department of Labor grants.23 Numerous other laws and draft legislation embody the increasing demand for evidence Bureau of Labor Statistics such as by incorporating requirements about using evidence or supporting innovation in award- U.S. Department of Transportation ing grants. Within the Executive Branch, calls encouraging Bureau of Transportation Statistics more and better evidence also emerged. In 2013, a memorandum to Federal departments specifi- U.S. Department of the Treasury cally encouraged using evidence that is available, Internal Revenue Service Statistics of Income Division 20. “GPRA Modernization Act of 2010” (Public Law 111–352, Jan- uary 4, 2011); https://www.congress.gov/111/plaws/publ352/PLAW- 111publ352.pdf (accessed August 10, 2017). National Science Foundation 21. “Agricultural Act of 2014” (Public Law 113–79, February 7, 2014); National Center for Science and https://www.congress.gov/113/plaws/publ79/PLAW-113publ79.pdf Engineering Statistics (accessed August 10, 2017). 22. “Bipartisan Budget Act of 2015” (Public Law 114–74, Sec. 821, November 2, 2015); https://www.congress.gov/114/plaws/publ74/ U.S. Social Security Administration PLAW-114publ74.pdf (accessed August 10, 2017). Oce of Research, 23. “Every Student Succeeds Act” (Public Law 114–95, Decem- ber 10, 2015); https://www.congress.gov/114/plaws/publ95/PLAW- Evaluation, and Statistics 114publ95.pdf (accessed August 10, 2017). 16 The Promise of Evidence-Based Policymaking

producing new evidence to increase knowledge, search organizations hosting these centers. The and continually innovating to improve govern- centers provide researchers with secure access to ment performance.24 And in 2017, the President confidential data, and the statistical agencies and specifically called for generating and using evi- general public benefit from research findings that dence to improve government’s effectiveness.25 make important contributions to statistical products and general knowledge. Partnerships for Policy Research and Evaluation Commission’s Charge and The capacity within Federal departments to per- Process form policy research and evaluation can be ar- rayed along a continuum, ranging from a strong Continuing this history of evidence-building ac- centralized approach to dispersed capacity across tions in the United States, the Commission on bureaus and units. Discussing the existing track Evidence-Based Policymaking was established record within the Federal government, the Amer- by the bipartisan Evidence-Based Policymaking ican Evaluation Association observed that “for the Commission Act of 2016 (Appendix A), jointly most part, these evaluations have been sporadic, sponsored by Speaker of the House Paul Ryan applied inconsistently, and supported inadequate- and Senator Patty Murray and signed by Presi- ly. Training and capacity building for evaluation dent Barack Obama on March 30, 2016. The Act have been inconsistent across agencies and, in directed the Commission to consider how to many cases, insufficient to achieve the needed strengthen government’s evidence-building and evaluation capacity and sustain it over time.”26 The policymaking efforts. Specifically, the Act direct- Commission agrees with this assessment. ed the Commission to study how the data that Individuals outside of government, such as government already collects can be used to im- non-governmental researchers, philanthropic prove government programs and policies. The organizations, universities, researchers, and oth- Commission was directed to submit to the Con- er partners, play an essential role in supporting gress and the President a detailed statement of and extending the evidence-building community its findings and recommendations. within government. Evidence produced by exter- The Commission recognizes that data are an nal actors can influence policymakers, encourage important building block for the generation of policy innovation, and drive the allocation of re- evidence and that many of the greatest gains for sources across new and existing programs. More- evidence building in the near term can be accom- over, external actors create a capacity that goes plished by addressing the challenge of secure, pri- well beyond that of government staff. vate, and confidential data access. A theme that Federal departments and universities have ini- runs throughout this report is that access to confi- tiated collaborations to improve evidence-build- dential data for evidence-building purposes should ing activities. For example, the Federal Statistical be increased, but only in the context of a modern Research Data Centers operate today as partner- legal framework providing for strengthened priva- ships between statistical agencies and the re- cy protections and increased transparency. Taking up the privacy challenge, the report recommends specific steps to improve privacy and transparen- 24. OMB, “Next Steps in the Evidence and Innovation Agenda” cy with regard to evidence-building activities. The (memorandum M–13–17); https://obamawhitehouse.archives.gov/ report also recognizes the need to strengthen the sites/default/files/omb/memoranda/2013/m-13-17.pdf (accessed August 10, 2017). government’s institutional capacity to support the evidence-building system. 25. Executive Office of the President, “Building and Using Ev- idence to Improve Government Effectiveness,” inAnalytical Per- Following the Commission’s first meeting in spectives: Budget of the United States for Fiscal Year 2018 (March July 2016, Commissioners engaged in an eight- 2017); https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/ month-long fact-finding process to learn about budget/fy2018/ap_6_evidence.pdf (accessed August 10, 2017). the current state of evidence production and use 26. American Evaluation Association (AEA), An Evaluation Road- in the Federal government, as well as the Federal map for a More Effective Government (Washington, D.C.: AEA, revised October 2013); http://www.eval.org/d/do/472 (accessed Au- government’s policies and practices to protect data gust 10, 2017). confidentiality (see Appendix C). The Commission The Promise of Evidence-Based Policymaking 17

solicited feedback from the public and experts by tions, the Commission developed a set of five guid- doing the following: ing principles that shaped its recommendations (see the box “Guiding Principles for Evidence-Based • Administering a survey to 209 offices of the Policymaking”). The fact-finding and deliberations Federal government that the Commission using these principles culminated in the conclu- identified as likely to be generating or using sions presented in this report and a strategy for evidence, in order to understand the current realizing the promise of evidence-based policy- or potential capacity of Federal agencies to making. Each of the 15 appointed Commissioners, engage in aspects of evidence-based policy- forming a bipartisan Commission with a range of making (see online Appendix E). perspectives on the issues examined, approved this final report and the recommendations herein. • Convening seven public meetings to hear from 49 invited expert witnesses on a range of issues (see online Appendix F). Realizing the Promise of Evidence-Based Policymaking • Issuing a Request for Comments in the Federal Register and accepted comments by email, The Commission believes that enabling improved which generated over 350 responses from the access to data under more modern privacy-pro- public (see online Appendix G). tective conditions will lead to more and better evidence. Improvement in access can occur safe- • Holding three open public hearings— ly with improved privacy protections. This report in Washington, D.C., Chicago, and San maps a path to realizing this vision. Francisco—during which 39 members of the public presented information directly to Vision for Improving Secure Access to Commissioners (see online Appendix G). Confidential Data • Meeting with more than 40 organizations or During the Commission’s fact-finding phase, Com- experts to solicit additional input. missioners heard about the many barriers to effective use of the data government already collects Following the fact-finding phase of the Commis- to generate evidence. The Commission believes sion’s work, Commissioners launched a five-month that enabling better use of such data will offer deliberative process to consider all of the gathered substantial gains for society. The Commission also input and information. At the outset of its delibera- believes that by better coordinating the access to

Guiding Principles for Evidence-Based Policymaking

1. Privacy. Individual privacy and confiden- ful channels for public input and comment tiality must be respected in the generation and ensuring that evidence produced is and use of data and evidence. made publicly available. 2. Rigor. Evidence should be developed using 4. Humility. Care should be taken not to well-designed and well-implemented over-generalize from findings that may be methods tailored to the questions being specific to a particular study or context. asked. 5. Capacity. The capacity to generate and 3. Transparency. Those engaged in gener- use data and evidence should be integrated ating and using data and evidence should within government institutions and ade- operate transparently, providing meaning- quately funded and staffed. 18 The Promise of Evidence-Based Policymaking

administrative data, government can improve the art capacity for integrating existing data and pro- security of these data while better protecting indi- viding secure data access for exclusively statistical vidual privacy. purposes (REC. 2-1). The NSDS will model best Chapter 2 describes the current flaws in the practices for secure record linkages, propel im- country’s data infrastructure and capacity to use plementation of privacy-enhancing technologies, data for evidence building. While the country’s and coordinate a new level of transparency about data infrastructure and capacity have evolved sub- how government data are used for evidence build- stantially over the past two and a half centuries, ing. Importantly, the NSDS will facilitate tempo- the Federal government, as a whole, has not kept rary linkages only for authorized projects, and will pace with emerging privacy-protecting technolo- not serve as a data warehouse (REC. 2-2). gies and protocols to support secure access to data. As explained in Chapter 2, the country’s laws While there are pockets of innovation, our govern- are not currently optimized either to support the ment has not yet broadly adopted approaches for use of data for evidence building across programs using data to generate insights that can enhance or to maximize privacy. Increasing connections decision-making that are becoming much more between data used in evidence building across prevalent in the private sector. Some other coun- topical areas will improve our ability to under- tries also have more consistently implemented the stand the impacts of programs on a wider range capabilities needed for developing and using high of outcomes. Some existing authorities enable quality evidence to inform public policy. sharing of information across agencies within the Federal government and allow access for non-Fed- “Data are the lifeblood of eral actors. These approaches could be expanded to enhance the analytical capabilities to cut across decision-making and the raw additional policy domains. material for accountability. Inconsistent laws and interpretations of le- Without high-quality data gal authorities for securely sharing and using providing the right information confidential data have inhibited the ability of the evidence-building community to access the on the right things at the right types of data. The Commission proposes right time; designing, monitoring to amend the Privacy Act and CIPSEA to require and evaluating effective policies stringent privacy qualifications for acquiring and combining data for statistical purposes, to en- becomes almost impossible.” sure that data continue to be effectively protect- – United Nations Secretary-General’s ed while improving the government’s ability to Independent Expert Advisory Group understand the impacts of programs on a wider on a Data Revolution for Sustainable range of outcomes (REC. 2-3). In some cases, the Development27 purposes for which administrative data may be used are defined narrowly, preventing their use As the amount of data available increases ex- for evidence building. The Commission proposes ponentially, improving government’s capacity a review of such statutes to ensure that limita- to securely analyze information that crosses the tions that preclude the use of administrative data silos of government is an ongoing challenge. The for evidence building are applied only when the Commission proposes to answer this challenge by Congress and the President deem the limitations building on and reorganizing existing resources still to be necessary (REC. 2-4). In some cases, within government through the National Secure existing laws specifically prohibit the collection Data Service (NSDS), establishing a state-of-the- or analysis of information to support evidence building. Again, the Commission calls for a reconsideration of such bans and restraint in the 27. United Nations, Secretary General’s Independent Expert Ad- visory Group on a Data Revolution for Sustainable Development. enactment of future bans (REC. 2-5). A World That Counts: Mobilizing the Data Revolution for Sustainable Chapter 2 also highlights the value of data Development (New York, NY: United Nations, November 2014); http:// collected by states and other jurisdictions about www.undatarevolution.org/wp-content/uploads/2014/12/A-World- That-Counts2.pdf (accessed August 10, 2017). Federal programs and policies, especially data The Promise of Evidence-Based Policymaking 19

on income, wages, and earnings, while noting unevenly across government. Many are not dy- that these data in most cases are not currently namic enough to meet the ever-changing risks being used productively for evidence building.28 associated with the use of data. While the Federal Multiple programs supported with Federal funds government is subject to many transparency re- and administered by states and other jurisdic- quirements, government could be more open in tions collect income, wage, or earnings data in explaining how individuals’ data are used for evi- the course of program operations. Some of these dence building. data are provided to the Federal government for Chapter 3 discusses privacy as a broad concept specific purposes, but numerous researchers and encompassing the full range of the Fair Informa- evaluators reported to the Commission that action Practice Principles, which provide a useful cessing the data for all states is nearly impossible. framework for the Commission’s recommenda- The Commission believes that data about earnings tions. Confidentiality is a distinct but overlapping are among the most important for enhancing the concept in that it refers specifically to the protec- generation of evidence about Federal government tion of information about individuals or other en- programs. The Commission strongly encourag- tities from unauthorized disclosure.29 Confiden- es the Congress and the President to make these tiality is a component of privacy, but protecting data available for statistical purposes (REC 2-6). and respecting privacy entails substantially more In addition, many Federal programs rely on states than just ensuring the confidentiality of informa- and other jurisdictions to collect data, but those tion. For convenience, the Commission often uses data are not currently available for evaluating the terms “privacy” or “privacy protection” in the programs and policies. An expectation should be broader sense of the full range of Fair Informa- established that these data will be available for ev- tion Practice Principles, including confidentiality. idence building (REC. 2-7). Therefore, throughout this report, privacy and pri- A related issue frequently reported to the vacy protection should be interpreted to include Commission is the confusion caused by variation the confidentiality of individuals’ information, across agencies in the approval processes through and, where appropriate, the protection of infor- which external researchers establish eligibility to mation about businesses and other entities for access confidential data held by Federal depart- which the term privacy in a narrower sense may ments. The Commission recognizes that while not normally apply. embedding a certain amount of friction in these Chapter 3 describes how risks to privacy have processes can enhance privacy—that is, that ac- evolved and shows why they must be continu- cess should not be too easy—but needless varia- ally assessed as new information is made more tion can create unintentional barriers that serve publicly available. Government has an obligation no deliberate purpose. To better facilitate access to be open and honest with the American pub- while protecting privacy, the Commission recom- lic. The public’s trust can be earned only through mends the creation of a single process for exter- transparency about these risks. The Commission nal researchers to apply and qualify for access to recommends that, prior to any public release of sensitive government data that are not otherwise de-identified confidential data, Federal depart- publicly available, subject to any restrictions ap- ments assess the risks that may be associated with propriate to the data in question (REC. 2-8). the release and the steps taken to mitigate those risks, then make these risk assessments publicly Vision for Modernizing Privacy available (REC. 3-1). This recommendation is one Protections for Evidence Building of several intended to improve how the Federal government manages privacy risks. Many existing privacy protections in the Unit- As technologies evolve, the capabilities to keep ed States are strong, but protections are applied data secure and protect confidentiality are constantly improving. The Commission calls on government to make better use of state-of-the-art 28. Throughout this report, the phrase “other jurisdictions” should generally be interpreted to mean “Tribes, territories, local governments (such as cities, counties, or districts), or other governing entities.” 29. Personal Privacy in an Information Society, 1977. 20 The Promise of Evidence-Based Policymaking

approaches that can be applied for enhancing se- held accountable for safely managing those risks. curity and confidentiality for data used in evidence The Commission recommends the creation of a building (REC. 3-2). single new Federal transparency portal to ensure With respect to all of these activities, the Com- that information about approved evidence-build- mission recognizes a need for leadership with- ing activities that rely on confidential data is read- in Federal departments. The Commission rec- ily available. Then, routine audits of government ommends that a senior official be designated to and external users of data should be carried out ensure appropriate collaboration with regard to to ensure that, as promised, data are accessed in decisions about data stewardship and the imple- the most privacy-protective fashion and for solely mentation of privacy-protective measures within statistical purposes (REC. 4-3). each department (REC. 3-3). The Commission also Approaches to statistical analysis of govern- recommends that existing policies for maintaining ment data that protect privacy will continue to the integrity and objectivity of Federal statistics evolve as technologies advance. Government will be codified in law, maintaining the public’s trust need to innovate continuously with respect to pri- by ensuring that the confidential data used to pro- vacy-protective approaches. The NSDS must play duce these statistics continue to be protected and an important role in the process of continuous used only for statistical purposes (REC. 3-4). improvement and be afforded the administrative These enhancements to privacy, coupled with flexibilities to support innovation and to engage the improved strategy for secure data access out- in partnerships that can bring needed expertise to lined in Recommendation 2-1, will advance how bear (REC. 4-4). government uses and protects the data it already Improving protections and capabilities for bet- collects, allowing the data to be accessed securely. ter using data also requires new approaches to understanding what data exist about Federal pro- Implementing the Vision grams and policies. The Commission recommends the creation of a complete inventory of Federal A core feature for implementing the Commission’s data assets with technical information that will vision is the creation of the NSDS as a new service help members of the evidence-building commu- that builds on and enhances existing Federal gov- nity inside and outside government know which ernment data infrastructure for statistical activi- data are available to evaluate programs and poli- ties (REC 2-1). The ability to temporarily combine cies (REC. 4-5). identifiable data within a secure environment, Even after the NSDS data capabilities have then remove direct identifiers for individual proj- been established, much work will remain to im- ects is a vital element of the evidence-building plement other aspects of the Commission’s vision. community’s ability to meet future demand from More privacy protective approaches and improved policymakers. For the NSDS to be a role model for access to data alone will not improve the volume the use of data and held accountable by the Amer- and quality of evidence. Ultimately, the capacity ican public, it should have five core features. These of the Federal evidence-building community must are described further in Chapter 4. be strengthened to increase its efficacy. First, the NSDS should build on the infrastruc- Chapter 5 emphasizes the need for leadership, ture and expertise already developed across gov- coordination, and resources to facilitate efficient ernment, such as at the Census Bureau, to ensure evidence-building activities that support contin- that statistical analyses of confidential data are uous learning in the Federal government. Depart- conducted in the most secure manner possible ments need to coordinate their evidence-building (REC. 4-1). As the data work of the NSDS pro- functions internally to ensure the activities can be ceeds, the public must be afforded opportunities effectively and efficiently implemented. The Com- to participate by providing input and guidance on mission recommends that Federal departments the policies and procedures for conducting these identify or establish a Chief Evaluation Officer efforts through representation on the NSDS’s to encourage and coordinate policy research and steering committee (REC. 4-2). evaluation (REC. 5-1). Programs can also be more The statistical uses of confidential data can effectively designed to enable evidence building. pose risks to privacy, and government must be The Commission recommends that departments The Promise of Evidence-Based Policymaking 21

create learning agendas that outline priorities and be applied or created to tailor these processes to support the development of all types of evidence better facilitate the need for increased evidence to help address the range of policymakers’ ques- building (REC. 5-4). Finally, appropriate resourc- tions (REC. 5-2). es must be available to support evidence build- In addition to coordination within depart- ing, including resources to enable the full suite of ments, better coordination is also needed across activities discussed throughout this report, from government. OMB and Federal departments must data stewardship to risk assessments to the evalu- be organized to implement this vision. This may ations of individual programs (REC. 5-5). require consolidation or reorganization of certain In short, government must improve its exist- activities at OMB where information policy and ing infrastructure and legal frameworks to enable other evidence-building activities are currently more and better evidence building. The strategy spread across the organization (REC. 5-3). outlined in this report both improves privacy pro- The Commission also heard about administra- tections and makes better use of data the govern- tive processes in government that inadvertently ment has already collected. The remainder of the inhibit evidence-building activities. These includ- report outlines the challenges to increasing the ed processes for reviewing information collec- volume, quality, and utility of evidence generated tions under the PRA and procurement policies. to support policymakers and makes specific rec- The Commission recommends specific flexibilities ommendations for improvements. ■

The Promise of Evidence-Based Policymaking 23

2 Secure, Restricted Access to Confidential Data

Equip the evidence-building community with secure, restricted access to data to facilitate the generation of high quality evidence about government programs and policies.

Recommendations 2-1: The Congress and the President 2-5: The Congress and the President should enact legislation establishing the should consider repealing current bans National Secure Data Service (NSDS) to facili- and limiting future bans on the collection and tate data access for evidence building while use of data for evidence building. ensuring transparency and privacy. The NSDS 2-6: The Congress and the President should model best practices for secure record should enact statutory or other changes linkage and drive the implementation of in- to ensure that state-collected administrative novative privacy-enhancing technologies. data on quarterly earnings are available for 2-2: The NSDS should be a service, not a solely statistical purposes. The data should be data clearinghouse or warehouse. The available through a single Federal source for NSDS should facilitate temporary data linkag- solely statistical purposes. es in support of distinct authorized projects. 2-7: The President should direct Federal 2-3: In establishing the NSDS, the Con- departments that acquire state-collect- gress and the President should amend ed administrative data to make them avail- the Privacy Act and the Confidential Informa- able for statistical purposes. Where there is tion Protection and Statistical Efficiency Act substantial Federal investment in a program, (CIPSEA) to require new stringent privacy Federal departments should, consistent with qualifications as a precondition for the NSDS applicable law, direct states to provide the to acquire and combine survey and adminis- data necessary to support evidence building, trative data for solely statistical purposes. At such as complete administrative data when the same time, the Congress should consider samples are already provided. additional statutory changes to enable ongo- 2-8: The Office of Management and ing statistical production. Budget should promulgate a single, 2-4: The Congress and the President streamlined process for researchers external should review and amend, as appropri- to the government to apply, become qualified, ate, statutes such as Title 13 of the U.S. Code and gain approval to access government data to allow statistical uses of survey and admin- that are not publicly available. Approval istrative data for evidence building within the would remain subject to any restrictions ap- CIPSEA secure environment. propriate to the data in question. 24 The Promise of Evidence-Based Policymaking

ata are an essential ingredient for the that access to confidential data for evidence build- Dsuccess of evidence-based policymaking, ing should be enhanced, but in a fashion that also providing the foundation for basic descriptive improves privacy protections. statistics that describe the status quo as well as conclusions from the most sophisticated program Findings evaluations. The routine administrative operations associated with many Federal activities re- During the Commission’s fact-finding phase, nu- sult in the collection of data from program partic- merous expert witnesses, commenters, and gov- ipants. These administrative data, along with the ernment agencies identified examples in which multitude of surveys and other data collections, successful access to data generated information have value for evidence building. important for improving programs and policies. Data collection is resource intensive for the They often involved an individual or research government and burdensome for the public. There team with sufficient persistence and creativity is strong reason to carefully increase the use of ex- to successfully navigate—or even pioneer the use isting data to the extent possible before imposing of—available processes for accessing data. For ex- new burdens on individuals, businesses, and other ample, Stanford University Professor of Econom- organizations. Leveraging administrative data can ics Raj Chetty and his collaborators produced improve statistical products, reduce the costs and path-breaking research on social mobility.1 Such administrative burden associated with creating success stories, however, highlight common bar- statistics and conducting research and evaluation, riers to accessing critical data that, in too many and thereby greatly expand the evidence base for cases, have prevented researchers from produc- setting Federal policy (see the box “Reducing the ing analyses of important policy questions. Many Respondent Burden of Income Questions on Sur- of these same challenges have been identified by veys”). The Commission believes the Federal gov- past commissions and panels tasked with explor- ernment can more efficiently and effectively use ing the Federal Statistical System or making rec- the data it collects, while simultaneously enhanc- ommendations about policy in a particular area.2 ing privacy protection. Accumulated experience offers valuable lessons In a well-designed system, access to data and about how to establish processes that can make the protection of privacy can work hand in hand. ongoing analyses of important programs and pol- The Commission’s conception of “secure data ac- icies a norm in government, rather than a rarity. cess” is prescribed narrowly to include access for To access confidential data for the development exclusively statistical purposes—that is, the anal- of statistics, evaluation, and policy research, mem- ysis of data to generate inferences about groups. A bers of the evidence-building community today, well-designed system for secure data access is also the means for achieving greater transparency with 1. Raj Chetty, Stanford University, Commission Meeting, Wash- respect to how data are being used, an important ington, D.C., July 22, 2016. element for increasing privacy and accountability. 2. National Academies of Sciences, Engineering, and Medicine, In- Access to data held by the government should novations in Federal Statistics: Combining Data Sources While Protecting occur only in service to the public interest. Deci- Privacy (Washington, D.C.: National Academies Press, 2017); Federal sions about allowing data access must be calibrat- Statistics: Report of the President’s Commission, Volume I–II (Washing- ton, D.C.: U.S. Government Printing Office, 1971);https://catalog. ed according to a project’s potential public ben- hathitrust.org/Record/011325636 (accessed August 10, 2017); James efits, the sensitivity of a particular dataset, and T. Bonnen, Theodore G. Clemence, Ivan P. Fellegi, Thomas B. Jabine, any risk that allowing access could pose to confi- Ronald E. Kutscher, Larry K. Roberson, and Charles A. Waite, “Im- proving the Federal Statistical System: Report of the President’s dentiality. Access can and should be restricted to Reorganization Project for the Federal Statistical System,” American eligible individuals who demonstrate an under- Statistician 35, no. 4 (November 1981, reprint): 184–196; National standing of their obligations for data stewardship. Commission on Hunger, Freedom From Hunger: An Achievable Goal for the United States of America, Recommendations of the National Com- Restricted access has been standard practice in mission on Hunger to Congress and the Secretary of the Depart- many areas of government for decades. However, ment of Agriculture, 2015; https://digital.library.unt.edu/ark:/67531/ duplicative processes for accessing confidential metadc799756 (accessed August 10, 2017); Commission to Elimi- nate Child Abuse and Neglect Fatalities, Within Our Reach: A National data can distract from confidentiality and the pro- Strategy To Eliminate Child Abuse and Neglect Fatalities (Washington, motion of transparency. The Commission believes D.C.: Government Printing Office, 2016). The Promise of Evidence-Based Policymaking 25

both inside and outside the Federal government, departments are also making such data available must navigate a complex array of processes, pro- to external researchers. Still, significant barriers to tocols, and approaches. They must negotiate legal data access remain. Starting a new statistical proj- documents and bureaucratic processes that in- ect using administrative data is complicated and crease in volume and complexity when using data time consuming. Even departments that regularly from multiple policy domains, jurisdictions, or use administrative data for evidence building find agencies. Often, such processes consider the value it onerous to navigate each originating agencies’ proposition of data use only in the context of the existing processes for acquiring administrative mission of the originating agency, irrespective of data for statistical use. Agencies just starting to its broader value. For example, generally Title 26 explore the power of administrative data for im- of the U.S. Code limits the use of tax data to those proving their programs and policies often waste projects that would improve “tax administration.” months wading through various applicable laws The application of this narrow standard to research with little result. on human services or transportation, for example, As the demand for statistics, evaluation, and may undervalue the available public good. These policy research increase, so too must secure data kinds of barriers limit the effective, efficient, and access with strong privacy protections. The Com- transparent use of existing data. mission identified four overarching challenges to Federal departments are making greater use of realizing the vision of secure access to confidential their own administrative data for statistical activ- data for evidence building that appropriately cal- ities today than in the past, and in some cases, ibrates public benefits, privacy, and transparency.

Reducing the Respondent Burden of Income Questions on Surveys Household survey data collection programs, cal agencies were able to rely more on the in- including key U.S. Census Bureau programs, come data the government already maintains are finding it more difficult to obtain accurate to administer tax, income support, and social income data from the survey population. Re- insurance programs. Recent work at the Cen- spondents have become less willing to partic- sus Bureau has begun to explore the potential ipate in surveys and are increasingly reluctant of administrative sources to replace questions to respond to questions about income.1 When about selected income sources in household they do answer questions about income, they surveys such as the American Community Sur- are providing less accurate responses.2 The vey and the Current Population Survey. While burden on respondents could be reduced and this work is in the early stages and current data the accuracy of the data improved if statisti- access is quite limited, the results are promising and suggest the administrative data can be a significant improvement.3 1. Christopher R. Bollinger, Barry T. Hirsch, Charles M. Hokayem, and James P. Ziliak, “Trouble in the Tails? What We Know about Earnings Nonresponse Thirty Years after Lil- 3. U.S. Census Bureau, “American Community Survey Re- lard, Smith, and Welch,” paper (June 2017); http://www2.gsu. search and Evaluation Report,” Center for Administrative edu/~ecobth/BHHZ_Trouble-in-the-Tails_6-8-2017.pdf (accessed Records Research and Applications Memorandum Series August 10, 2017). 16–72015; Bruce D. Meyer, and Nikolas Mittag, “Using Linked 2. National Research Council, Nonresponse in Social Science Survey and Administrative Data to Better Measure Income: Surveys: A Research Agenda, edited by Roger Tourangeau and Implications for Poverty, Program Effectiveness, and Holes in Thomas J. Plewes, Panel on a Research Agenda for the Future the Safety Net,” National Bureau for Economic Research Work- of Social Science Data Collection, Committee on National Sta- ing Paper 21676 (Washington, D.C.: 2015); C. Adam Bee and tistics, Division of Behavioral and Social Sciences and Educa- Joshua Mitchell, “The Hidden Resources of Women Working tion (Washington, D.C.: The National Academies Press, 2013); Longer: Evidence from Linked Survey-Administrative Data,” Bruce D. Meyer, Wallace K.C. Mok, and James X. Sullivan, in Women Working Longer: Increased Employment at Older Ages, “Household Surveys in Crisis,” Journal of Economic Perspectives edited by Claudia Goldin and Lawrence F. Katz (Chicago: Uni- 29, no. 4 (Fall 2015): 199–226. versity of Chicago Press, 2016). 26 The Promise of Evidence-Based Policymaking

Challenge #1: The current legal framework data readily among themselves. In some cases, can stand in the way of data stewardship even when data sharing is allowable for agency staff, impediments remain to providing external and the sharing of data to support evidence researchers with appropriate access to combined building. data. In other countries with a more centralized statistical system, data sharing is both less needed The current legal environment for data collection, and more straightforward. protection, and sharing lacks consistency, lead- Both program and statistical agencies also must ing to confusion and inefficiency among depart- consider other applicable laws. These include the ments, external researchers, and other members Paperwork Reduction Act (PRA), the Privacy Act, of the evidence-building community. Laws gov- and the Confidential Information Protection and erning the data lifecycle for any dataset include Statistical Efficiency Act (CIPSEA). Generally the the statute that authorized the collection of the PRA does not provide the authority to collect data, data, statutes generally applicable to data col- but it does govern the process by which data collection and management processes, and various lection occurs. The PRA specifically includes re- legal provisions governing privacy and confiden- quirements for transparency with regard to the tiality protections. The Commission recognizes information that government collects and why, by that some variation in the laws that govern the extension implicating potential secondary uses of protection of data is sensible given the contextu- collected data. As the name implies, the PRA also al nature of privacy. It is unclear, however, that attempts to reduce the burden of data collection all of the variation in the legal structure is inten- on the public by encouraging data sharing be- tional; rather, some of the variation may be the tween agencies through the establishment of gov- result of the complexity of government together ernment-wide statistical and information policies with the various statutes having been developed and coordination procedures. independently at particular times and with par- Similarly, the Privacy Act does not provide the ticular goals in mind. The Congress and the Pres- authority to collect data, but it does set require- ident seemingly recognized as much when form- ments regarding how those data may be disclosed. ing the Commission and charging it to review the Specifically, the Privacy Act requires public and net impact of accumulated laws and policies on individual notices about data held in government evidence building.3 systems and limits disclosure of these data with- The authority to collect data typically is provid- out consent. This includes restricting the second- ed in a program or statistical agency’s authorizing ary uses of data without consent unless one of a statute. In the case of program agencies, authoriz- limited number of exceptions applies. For exam- ing statutes may stipulate the nature and opera- ple, the Congress exempted from individual con- tions of the objectives and policies of the agency. sent requirements data provided by another agen- Program statutes vary in their prescriptiveness cy to the U.S. Census Bureau for the purposes of with regard to confidentiality and the use of data planning or carrying out a census or survey.4 for evidence building, with many statutes not ad- CIPSEA was enacted to address some gaps and dressing the issue at all. variations in existing statistical laws. (See the The laws that authorize statistical agencies box, “The Confidential Information Protection generally include restrictions on who can access and Statistical Efficiency Act of 2002.”) While the data and for what purposes. While data collection authorizing statutes for some statistical agencies, was a primary intent of the laws that created these such as the Census Bureau, provide collection agencies, the existing structure has not produced authority together with strong confidentiality the same level of coordination and integration protections, those for some other agencies do that is typical in other countries because the Fed- not specifically address privacy and confidenti- eral Statistical System is decentralized. One result ality. Many statutes also limit data sharing, even is the inability of U.S. statistical agencies to share among Principal Statistical Agencies (PSAs), leading to inefficiencies such as duplicative data

3. Evidence-Based Policymaking Commission Act of 2016 (Public Law 114–140, Section 4(a), March 30, 2016). 4. Privacy Act, 5 USC § 552a(b)(4) (1974). The Promise of Evidence-Based Policymaking 27

Figure 3. Functional Separation of Administrative and Statistical Data

Administrative Data Statistical Data

Information Gathered from One-Way Filter Data Collected, Compiled, and Tax Filings and in Connection Provides Processed for the Purpose of with Government Beneﬁt Privacy and Describing, Estimating, or Programs, Economic Data Format Analyzing Characteristics of Development Programs, and Enhancements Groups, Without Identifying the Other Federal Programs Individuals or Organizations that Comprise Such Groups

Non-Statistical Purpose Statistical Purpose

The Confidential Information Protection and Statistical Efficiency Act of 2002

The Confidential Information Protection and provide a prevailing standard for these con- Statistical Efficiency Act of 2002 (CIPSEA) is a cepts in law. Statistical purpose is defined un- law with two components. The first provides a der CIPSEA as “the description, estimation or set of uniform confidentiality protections for analysis of the characteristics of groups, with- data acquired under a pledge of confidentiality out identifying the individuals or organizations and for exclusively statistical purposes. The sec- that comprise such groups” and “includes the ond provides a limited authority to share pro- development, implementation, or maintenance tected data. Among the motivations for CIPSEA of methods, technical or administrative proce- was a need to ensure a consistent statutory dures, or information resources that support” authority for agencies handling statistical data those purposes. Statistical activities are defined to protect those data from non-statistical use.1 as “the collection, compilation, processing, or The Federal government now has 15 years of analysis of data for the purposes of describing demonstrated success with CIPSEA.2 or making estimates concerning the whole, The CIPSEA framework codifies the concept or relevant groups or components within the of “functional separation” (see Figure 3). Specif- economy, society, or the natural environment” ically, CIPSEA defines statistical activities, sta- and include “the development of methods or tistical purposes, non-statistical purposes, and resources that support those activities.” qualified agents and agencies. Such definitions CIPSEA’s subtitle A fosters public trust by are essential to the application of CIPSEA, and generally prohibiting disclosure of protected information in identifiable form, controlling access to and use of protected information, and 1. U.S. House of Representatives, “Report to Accompany H.R. 2458,” House Report No. 107–787 (2002); https://www.congress. ensuring that information is used exclusively gov/congressional-report/107th-congress/house-report/787 for statistical purposes. While some statistical (accessed August 10, 2017). statutes explicitly provide for data protection, 2. Brian Harris-Kojetin, “CIPSEA at 15 Years: Benefits to others, such as the authorizing statute for the Federal Statistics and Unmet Needs - Overview,” (paper U.S. Bureau of Labor Statistics, do not. CIPSEA presented at the 133rd Meeting of the Committee on National Statistics, Washington, D.C., May 12, 2017); http://sites. Subtitle A provides a remedy for this. nationalacademies.org/dbasse/cnstat/dbasse_178400 (accessed August 10, 2017). —continues 28 The Promise of Evidence-Based Policymaking

collections. CIPSEA was created as a partial an- for exclusively statistical purposes to a person or swer. The law provides common statutory protec- agency not entitled to receive it, including a fel- tions for data acquired for exclusively statistical ony charge and imprisonment for up to 5 years purposes under a pledge of confidentiality and and/or fines up to $250,000. The CIPSEA penalties currently applies to all of the PSAs and to other are equal to, or exceed, other such provisions in recognized statistical units. statutes, including provisions contained in Titles CIPSEA also established common penalties for 13 and 26 of the U.S. Code. any officers, employees, or agents of an agency For the Commission’s purpose, the vital compo- who, knowing that disclosure is prohibited under nents of the CIPSEA legal framework include the CIPSEA, willfully disclose protected data collected ability to offer strong confidentiality protections,

The Confidential Information Protection and Statistical Efficiency Act of 2002—continued

Statistical Agencies or Units Recognized by OMB for CIPSEA Purposes 4

Bureau of Economic Analysis National Center for Science and Engineering Bureau of Justice Statistics Statistics Bureau of Labor Statistics Office of Research, Evaluation, and Statistics, Social Security Administration Bureau of Transportation Statistics Statistics of Income Division, Internal Census Bureau Revenue Service Economic Research Service Center for Behavioral Health Statistics and Energy Information Administration Quality, Substance Abuse and Mental Health Services Administration* National Agricultural Statistics Service Microeconomic Surveys Statistical Unit, National Center for Education Statistics Federal Reserve* National Center for Health Statistics National Animal Health Monitoring System, Animal and Plant Health Inspection Service*

* Recognized statistical units

CIPSEA grants the Director of the U.S. Of- CIPSEA’s subtitle B provides for the limited fice of Management and Budget (OMB) the sharing of business data among three desig- authority to promulgate rules and provide im- nated statistical agencies: the U.S. Bureau of plementing guidance, including the qualifica- Economic Analysis, the Bureau of Labor Statis- tion and recognition of agencies or units that tics, and the U.S. Census Bureau. The authori- may exercise CIPSEA authority. In its CIPSEA ty is designed to reduce paperwork burden on implementation guidance, OMB recognized the businesses, improve comparability of econom- 13 Principal Statistical Agencies (PSAs) as sta- ic statistics, and increase understanding of the tistical agencies under CIPSEA, and also recog- economy. nized three additional statistical units that had demonstrated functional separation in accordance with OMB’s guidance.3

3. “Implementation Guidance for Title V of the E-Govern- ment Act, Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA),”Federal Register 72, no. 115 4. List of Principal Statistical Agencies and Recognized Units; (June 15, 2007): 33361; https://www.gpo.gov/fdsys/granule/FR- https://obamawhitehouse.archives.gov/omb/inforeg_statpolicy/ 2007-06-15/E7-11542/content-detail.html (accessed August 10, bb-principal-statistical-agencies-recognized-units (accessed Au- 2017). gust 10, 2017). The Promise of Evidence-Based Policymaking 29

ensure exclusively statistical uses, and within that words, even the parts of the Federal government structure, enable limited and secure sharing of that today are leaders in producing evidence report data. challenges in navigating the complex array of legal CIPSEA permits the sharing of business data requirements to use data for evidence building. exclusively for statistical purposes among three Perhaps more than any other single data need, PSAs to create statistical efficiency within the the Commission repeatedly heard calls for im- context of CIPSEA’s legal framework.5 This specific proving access to income, wage, and earnings provision enabled several new data sharing initia- data for evidence-building purposes. Effects on tives that have improved efficiency for all three income are central to the evaluation of numer- agencies.6 ous Federal programs and activities, but existing Except for that provided under CIPSEA, the au- laws and practices limit the ability of researchers thority to share data for evidence building is rarely both internal and external to the government to explicit. In cases where authorizing laws are am- access even the income data the Federal govern- biguous, agency interpretations ultimately govern ment already collects (see the box “Income Data: access to and use of data. In some cases multiple Federal Tax and Unemployment Insurance Earn- agencies interpret the same law differently. This ings Data”). Programs such as tax administration can cause confusion and limit the efficient use of depend on compliance from the public, making existing data for evidence building.7 The complex it particularly important to ensure the privacy of web of statutes, regulations, and implementing records generated in the course of administering guidance—or absence thereof—drives risk aver- those programs. It is equally important, however, sion in agencies, causes frustrations for the evi- to calibrate the need for privacy with the public dence-building community, and limits the value good that research findings based on such data of data for statistical activities. In effect, the ex- can provide. Of the 22 offices responding to the isting legal environment limits the government’s CEP Survey of Federal Offices that it was hard to ability to steward data responsibly as a valuable access Federal Tax Information for evidence build- resource for the American people and for policy- ing, half cited statutes prohibiting data sharing as makers. the reason for the difficulty. In the CEP Survey of Federal Offices, 52 percent Another barrier identified in testimony to the of responding offices identified legal limitations as Commission is the explicit prohibition in some a major or moderate barrier to using data for sta- laws on the collection or integration of data, which tistics, evaluation, research, and policy analysis prevent the government from building evidence purposes. While 10 out of 13 PSAs and 8 out of 10 to improve Federal programs and policies. For de- evaluation offices reported that legal limitations cades, different iterations of evidence-prohibiting were a major or moderate barrier, less than half (41 provisions have been included by either the Con- percent) of other types of offices cited legal lim- gress in statutes or the Executive Branch in direc- itations as a major or moderate barrier.8 In other tives and policies. Many of these prohibitions on collecting or accessing data to develop evidence 5. Confidential Information Protection and Statistical Efficiency exist in domains that have major implications for Act of 2002 (CIPSEA)(Public Law 107–347, Title V, 2002). policy areas of national importance and they of- 6. For a summary of several of the successful data sharing initia- ten involve programs that represent a substantial tives that were enabled by CIPSEA Subtitle B, see U.S. Bureau of share of the Federal budget. Labor Statistics (BLS), “Confidentiality of Data Collected by BLS Through the Commission’s fact-finding phase, for Statistical Purposes,” (May 16, 2016); https://www.bls.gov/bls/ cipsea-report.htm (accessed August 10, 2017). stakeholders identified several examples of data collection, integration, and analysis bans. The 7. U.S. Office of Management and Budget (OMB), “Barriers to Using Administrative Data for Evidence Building,” white paper Commission received substantial public com- for the Commission on Evidence-Based Policymaking (Washing- ments regarding the student unit record ban, en- ton, D.C.: OMB, Executive Office of the President, 2016);https:// acted in 2008.9 The student unit record ban effec- obamawhitehouse.archives.gov/sites/default/files/omb/mgmt-gpra/ barriers_to_using_administrative_data_for_evidence_building.pdf (actively limits some aspects of evidence production cessed August 10, 2017). related to pre-kindergarten through 12th grade and 8. Based on 79 offices that reported they collect or use data for statistics, evaluation, research, or policy analysis or spend a portion of their budget for such purposes. 9. Database of Student Information Prohibited, 20 USC § 1015c. 30 The Promise of Evidence-Based Policymaking

postsecondary education by prohibiting the Fed- ing the outcomes of government programs eral government’s development or maintenance and policies are collected at the state or lo- of a new database with data on all students. The cal level and are inaccessible for evidence Commission heard many substantive comments about the student unit record ban, and received building. more feedback on the issue than on any other single topic within the Commission’s scope. Nearly Many Federal programs are administered by states two-thirds of the comments received in response and localities, most often in cooperation with to the Commission’s Request for Comments raised the Federal government. Examples of programs concerns about student records, with the majority that operate under this model include Medicaid, of those comments in opposition to overturning the Supplemental Nutrition Assistance Program the student unit record ban or otherwise enabling (SNAP), Unemployment Insurance (UI) programs, the Federal government to compile records about Housing Assistance programs, and the Temporary individual students. Assistance for Needy Families (TANF) program. Another ban brought to the attention of the These programs have a broad variety of funding Commission is embedded in the Workforce Inno- and administrative structures that include the vation and Opportunities Act (WIOA), a program Federal and state government as partners and of- for workforce training and development.10 The ban ten include cities, counties, and grantees as part- is a prohibition on developing a national database ners as well.12 The administrative data generated of participants in WIOA training programs that by the operation of these programs are a valuable includes personally identifiable information.11 source of data for use in evidence building, but With these bans, and a similar ban covering data there are numerous, well-documented barriers to on Head Start participants, much of government’s accessing and using these data.13 investment in human capital programs—from early Barriers to accessing state-level data include: childhood through adulthood—largely is not substate laws or legal interpretations of Federal law ject to the type of rigorous, national-scale evalua- that either expressly prohibit or tightly restrict tion that leveraging administrative data could make data sharing for the purposes of evidence build- possible. Other examples of bans or restrictions on ing; lack of capacity within states to transform use of government data to conduct research popu- the administrative data to make them suitable late the U.S. Code, although the Commission was for evidence building; and administrative and/ not approached in public comments or by expert or procedural variations for accessing data across witnesses to address these other bans. Prohibitions on data collection and use argu- ably conflict with the Commission’s vision to im- 12. For example, the Federal government funds the full cost of prove government based on credible evidence. SNAP benefits and reimburses the state for approximately 50 Within the broad array of factors that influence percent of its administrative costs. Medicaid is jointly funded by the Federal and state government, where the Federal government policymaking, a ban may be an appropriate tool pays states for a specified percentage of program expenditures for setting priorities or achieving certain other calculated as the Federal Medical Assistance Percentage. TANF goals. In the context of evidence building, how- operates as a block grant to states, and the program may be state or county administered, depending on the structure of the state. ever, a ban on data collection or use cannot easily be reconciled with a goal of increased reliance on 13. OMB, “Barriers to Using Administrative Data for Evi- dence-Building;” Office of Planning, Research and Evaluation evidence to inform policymaking. (OPRE), Administration for Children and Families, U.S. Depart- ment of Health and Human Services, “Using Administrative Data in Social Policy Research,” OPRE Report 2016–62 (Washington, Challenge #2: Many high-value administra- D.C.: OPRE, 2016); Northwestern University/University of Chi- tive data associated with Federally funded cago Joint Center for Poverty Research, “Administrative Data for Policy-Relevant Research: Assessment of Current Utility and Rec- programs that could be useful to measur- ommendations for Development,” V. Joseph Hotz, Robert Goerge, Julie Balzekas, and Francis Margolin, editors; Helen Lee, Anne Warren, and Lakhpreet Gill, "Cheaper, Faster, Better: Are State Administrative Data the Answer? The Mother and Infant Home 10. Workforce Data Quality Campaign, submission to the Com- Visiting Program Evaluation-Strong Start Second Annual Report," mission's Request for Comments. OPRE Report 2015-09 (Washington, D.C.: OPRE, 2015); and Kel- 11. Prohibition on development of national database, 29 USC § ly Maxwell, “Issues in Accessing and Using Administrative Data,” 3341(b). OPRE Report 2017–24 (Washington, D.C.: OPRE, 2017). The Promise of Evidence-Based Policymaking 31

Figure 4. Examples of State-Collected Administrative Data Productive for Evidence Building

Sharing with the Federal Government: • Adoption and Foster Care • Adoption and Foster Care Mandatory Analysis and Reporting System Analysis and Reporting System • Medicaid Data Coverage: • Children’s Health Insurance • National Directory of New Hires Universe Program • Medicaid • National Child Abuse and Neglect Data System Sharing with the • Supplemental Nutrition • National Directory of Assistance Program Sources of Federal Government: New Hires • Temporary Assistance for State-Collected Mandatory • Supplemental Needy Families Nutrition Assistance Administrative Data Coverage: • Unemployment Insurance Program Data Sample • Temporary Assistance Beneﬁts for Needy Families • Unemployment Insurance Beneﬁts Sharing with the • Children’s Health Insurance Program • Unemployment Insurance Federal Government: • National Child Abuse and Neglect Quarterly Wages Voluntary Data System • Vital Statistics Cooperative Program • Unemployment Insurance Quarterly Data Coverage: Wages Varied • Vital Statistics Cooperative Program

For a list of additional datasets productive for evidence building, see Appendix D.

different state agencies or programs, leading to SNAP data submitted to the USDA may only be protracted and often duplicative administrative used for administrative or enforcement purposes, requirements. These barriers are compounded for not for evidence building.14 projects in which multiple administrative datasets Figure 4 describes the current landscape of are to be combined, as each dataset has different data availability at the Federal level for a set of access requirements. Similar barriers often exist high-priority state-collected administrative data- to access data from local or other jurisdictions. sets. The figure illustrates where the data sources The Federal government invests significant diverge based on three important distinctions: (1) funds to support the operation of many jointly ad- whether the provision of data from the state to ministered Federal-state programs, providing a ba- the Federal government is mandatory or volun- sis for some type of Federal reporting requirement. tary, (2) whether and to what extent the data be- Currently, the volume and type of state-collected ing shared represent the universe or just a sample, administrative data that are reported to Federal and (3) whether the data contain sufficient per- program agencies vary considerably across pro- sonal identifiers to enable integration with other grams. Each program operates under its own set of data sources. statutory and regulatory requirements for submit- Increasing access to administrative data doc- ting program data to their Federal sponsor, rang- umenting individual earnings is of particular im- ing from the Medicaid program—for which states portance, given the many Federal programs and are required to submit data on all beneficiaries to policies that seek to have an impact on earned the Centers for Medicare and Medicaid Services income. Accessing quarterly earnings data di- (CMS)—to SNAP and TANF—under which states rectly from all 50 states, the District of Columbia, are required to submit only a sample of data to a and the territories can be challenging and time Federal agency. Additionally, some of the data submitted to the Federal government have statutory 14. See use restrictions in the Food, Conservation, and Energy Act use restrictions; for example, the individual-level of 2008 (Public Law 110–246). 32 The Promise of Evidence-Based Policymaking

Income Data: Federal Tax and Unemployment Insurance Earnings Data Wage and income data exist in at least two Researcher access points for FTI are the U.S. forms, data from the Unemployment Insur- Treasury Department and the Census Bureau. ance (UI) programs and Federal Tax Informa- Statutory restrictions on accessing data are the tion (FTI). These data sources can be viewed as major barrier to the use of FTI. The Federal tax complementary, covering somewhat different code (Title 26 of the U.S. Code) designates FTI as populations and different types of income with confidential—disclosure to any party is prohib- differing periodicity. Specifically, the informa- ited, except under explicit statutory exceptions. tion from the UI programs consists of individual FTI can be used by Treasury employees and a wage records and is available quarterly, whereas limited set of approved researchers engaging in FTI offers a broader view of income earned from statistical projects in support of tax administra- sources other than wages such as transfers, per- tion. The Census Bureau and the U.S. Bureau of mits the analysis of household as well as indi- Economic Analysis also are authorized limited vidual income, and is available annually. access under Title 26 and associated regulations Federal law requires states to collect quarterly for a narrow set of uses related to improving wage data as part of their administration of state their statistical products. The Title 26 limita- UI programs operated as part of the Federal-state tions then transfer to any data products these UI system.1 The U.S. Department of Labor over- agencies produce with comingled FTI and access sees and provides funds for the UI system, but to comingled microdata also requires Internal each state administers its own UI program, which Revenue Service (IRS) approval. includes the collection of data on individual em- The Treasury Department is the primary ac- ployees and their earnings in each quarter. cess point for FTI. There are several ways that Researchers wishing to access UI data current- Treasury provides access to tax data for research. ly have three primary access points: originating One way is through the IRS Statistics of Income state systems, the National Directory of New Hires Division’s Joint Statistical Research Program. An maintained by the U.S. Department of Health and important characteristic of the Statistics of In- Human Services’ Administration for Children and come program is that it aims to limit, as much Families, and the U.S. Census Bureau’s Longitudi- as possible, non-IRS researchers’ direct access to nal Employer-Household Dynamics Program. Yet, directly identifiable taxpayer data by assigning there are significant challenges to accessing and Statistics of Income employees as co-research- using each of these sources of UI data. ers in all phases of the work, including assem- Accessing quarterly earnings data direct- bling and cleaning the required data, performing ly from multiple states can be challenging and analyses, and writing reports on the findings. time consuming, if not impossible. Both the Na- For this reason, the program has very limited tional Directory of New Hires and the Longitu- capacity. For example, in 2014, only 12 projects dinal Employer-Household Dynamics Program were selected from among 80 proposed. Numer- have existing authority to allow researchers ac- ous researchers have gained access to tax data cess to the wage record data for some statistical through alternative means, including through activities, but the authority is narrow, making the Offices of Tax Analysis and Tax Policy within secure access to the data excessively restrictive.2 the Treasury Department, which also engage in independent and occasionally joint research.

1. The Federal Unemployment Tax Act (26 USC § 3301 et seq.) and titles III, IX, and XII of the Social Security Act (42 USC enced by participants in Federally funded state employment and chapter 7) form the framework of the system. training programs. Either a state or the Employment and Train- 2. The Wage Record Interchange System (WRIS), maintained ing Administration at DOL may propose research projects using by the Department of Labor (DOL), facilitates the exchange of the WRIS, but each proposed project must demonstrate a direct UI wage records between states for performance accountability benefit to a qualifying program or activity and approval must be purposes, enabling improved reporting on the outcomes experi- obtained individually from each state whose data will be used. The Promise of Evidence-Based Policymaking 33

Data About Births and Deaths in the Population

Data on vital events (births and deaths) are of In 2007, NCHS released a new policy on the great value for statistics, evaluation, and pol- release of and access to vital statistics microda- icy research. For studies of health, housing, ta to comply with state laws and policies.1 The and other policies, there is perhaps no more revised policy reflects the dual goals to make important outcome variable than death. Vital data available as widely as possible while being records also are critical for public health pro- responsive to concerns about confidentiality. grams and serve a variety of administrative The current agreement with the states on purposes. the re-release of restricted data containing geo- The U.S. National Center for Health Sta- graphic detail requires a review of all such data tistics (NCHS) is the Federal agency mandat- requests by the National Association for Pub- ed to produce national health statistics based lic Health Statistics and Information Systems on a cooperative, decentralized system. Data (NAPHSIS), which represents state vital reg- from more than six million vital-event records istrars.2 The review by NAPHSIS is conducted are collected each year by vital registrars in prior to the NCHS review, and applies to both all states and U.S. territories and transmit- Federal and non-Federal requests for restrict- ted to NCHS for processing and dissemina- ed data files. Researchers in Federal agencies, tion. NCHS has a statutory mandate (42 USC as well as their on-site or off-site contractors, § 242k) to collect data annually from the re- also can submit project proposals that request cords of births, deaths, marriages, and divorces exact dates of vital events. If needed, the file in registration areas. Currently the data col- with exact dates can include geographic detail. lection is limited to data from birth and death Non-Federal researchers (including Federal records (including fetal deaths), as NCHS dis- grantees) can gain controlled access to exact continued the collection of individual-record dates of vital event files only through the NCHS marriage and divorce reports after 1995. Research Data Center, with approval. Data are collected through the Vital Statis- tics Cooperative Program. These data are provided through contracts between NCHS and vital registration systems operated in the 57 1. NCHS defines microdata as all raw data, including public jurisdictions legally responsible for the regis- use and restricted data files. tration of vital events, namely states. The con- 2. NCHS, National Vital Statistics System, “NCHS Data Release and Access Policy for Microdata and Compressed Vital Statis- tracts support the cost of training and techni- tics Files;” https://www.cdc.gov/nchs/nvss/dvs_data_release.htm cal assistance to help standardize data quality. (accessed August 10, 2017).

consuming, and in many cases, not possible. Many have enabled states to improve their capacity for residents of one state work in a different state; evidence building. One example of this kind of in- thus, evidence-building efforts frequently neces- vestment has been the grants administered by the sitate accessing quarterly earnings data from mul- National Center for Education Statistics to support tiple states. While there are currently two possible the development of Statewide Longitudinal Data avenues by which researchers can access data on Systems to assess K–12 education outcomes and quarterly earnings from multiple states through investments, creating data systems at the state a single access point (see the box “Income Data: level similar to those prohibited by the student Federal Tax and Unemployment Insurance Earn- unit record ban at the Federal level. Such invest- ings Data”), both avenues are relatively narrow ments also have been valuable in enabling states to and include numerous restrictions. comply with reporting requirements. For example, Federal resources and technical assistance to while states are required to submit extensive Med- states for modernization of information systems icaid data to CMS, the Federal government spends 34 The Promise of Evidence-Based Policymaking

Longitudinal Employer-Household Dynamics Program

The Longitudinal Employer-Household Dynam- state includes an option for the state to stream- ics Program (LEHD) is an example of a part- line the review and approval process by allow- nership that incorporates voluntarily provided ing the Census Bureau to make the determina- state-held administrative data with survey and tion as to whether a proposed research project other administrative data held by the U.S. Cen- is consistent with the uses authorized by the sus Bureau to support statistical analysis and state, but currently only 12 states have selected research. However, the current structure of the this option.1 All of the other states review and program limits broader use of the underlying approve each project individually, meaning that state data for evidence building beyond the nar- data access is cumbersome and time-consum- row purposes of LEHD. ing, if it can be accomplished at all. Under the Local Employment Dynamics Some of the individuals who commented to Partnership, which is a voluntary Federal-state the Commission noted that UI wage records partnership started in 1999, states agree to remain underutilized because of limitations to share historical and ongoing administrative re- access. Commenters expressed support for in- cords of Unemployment Insurance (UI) earn- creasing access to the LEHD data. The data use ings data and the Quarterly Census of Employ- agreements with each state typically need to be ment and Wages data with the Census Bureau. renewed every five years. While at present, 48 In return, the Census Bureau produces a lon- states, the District of Columbia, and two U.S. gitudinal data infrastructure from which new territories have agreements in place with the statistics about the dynamics of local employ- Census Bureau, their continued participation ment and the locations of jobs and workers can is not guaranteed. The UI program is governed be produced. and partially funded by the Department of La- Researchers seeking access to LEHD data, in- bor, but it does not have direct access to the cluding the UI wage record microdata, can ap- microdata and is only able to access LEHD mi- ply to access the data through a Federal Statisti- crodata for projects supporting the Census Bu- cal Research Data Center. The researcher must reau mission. comply with the requirements for conducting research at a research data center. Among other 1. If the research project will include the release of state or things, research projects using the LEHD must substate-level specific results (as opposed to results from a advance the mission of the Census Bureau, a group of states), the state will have the opportunity to review and approve the external research project. If the intent of the requirement that precludes many types of valu- external researcher is to release pooled results only, states can able research. It also must be consistent with choose to waive their opportunity to review each proposed applicable state law. The data use agreement project. If this option is not selected, even external projects that will release pooled results only would need to be re- that the Census Bureau enters into with each viewed and approved by the relevant states.

$2–3 billion per year to support more efficient state tive Program is based on the relationship between information technology systems for tracking Med- the U.S. National Center for Health Statistics and icaid eligibility, enrollment, and claims.15 the 57 jurisdictions that collect vital records cov- Some Federal agencies also have developed ering births and deaths in the United States (see productive partnerships with states and other the box “Data about Births and Deaths in the Pop- jurisdictions to establish mutually advantageous ulation”). In exchange for collecting data in a uni- systems for making data accessible for evidence form manner, states receive funding, training, and building. For example, the Vital Statistics Coopera- technical assistance. Other examples of Federal-state partnerships are not necessarily financial in nature. For example, 15. CEP staff correspondence with Department of Health and Human Services, Assistant Secretary for Planning and Evaluation the Census Bureau offers data analysis services to staff in March 2017. state programs willing to make their data accessible The Promise of Evidence-Based Policymaking 35

for Federal evidence building. One witness during needed for comparability.17 Second, some states a public Commission meeting—Erin Ulric from provide data on the full set of program partici- Colorado’s Women, Infant, and Children (WIC) pants or beneficiaries, while others provide only a Program—described how Colorado benefits from representative sample. To be integrated with other receiving statistics generated by the Census Bureau data and used for evidence building, administra- to monitor program performance and identify tive datasets are more useful when they include populations for targeted outreach. In this sense, all participants or beneficiaries. Supplying the secure access to the Colorado WIC data by eligible universe file also can reduce the burden on states, researchers enables the state to improve the oper- as implementing robust sampling procedures re- ation of its program by receiving relevant aggre- quires technical expertise. Furthermore, when the gate statistics that inform day-to-day operations. datasets include personal identifiers so program The Longitudinal Employer-Household Dynamics participation can be statistically associated with Program (LEHD) also operates as a non-financial outcomes of interest, such as college graduation partnership (see the box “Longitudinal Employ- or earnings, members of the evidence-building er-Household Dynamics Program”). community are better able to produce valid and If the combination of laws that apply to Fed- reliable estimates about program and policy out- eral programs and policies can create confusion, comes. the same can be especially true for state-held data. When individual jurisdictions interpret Federal Challenge #3: Cumbersome and idiosyn- laws, inconsistencies can arise in whether they cratic data access procedures create confu- permit the use of data on the very same program. Several Federal agencies have taken actions to ad- sion, impose unnecessary costs, and are a dress inconsistent interpretations across states. barrier to evidence building, without always For example, to encourage greater use of data from providing significant privacy benefits. TANF and SNAP, the U.S. Department of Health and Human Services’ Administration for Children The Commission heard repeatedly about the dif- and Families and the U.S. Department of Agricul- ficulties that cumbersome and onerous proce- ture’s Food and Nutrition Service issued memo- dures, often the result of idiosyncratic processes randa to state program directors clarifying what that vary across government, cause for members data uses are permissible under their respective of the evidence-building community seeking to program laws.16 securely access confidential data. For researchers Beyond accessibility, members of the evi- and evaluators external to government, no stan- dence-building community noted two additional dard for applying for data access currently exists challenges related to using state-collected ad- across government agencies, making it necessary ministrative data for evidence-building activities. to navigate different and varied processes for each First, Federal programs may not require states or agency. local jurisdictions to collect the same kind of data The CEP Survey of Federal Offices found that in the same way. As a result, while data within a all 13 PSAs, 5 out of 8 evaluation offices, and 29 single state, program, or jurisdiction may be ana- percent of other offices (14 out of 49) provide lyzed with relative ease, when data from multiple external researchers access to the data they col- sites or states are brought together for analysis, lect.18 Some agencies have formal programs for substantial cleaning and consistency edits are external researcher access, with well-established procedures for applying for access. Under these programs, researchers apply for access to data and the agency “qualifies” eligible researchers and approves their projects based on agency-defined 16. Susan Golonka, “Re: U.S. Census Bureau Request for TANF Data,” (memorandum, Washington, D.C.: U.S. Department of Health and Human Services, Administration for Children and 17. Harry Hatry, “Using Agency Records,” in Handbook of Practical Families, December 7, 2016); Thomas Louis and Richard Lucas, Program Evaluation, edited by Kathryn E. Newcomer, Harry Hatry, “Memorandum of Record Regarding the Sharing of State SNAP and Joe Wholey, 325–343 (Hoboken, N.J.: Wiley, 2015). and WIC Recipient Data With the U.S. Census Bureau,” (memorandum, Washington, D.C.: U.S. Department of Agriculture and 18. Two of the 10 responding evaluation offices and 18 of the 67 U.S. Department of Commerce, August 12, 2014). responding other offices reported they do not collect data. 36 The Promise of Evidence-Based Policymaking

processes. Commenters said these processes can Act in 2017, which found that “administrative often be slow and confusing. Other agencies do costs faced by researchers may be reducing the re- not have formal programs for external research- turn on investment of Federally funded research er access, so the application process can be much and development” and that “it is a matter of crit- more ad hoc. ical importance to United States competitiveness that administrative costs of Federally funded re- “Key issues encountered include: search be streamlined so that a higher portion of Federal funding is applied to direct research.”19 inconsistent, but often lengthy Some of the idiosyncrasies of data access pro- and cumbersome processes for cedures result from a lack of capacity within agen- accessing data. Each agency cies to devote to this work. The Commission found negotiates its own data sharing that agencies vary significantly in their capacity to provide external researcher access to data. A lack agreements, which can add up of sufficient capacity restrains agencies’ ability to to significant time and cost for undertake key administrative functions necessary national studies that require data for providing access to confidential data, including the development and dissemination of metadata from multiple agencies in many and even the government’s ability to explain pro- states, particularly when there cesses and procedures to eligible researchers. A may be variations in exactly what lack of capacity can also result in delays through- the process requires and how long out a research project, from approving a researcher’s access request to reviewing the researcher’s it takes.” output before it can be released. – Andrew Weigan, President and Chief The Federal Statistical Research Data Centers Executive Officer at Social Policy Research (FSRDCs) are an example of a successful partner- Associates, Commission Public Hearing, San ship to expand the Federal government’s capacity Francisco, February 9, 2017 to facilitate external researcher access to data for statistical purposes (see the box “Federal Statisti- These challenges are compounded when a re- cal Research Data Centers”). The Census Bureau searcher seeks to access multiple agencies’ data. and other Federal agencies have collaborated with In this situation, the researcher must apply to and host institutions to enable approved researchers be approved for access by all of the agencies. The to access confidential data from multiple agencies result can create scenarios in which one agency through a network of secure data enclaves. While provides an approval and another does not, neces- effective for facilitating data access, the FSRDCs sitating ongoing negotiations between the agen- are still constrained by a lack of standard access cies and the researcher. procedures across the agencies making their data Not only does this result in an inefficient pro- available through an FSRDC. In practice, this cess, these inefficiencies might not enhance con- means that a researcher seeking data other than fidentiality or privacy because the standards for or in addition to Census Bureau data must still access and use are not based on a common un- apply to the originating agency for access, which derstanding of risk for linked data. Nor do these can often result in delays and administrative hur- access procedures result in standard information dles necessitating months or years of effort to gain about use and practice to provide transparency access. Progress is being made on this front, with and an ability to be audited. work on new data access agreements that will Moreover, inefficiencies in data access process- enable the FSRDC staff to approve projects on be- es for evidence building create administrative ex- half of an originating agency. However, progress penses and researcher burdens that can impede is only available to the extent applicable statutes Federally funded research. The costs associated provide agencies such discretion, as discussed ear- with excessive administrative burdens were ac- lier in this chapter. knowledged by the Congress and the President in a somewhat different context with the enactment 19. American Innovation and Competitiveness Act (Public Law of the American Innovation and Competitiveness 114–329, Section 201, January 6, 2017). The Promise of Evidence-Based Policymaking 37

Federal Statistical Research Data Centers

The U.S. Census Bureau, on behalf of the Fed- to secure access to data, geographic proximi- eral statistical agencies, operates a network of ty for researchers is key. The FSRDC network 27 Federal Statistical Research Data Centers has grown organically with demand. The first (FSRDCs) in 18 states and the District of Co- remote FSRDC opened in Boston in 1994. The lumbia to provide secure access to a range of most recent FSRDCs opened at the University Federal restricted-use microdata for statistical of Kentucky and the University of Colorado purposes. Boulder in August 2017. With these additions, The FSRDCs are partnerships between Fed- the FSRDC network will have doubled in size eral statistical agencies and research institu- since 2013. As of 2016, the FSRDC network tions across the country. Because the current supported about 900 researchers working on structure requires physical presence in order about 250 different projects.

Federal Statistical Research Data Center (FSRDC) Locations

Northwest

New York - Minnesota Cornell Boston Wisconsin Michigan Yale New York - Baruch Chicago Central Plains Penn State Philadelphia California - Berkeley Maryland California - Stanford Georgetown Rocky Missouri Univ. of U.S. Census Bureau Mountain Illinois Kansas Kentucky Urbana- Triangle - Duke City California - UCLA Champaign California - USC Triangle - RTI California - Irvine Atlanta

UT - Austin Texas Legend Current FSRDC Planned FSRDC

Once access has been approved, researchers en- fact-finding phase, the Commission heard con- counter other structural issues with FSRDCs such cerns that the FSRDC network, as it exists cur- as the requirement to be physically present at an rently, would be overwhelmed by the additional FSRDC site, the limited number of seats avail- demands of supporting an increase in data access able at a given FSRDC, the limitations of current for a higher volume of evidence building. computing resources available, and the wait time Members of the evidence-building community to obtain disclosed results.20 Further, through its inside government face challenges similar to those confronted by external researchers. The non-stan- 20. According to the Census Bureau, the average wait time in 2017 dardized and lengthy processes by which access is approximately three weeks for obtaining disclosed results. is negotiated and memorialized via an agreement 38 The Promise of Evidence-Based Policymaking

were among the most common data access barriers close to 30 percent (23 out of 79) of responding of- cited by stakeholders. Formal data access agree- fices indicated that lack of staff, policies, and proce- ments (e.g., Memoranda of Understanding or dures to establish data sharing agreements in their MOUs) between two or more agencies can take own office made it hard for them to get access to years to develop. Delays associated with negotiat- the specific dataset they needed.24 ing MOUs are compounded by the challenges described earlier in this chapter when legal authori- Challenge #4: Many Federal departments ties are inconsistent. A lack of clear legal authority do not sufficiently or regularly assess their can result in extended reviews and negotiations by lawyers within multiple offices or departments information resources to determine access prior to granting access. An agency’s ability to ef- controls and privacy protections appropri- ficiently execute MOUs can also be impacted by ate to the sensitivity of the data. capacity constraints. As described earlier in this chapter, a variety of laws “I think ideally you would have govern access to government data for evidence building. Federal agencies follow these laws, asso- a much more streamlined [data ciated regulations, and agency legal interpretations access] procedure that was when making decisions about data security proto- really oriented around statistical cols and access restrictions for the data they hold. research. The Commission finds that many PSAs imple- ” ment tiered access programs that set data access – Raj Chetty, Professor of Economics at and security requirements based on an assessment Stanford University, Commission Meeting, of dataset sensitivity. Tiered access is an applica- Washington, D.C., July 22, 2016 tion of data minimization, a key privacy safeguard for evidence building as embodied in the Fair In- In 2014, OMB developed an optional model formation Practice Principles (described in Chap- agreement for agencies to consider, though uptake ter 3). Data minimization means giving access has been limited to date.21 Some states also have to the least amount of data needed to complete pursued similar agreements for use among internal an approved project. For example, an eligible re- state agencies. Michael Basil, General Counsel in searcher’s project might earn approval for access the Illinois Department of Innovation and Technol- to confidential information at a highly secure re- ogy, described the state’s effort to develop an En- search data center that requires expert review of terprise Memorandum of Understanding (eMOU) all output. Another researcher’s project may need in an effort to improve the use of data at the state only access to a data query tool that runs an anal- level.22 In 2016, an expert panel sponsored by Ac- ysis, checks for disclosure risk without ever show- tionable Intelligence for Social Policy developed ing individual records, and provides group statis- model templates for Memoranda of Understanding tics (see the box “Data Query Tools”). If researchers to help states develop integrated data systems.23 All find that a data tool meets their needs, they may of these efforts suggest opportunities for develop- never need to access the underlying confidential ing a common MOU template for use in the Federal data. Similarly, researchers, journalists, and other government. In the CEP Survey of Federal Offices, members of the public may be able to answer their questions by using data that have been masked, 21. Office of Management and Budget (OMB), “Guidance for Pro- de-identified, and approved for release as a pub- viding and Using Administrative Data for Statistical Purposes” lic use file or by reference to published statistics. (memorandum M–14–06, Washington, D.C.: OMB, Executive Of- A well-designed and properly implemented data fice of the President, February 14, 2014). minimization strategy like tiered access can reduce 22. Michael Basil, Illinois Department of Innovation and Technol- the risk of unauthorized use and unintended harm ogy, Commission Meeting, Washington, D.C., September 9, 2016. to individuals. 23. John Petrila, Barbara Choh, Wendell Pritchett, Paul Stiles, Victoria Stodden, Jeffrey Vagle, Mark Humowiecki, and Natassia Rozario, "Legal Issues for IDS Use: Finding a Way Forward," (Phil- adelphia: Actionable Intelligence for Social Policy, 2017); https:// 24. Based on 79 offices that reported they collect or use data for www.aisp.upenn.edu/wp-content/uploads/2016/07/Legal-Issues.pdf statistics, evaluation, research, or policy analysis or spend a por- (accessed August 10, 2017). tion of their budget for such purposes. The Promise of Evidence-Based Policymaking 39

Data Query Tools Online query tools allow researchers to find conduct research on restricted access data answers to their questions without ever ac- files without needing access to the underlying cessing individual records. Researchers submit data. The simplest query tools typically include requests for analysis through the tool. The que- a table generator that allows the researcher to ry tool conducts the analysis behind a secure select variables for rows and columns; the que- server wall and returns only the result to the ry tools generate the results. More advanced researcher. This approach can be applied to query tools allow researchers to submit code support access to data in a way that limits dis- for statistical analyses on the restricted use closure risks. data. Query tools using underlying confidential The National Center for Education Statis- data can automatically evaluate output for dis- tics uses such technology in its DataLab ap- closure risk before returning the output to the plication. The application allows analysts to researcher.

The National Center for Health Statistics uses a holders, including researchers and privacy advo- variation of tiered access based on data sensitivity. cates, to establish access restrictions based on law, The agency produces a public use version of the context, and sensitivity. Such a review would en- National Health and Nutrition Examination Survey sure that departments give appropriate access to that has been masked to protect individual iden- Federal data needed for evidence building while tities, but requires researchers needing access to protecting sensitive personal information. human genome data to train and qualify as “sworn Even with a tiered access approach, the Fed- agents” of the National Center for Health Statistics eral government currently lacks a consistent and before accessing the data in a secure facility. Today, objective system for classifying statistical and ad- Federal agencies can take advantage of new tech- ministrative data based on sensitivity. The absence nologies that allow versions of confidential data- of such a classification system limits government’s sets that are safe for wide public use. Some of those ability to describe access restrictions and data se- technologies are described further in Chapter 3. curity and privacy protocols appropriate for each To determine appropriate access, Federal de- level of sensitivity, in a way similar to the well- partments should consider the sensitivity of the known classification and handling schedules for data in context. A recent law review article on national security information. sensitive information indicated that “the conclu- Harvard University has developed a “data tags” sion that a particular type of information should system to code the sensitivity of various datasets be treated as sensitive gives rise to special rules of that illustrates the potential of such a system. collection, use, and disclosure as a means to pre- The assigned data tags govern the encryption vent security or privacy harm.” 25 Existing guide- protocols used for data storage and transmission lines do not require data access decisions to be as well as the requirements for data access.26 The based on a formal analysis of the sensitivity of national archive of Denmark is considering using the data. The Commission did find, however, that a similar system to assess the sensitivity of its most agencies conduct informal analyses of the datasets in preparation for the implementation in sensitivity of data, and several agencies, including 2018 of the European Union’s General Data Pro- all PSAs, conduct formal analyses as part of their tection Regulation.27 standard disclosure review processes. While staff in Federal departments include experts with the ability to assess the sensitivity of the data, they can benefit from input from stake- 26. Latanya Sweeney, Mercè Crosas, Michael Bar-Sinai, “Sharing Sensitive Data with Confidence: The Datatags System,”Technology Science (October 16, 2015); https://techscience.org/a/2015101601 (accessed August 10, 2017). 25. Paul Ohm, “Sensitive Information,” Southern California Law Review 88 (2015): 7; https://papers.ssrn.com/sol3/papers.cfm?abstract_ 27. “EU General Data Protection Regulation (GDPR).” EU GDPR id=2501002 (accessed August 10, 2017). Portal; http://www.eugdpr.org/ (accessed August 10, 2017). 40 The Promise of Evidence-Based Policymaking

Figure 5. Model of Sensitivity when taken together with the privacy enhance- Levels for Federal Data ments proposed in Chapter 3, the following recommendations will safely harness the potential Sensi- of data to produce more and better evidence in Level Description tivity the future. 5 Crimson Maximally restricted. Highly sensitive. Identifiable REC. 2-1: The Congress and the Presi- records from data dent should enact legislation estab- collected with a promise of confidentiality. lishing the National Secure Data Service (NSDS) to facilitate data access for evi- 4 Red Restricted. Sensitive. Identifiable records from dence building while ensuring transparen- data collected with a cy and privacy. The NSDS should model promise of confidentiality. best practices for secure record linkage 3 Yellow Restricted. Crimson or and drive the implementation of innova- Red datasets modified by tive privacy-enhancing technologies. technologies that mask individual records (e.g. data query tools, differential Addressing the Federal government’s capacity for privacy). evidence building should begin by enabling con- 2 Green Minimally restricted. sistent, transparent, and accountable secure ac- Not sensitive. Data files cess to data. Recommendation 2-1 will establish made available to the NSDS to help address the existing gaps in the ca- public but subject to pacity to meet this need on a consistent, govern- procedures designed to ment-wide basis. Because the evidence-building raise accountability by users, community is highly decentralized, gaps exist in such as registration before the consistent application of these functions and accessing. a service with the primary mission to address 1 Blue Public data. Most safe. these gaps is needed (see Figure 6). Specifically, Open data. a service charged with facilitating access and ensuring protection of data for evidence building is a direct approach to ensure evidence is more rou- Based on the Harvard model, Figure 5 provides tinely generated using privacy best practices. an example of sensitivity levels that could be ap- The NSDS should support secure access to con- plied to Federal data. fidential datasets in a setting that meets stringent protection standards. The NSDS will apply Recommendations strict data minimization techniques to ensure researchers accessing combined data will use data- The Commission concludes that a priority for ev- sets with as much information removed as is pos- idence building now and in the future is to equip sible while still meeting the research need. When the evidence-building community with secure, re- two or more datasets will be combined, only a stricted access to data to facilitate the generation of narrow group of qualified and trained employees high quality evidence about government programs will have access to direct identifiers to conduct and policies. Recognizing the benefits of maximiz- the linkage. ing privacy protections and the utility of existing The NSDS also should be responsible for devel- data, the Commission identified a series of rec- oping and implementing state-of-the-art meth- ommendations to better enable secure, restricted ods to safely combine confidential administrative access to increased amounts of the data already and survey data from two or more Federal depart- collected by government. Many of these data are ments for evidence building. These combined data highly relevant for informing and evaluating an will provide new opportunities for evidence build- array of Federal programs and policies. Based ing. The NSDS’s existence will highlight how im- on the evidence examined by the Commission, portant it is for the Federal government to use the The Promise of Evidence-Based Policymaking 41

data it already collects with an unparalleled level Figure 6. Key Functions of transparency about who is using which data for of the National Secure what statistical purposes. Data Service (NSDS) The NSDS should help develop and apply cutting-edge technologies to create highly privacy-protective versions of combined datasets. A Privacy and well-implemented tiered-access system will limit Transparency the need for access to combined datasets containing identifiable records. Like the existing PSAs, the NSDS must have an exclusively statistical mission. Distinct from PSAs, NSDS which typically have a specific topical focus, the National Secure Data NSDS will focus more functionally on data access Data Service Access Secure and confidentiality for use in evidence building, and Linkage especially across topical areas. The NSDS should Analysis be designated by the Congress and the President as a PSA under CIPSEA to afford the entity protections already in law and embodied in other recommendations throughout this report. The statute creating the NSDS should expressly prohibit its use for non-statistical activities. REC. 2-2: The NSDS should be a ser- The NSDS should be located in the U.S. Depart- ment of Commerce, building upon the Depart- vice, not a data clearinghouse or ment’s extensive portfolio of statistical and warehouse. The NSDS should facilitate data-related bureaus and expertise existing within temporary data linkages in support of dis- the Census Bureau, the U.S. Bureau of Economic tinct authorized projects. Analysis, the U.S. National Institute of Standards and Technology, and the U.S. National Oceanic The Evidence-Based Policymaking Commission Act and Atmospheric Administration, among others. charged the Commission with considering whether However, it should be situated in such a way as the country needed a data “clearinghouse.”28 Con- to provide independence sufficient to set strate- sistent with this charge and after careful consider- gic priorities distinct from any existing Commerce ation of the issue, the Commission has concluded agency and to operate apart from policy and that a clearinghouse should not be created. related offices. The NSDS also must be organized Specifically, the Commission interprets the in such a way as to prioritize support for evidence word “clearinghouse” as connoting a data storage building across government, rather than support facility that permanently stores records from mul- specific to any one department, as is the case for tiple databases from multiple agencies and, there- existing PSAs. fore, grows with each new data linkage. Previous The institutional placement and governance panels and commissions either have come close of the NSDS should be considered in terms of any to recommending this type of clearinghouse or implications for key objectives: (1) transparency did recommend one.29 These previous efforts pro- and trust, (2) support for evidence building, (3) voked well-founded concerns about the potential strategic coordination and cooperation, (4) confi- privacy harm such a clearinghouse could raise. dentiality and security, (5) authority and flexibil- As further elaborated in Chapter 4, the Com- ity, (6) scalable functionality, and (7) sustainabil- mission rejects the clearinghouse model in favor ity. An institutional placement within an existing of the NSDS. The NSDS should be designed to link Federal department has a practical benefit, particularly the ability to leverage shared services for 28. Evidence-Based Policymaking Commission Act of 2016 (Public administrative functions, as well as existing pro- Law 114–140, Section 4(b), March 30, 2016). fessional staff, established levels of public trust, 29. As described in Chapter 4, a committee in the 1960s proposed and operational knowledge. creating a national data bank to address the decentralized nature of the Federal government’s data infrastructure. 42 The Promise of Evidence-Based Policymaking

data on an individual project basis only. In contrast Weighing its charge and the “optimal arrangement” to a clearinghouse, it should not lead to the estab- for data collected by government, the Commission lishment of a store of data that grows with every concluded that more of the data the government research project conducted. The data linked for a holds should be accessible for statistical purposes project through the NSDS should be kept structur- to inform programs and policies. Such access op- ally separate from other data linked through the timally will be grounded in clearly articulated au- NSDS for other projects. By strictly enforcing this thority from the Congress for program agencies to design, the NSDS will further the goal of increased make their data available for statistical purposes access to and use of data for specific research and and for agencies envisioned as recipients to use evaluation efforts, without unduly increasing the the data for those purposes. Absent clear direction potential for privacy harm. from the Congress, many Federal departments cur- The Commission envisions the NSDS working in rently may choose not to use existing authorities concert with the existing network of PSAs to facili- or may interpret their statutes as lacking these au- tate secure access to government data for evidence thorities. To provide clarity about permissible sta- building conducted by those inside and outside of tistical uses, the Commission recommends build- government. The PSAs currently house and man- ing on the existing CIPSEA framework. age much of the data critical to Federal evidence The joint benefits to privacy and data access building in order to fulfill their statutory missions. envisioned by the Commission can be realized si- They collectively have appropriate infrastructure, multaneously within an appropriate enabling legal technical expertise, and legal authority to protect framework. The existing CIPSEA framework, in- data. Therefore, the NSDS will both call upon PSAs cluding the provision that authorizes the sharing to address requests by researchers and evaluators, of business data among three designated statistical and assist PSAs in furthering their own research to agencies, permits effective statistical uses of data improve the quality and efficiency of existing sta- within a protected enclave. The Commission rec- tistical programs. Such a relationship will allow the ommends that the CIPSEA framework be extended NSDS to facilitate access and create approved link- by providing NSDS with the powers and exemp- ages, while also ensuring the underlying data re- tions necessary to use, for statistical purposes, main dispersed in their originating agencies. data collected by other agencies. To ensure that the The NSDS may also support evidence building acquisition and use of data by the NSDS occurs in a in states and localities, especially as part of cre- privacy-protecting manner that is at the frontier of ating mutually beneficial partnerships around use what is feasible, the Commission’s recommenda- of Federal, state, and locally held data. The NSDS tion also requires that NSDS adhere to specific new could also offer analytical services to Federal, state, and stringent privacy qualifications outlined else- and local agencies that lack capacity to undertake where in the Commission’s recommendations. The the work on their own, providing both needed ca- Commission does not intend for the new authori- pacity and ensuring privacy best practices. ty given to the NSDS to curtail existing authorities to acquire and link data currently being utilized REC. 2-3: In establishing the NSDS, by the PSAs, but does see value in encouraging all such activity to occur within the CIPSEA frame- the Congress and the President work. In fact, the Commission believes that some should amend the Privacy Act and the PSAs will elect to use the NSDS to facilitate their Confidential Information Protection and own research with potential to address the quality, Statistical Efficiency Act (CIPSEA) to re- burden, or cost-efficiency of statistical production, quire new stringent privacy qualifications and that the NSDS will call upon the PSAs to pro- as a precondition for the NSDS to acquire vide data to facilitate others’ research activities. As a companion to the CIPSEA amendment, the and combine survey and administrative Commission also recommends that the Congress data for solely statistical purposes. At the and President amend the Privacy Act to extend to same time, the Congress should consider NSDS the existing exception to consent require- additional statutory changes to enable on- ments that applies when agencies disclose records going statistical production. within a system of records to the Census Bureau. CIPSEA authorization alone may not be sufficient to The Promise of Evidence-Based Policymaking 43

enable needed access to data, as agencies may also programs.”30 Similarly, under Title 26 of the U.S. have to satisfy the Privacy Act’s requirement to ob- Code, the predominant purpose of research contain consent before disclosing information to third ducted using confidential Federal tax information parties. The recommendation acknowledges that must be “tax administration.”31 Likewise, the law the evidence-building landscape has evolved sig- generally applicable to the Census Bureau, Title 13 nificantly since the enactment of the Privacy Act. of the U.S. Code, permits access to data only for With the Census Bureau exemption, the Congress the narrow purpose of improving Census Bureau demonstrated its belief that the provision of ad- programs. ministrative data under strong confidentiality pro- Where deemed appropriate, the Congress tections for exclusively statistical purposes was safe should act to amend relevant statutes to autho- enough to exempt it from the Privacy Act’s consent rize access and use of data for statistical purposes. requirements. In light of the stringent privacy and The recommendation encourages the Congress to transparency protections built into its procedures, review its prior determinations in the context of the Commission believes there is justification to ex- the desire for increased use of evidence and the tend the same narrow exemption to the NSDS. modernized legal and security environment con- The Commission does not envision that the templated in this report. NSDS will be a locus of statistical production. That role will remain, as today, with the statistical agen- REC. 2-5: The Congress and the Presi- cies in which the Congress has vested the responsi- dent should consider repealing cur- bility for such work. For the purpose of improving the efficiency and precision of key statistical indi- rent bans and limiting future bans on the cators and datasets, these agencies should be able collection and use of data for evidence to engage with the NSDS to carry out research on building. and development of new statistical products using data linked under the authorities newly granted Legislative bans on the collection and use of data to the NSDS. But once research and development create barriers to the use of information for evi- gives way to production, that activity—along with dence building. The Commission recognizes that the linking of data necessary to support it—will the creation of bans may be a mechanism for set- need to revert to the relevant statistical agency or ting priorities for the collection and use of data in agencies. The Commission encourages the Congress a complex policy context. While it is clearly within to consider conforming amendments, including to the purview and responsibility of the Congress to CIPSEA and the Privacy Act, to allow these improve- establish such priorities, absent articulated crite- ments in statistical production processes to occur. ria, bans on data collection and use create a serious impediment to evidence-based policymaking, REC. 2-4: The Congress and the Presi- and could make it difficult or impossible to hold government activity accountable. Provided that dent should review and amend, as ap- the collection and use of data occur consistent propriate, statutes such as Title 13 of the with the Commission’s guiding principles, partic- U.S. Code to allow statistical uses of survey ularly with regard to privacy protection, such col- and administrative data for evidence build- lection and use generally should be allowed. ing within the CIPSEA secure environment. The Commission concluded that any use of data may carry both value and the risk of harm. There The Commission identified several datasets that are is also, however, a risk of harm associated with highly valuable for evidence building (see Appendix the investment of billions of dollars in taxpay- D). Several of the statutes that govern use of these er funds without adequate monitoring and study datasets contain explicit prohibitions or strong of the effectiveness of the taxpayers’ investment. restrictions on use for the purposes described in this report. For example, the Higher Education Opportunity Act limits the use of data collected to 30. Higher Education Opportunity Act (Public Law 110–315, Au- operate Federal student financial assistance gust 14, 2008); https://www.gpo.gov/fdsys/pkg/PLAW-110publ315/ programs to “administration of financial aid pdf/PLAW-110publ315.pdf (accessed August 10, 2017). 31. Internal Revenue Code, 26 USC § 6103(h). 44 The Promise of Evidence-Based Policymaking

In considering the application of a ban on data REC. 2-6: The Congress and the Presi- collection and use, the Congress should create a dent should enact statutory or other means to weigh the various, and sometimes competing, interests at stake. The Commission empha- changes to ensure that state-collected ad- sizes that encouraging the review of bans does not ministrative data on quarterly earnings mean privacy protections should be undermined or are available solely for statistical purposes. that data collection and use should be guaranteed. The data should be available through a sin- Rather, the Commission is encouraging the Con- gle Federal source for solely statistical pur- gress to develop reasonable criteria to help ensure that legislators have the tools and opportunity to poses. carefully weigh the implications of significant bans. The Commission finds that increasing access to As noted previously, the Commission under- state-collected administrative data on quarterly stands the complexity, motivations, and concerns earnings is of primary importance. Numerous Fed- involved in weighing privacy against other consid- eral programs and policies seek to impact earned erations. History is replete with examples of get- income. Data on quarterly earnings are already ting this balance wrong. Yet, with modern priva- collected by states through their administration cy-enhancing technologies, a well-developed legal of unemployment insurance programs, but these framework, and a strong commitment to transpar- data are not easily accessible by researchers for use ency, the Commission believes it is possible to use in evidence-building activities. For a more com- data to evaluate the effectiveness of policies and plete picture of wage recipients, access to quarterly ensure accountability to stakeholders, while also wage data from multiple states is critical. protecting privacy. The Commission also believes The Commission identified several strate- it is incumbent on the Congress and the President gies for improving access to administrative data to regularly and thoroughly consider the value of on quarterly earnings for statistical purposes, policies that preclude evidence building against including expanding access to existing multi-state the prospective public value of the evidence that data sources such as the National Directory of could be produced. New Hires or LEHD or developing a new national UI wage record system (see the box “Expanding

Expanding Access to Quarterly Earnings Data

The Commission identified several possible state wage record data for statistical activities. ways to achieve the goal of improving access to Currently, they have uneven access to these administrative data on quarterly earnings for data, despite the mandatory reporting require- statistical purposes. One approach could be to ment for NDNH and the existence of voluntary expand access for statistical purposes to wage reporting for the LEHD program. records maintained by the National Directory Creating a single system to which states re- of New Hires (NDNH) system or to wage report UI wage record data could reduce the bur- cords maintained by the U.S. Census Bureau as den on states, who currently report the same part of the Longitudinal Employer-Household data to multiple entities, and could reduce the Dynamics Program (LEHD). burden on Federal agencies that spend a signif- Alternatively, the Congress and the Presi- icant amount of resources negotiating multiple dent could create a national Unemployment agreements and memoranda of understand- Insurance (UI) quarterly earnings data system ing with the states. A single system also could within the U.S. Department of Labor. A na- streamline access to this highly valuable data tional UI quarterly earnings data system could source for statistical activities, while enhanc- build on the same UI wage record data that cur- ing the security and privacy of the data through rently support both the NDNH system and the development of standardized procedures for LEHD program. In fact, states consistently have data submission and access. expressed interest in being able to access inter- The Promise of Evidence-Based Policymaking 45

Access to Quarterly Earnings Data”). Each of these microdata from states on participants of all three approaches has strengths and weaknesses. As programs, sample data have limited use in evidence such, the Commission chose to focus its recom- building. Universe data, which includes all program mendation on the desired outcome—the availabil- participants or beneficiaries, are the data that have ity of national administrative data on quarterly maximum value for evidence building. In addition, earnings, accessible from a single Federal source, existing data are inaccessible for evidence building. for use in evidence building—rather than articulat- Recent efforts by the Census Bureau to enter into ing a specific approach. data sharing agreements with states to access these data have been advancing, although slowly. As of REC. 2-7: The President should direct July 2017, there were 15 states sharing SNAP data and 9 states sharing TANF data through this effort, Federal departments that acquire with varying years of data available, limiting the state-collected administrative data to ability to easily conduct multi-state analysis.33 make them available for statistical purpos- In cases where barriers to access are signifi- es. Where there is substantial Federal in- cant and deemed to be misaligned with both the vestment in a program, Federal depart- amount of Federal investment in a program and ments should, consistent with applicable evidence-building needs, the Congress should consider whether to enact statutory changes to require law, direct states to provide the data nec- the submission by states of administrative data on essary to support evidence building, such all program participants that could be combined as complete administrative data when with other data sources for exclusively statistical samples are already provided. activities. While significant, the challenges of expanding the existing statutory authority to require The Commission strongly believes that increas- the submission of universe data in the case of SNAP, ing access to state-collected administrative data UI benefits, and TANF, for example, could be less- is vital to efficiently increasing the volume and ened considerably with the provision of supports quality of evidence that can be produced and to help states comply with a new requirement. The used for policymaking. Availability of state-col- Commission expects that doing so for these three lected administrative data at the Federal level is programs could lead to especially valuable informa- highly variable. In considering which administration for policymakers and program managers. tive data to prioritize for enhanced accessibility, Recognizing the potential value of state-held policymakers might consider the level of Federal datasets related to Federal programs, the Com- investment in the program, the value of the data mission recommends that Federal departments for evidence-based policymaking, and the level conduct a thorough review of relevant statutes of difficulty that might be encountered in seek- and regulations governing the accessibility of ing to increase access to a particular data source. state-collected data as well as any statutory use For example, SNAP, UI, and TANF represent three restrictions related to evidence building to identi- important programs in the current safety net in fy barriers that impede access to the data by qual- the United States, and each involves substantial ified researchers seeking to use the data for sta- Federal and state expenditures. 32 tistical purposes. The Congress and the President Currently there are two distinct challenges in should require Federal agencies to submit a report using state-collected administrative data for Fed- to the Congress summarizing the barriers they eral evidence-building purposes—the data that are identify from a review of the relevant statutes and available for evidence building are limited and what regulations governing these collections. are available are not easily accessible. While the Across the existing jointly administered Fed- Federal government already has the statutory au- eral-state programs in which the submission of thority to collect a sample of linkable administrative universe administrative microdata is statutorily required, the Federal government provides a

32. Trudy Renwick and Liana Fox, The Supplemental Poverty Measure 2015 (Washington, D.C.: U.S. Census Bureau, September 2016) 33. U.S. Census Bureau, “Administrative Data Inventory;” https:// table 5b; https://www.census.gov/content/dam/Census/library/ www2.census.gov/about/linkage/data-file-inventory.pdf (accessed publications/2016/demo/p60-258.pdf (accessed August 10, 2017). August 10, 2017). 46 The Promise of Evidence-Based Policymaking

Figure 7. Research Process REC. 2-8: The Office of Management After Instituting NSDS and Budget should promulgate a single, streamlined process for researchers external Researcher Begins Project Involving Two Agencies to the government to apply, become qualified, and gain approval to access government data that are not publicly available. Approval Researcher Submits Application would remain subject to any restrictions appropriate to the data in question. NSDS When the regulatory framework permits data ac- Qualiﬁes Researcher cess for statistical purposes, researchers still need Analyzes Legal Concerns clear and well-documented protocols and proce- Analyzes Privacy Concerns Analyzes Transparency Concerns dures for data access. A centralized and known set of procedures for requesting access to data allows greater focus on protecting privacy and avoids inadvertently establishing onerous roadblocks to — ApprovedX — Denied accessing data. A streamlined process will ensure that data access requests are all adequately scruti- nized. In fact, the approach for improving access NSDS Solicits Data Researcher May would acknowledge differences in the sensitivity From Both Agencies Work to Address levels of data and set appropriate access require- Data are Received, Concerns and ments and data security and privacy protocols in Privacy Protected, and Re-Apply accordance with applicable laws or regulations. Pre-Formatted as Comparable Sets The Commission recognizes OMB as the primary actor for Recommendation 2-8 in order to capitalize on OMB’s existing legal authority and Data Accessible experience in setting information and statistical to Researcher policy that applies government wide. OMB should Researcher Accesses include key stakeholders, both within and exter- Data Within nal to the government, throughout the process of Secure Environment establishing new procedures and standards related to access and sensitivity. The NSDS will play a lead role in implementing Researcher Provides Process Feedback these new standards (see Figure 7). NSDS will be responsible for consulting and collaborating with Federal departments to assign appropriate sensi- significant amount of support to enable states to tivity levels to datasets they hold. Because sensitiv- comply with reporting requirements. The provi- ity may change over time, it would be appropriate sion of funding for information technology infra- for sensitivity analyses to be reviewed whenever structure and technical assistance can assist states important new information about potential risks in building the necessary data collection systems becomes available. Sensitivity levels could be and reporting structures to be able to comply with based on a standard classification system for data a new requirement for increased data collection sensitivity, as discussed earlier in the chapter. and transmission and to address issues of data in- Evaluators and researchers alike could similar- tegrity. Federal departments should explore and ly be more readily informed about what data are adopt incentives to support either compliance available at what levels of sensitivity with what with a requirement for states to make data avail- access restrictions. As appropriate, the Federal able to the Federal government or to incentivize government should provide detailed technical the voluntary action of making administrative data documentation useful in planning projects data available to the Federal government in cases and designing research studies, as discussed fur- where a requirement is not established. ther in Chapter 4. ■ The Promise of Evidence-Based Policymaking 47

3 Enhancing Privacy Protections for Federal Evidence Building

Improve data quality, integrity, and security by updating laws to require risk assessments for data releases, embracing cutting-edge technologies, and prioritizing data stewardship and the public trust.

Recommendations

3-1: The Congress and the President 3-3: The President should direct Federal should amend the Privacy Act and the departments to assign a senior official Confidential Information Protection and Sta- the responsibility for coordinating access to tistical Efficiency Act (CIPSEA) to require Fed- and stewardship of the department’s data re- eral departments to conduct a comprehensive sources for evidence building in collaboration risk assessment on de-identified confidential with senior department information technolo- data intended for public release. De-identified gy, privacy, and other leaders. A Principal Sta- confidential data subject to the Privacy Act and tistical Agency head, or other appropriately CIPSEA should be made available only after a qualified senior official, should serve this func- disclosure review board (1) approves the re- tion. lease and (2) publicly provides the risk assessment and a description of steps taken to miti- 3-4: The Congress and the President gate risk. should enact legislation to codify relevant portions of Office of Management and 3-2: The President should direct Federal Budget Statistical Policy Directive #1 to protect departments, in coordination with the public trust by ensuring that data acquired un- National Secure Data Service, to adopt state-of- der a pledge of confidentiality are kept confi- the-art database, cryptography, privacy-preserv- dential and used exclusively for statistical pur- ing, and privacy-enhancing technologies for poses. confidential data used for evidence building. 48 The Promise of Evidence-Based Policymaking

he Federal government has a legitimate need Figure 8. The Privacy/Data Use Tand responsibility to use data for evidence Frontier Curve building. At the same time, the public has a legitimate interest in knowing that their government More is protecting their privacy while using their data. With emerging technologies, the Commission believes that the country can have both (see Figure Future 8). The knowledge and technology now exist to achieve the Commission’s vision for improving secure access to confidential data while simultane- Data ously enhancing privacy. Through law and policy, it Use is possible to ensure that Federal departments are consistently adopting the most up-to-date meth- Current ods to protect data used for evidence building.

Privacy, when properly Less More “ Privacy implemented through good programs and early on, promotes Privacy-Enhancing Technologies Allow Greater Protection of Privacy innovation in every case and While Increasing Access to Data allows for and encourages the wide scale adoption of new provide secure access to confidential data. In addi- technology. In fact, privacy, when tion, privacy considerations have underscored all implemented correctly, fosters of the Commission’s deliberations and priorities. more information sharing, not Privacy, data security, confidentiality, and public transparency were frequent themes in written less. ” comments provided to the Commission and in – Marc Groman, then-Senior Advisor for each of the three public hearings. Finally, Commis- Privacy at OMB and Chair of the Federal sioners and staff sought out additional experts on Privacy Council, Commission Meeting, privacy-protective technologies and approaches to Washington, D.C., September 9, 2016 develop the recommendations in this chapter. Just as the Fair Information Practice Principles Throughout its deliberations, the Commission guide the Federal government’s privacy policy, the prioritized consideration of privacy on behalf of the Commission believes that these Principles should American public. The importance the Commission similarly guide the use of data for evidence-based attached to privacy is captured in the first of the policymaking in accordance with the Commis- guiding principles for evidence-based policymak- sion’s guiding principle on privacy (see the box ing stated in Chapter 1: “Individual privacy and “Fair Information Practice Principles”). By con- confidentiality must be respected in the generation sidering each of the Fair Information Practice and use of data and evidence.” The Evidence-Based Principles in the context of evidence building, Policymaking Commission Act called for the Presi- the Commission has taken an ethical approach to dent and the Congress to appoint one-third of the data use that addresses both privacy and the need Commissioners specifically for their expertise in to generate accurate and reliable evidence. Many “protecting personally identifiable information and of the Commission’s recommendations in other data minimization.”1 The Commission devoted two chapters address transparency, individual partic- public meetings to testimony about privacy and two ipation, purpose specification, data minimization, other meetings included substantial discussions of use limitation, and accountability and auditing. how state, Federal, and international governments The recommendations in this chapter specifically address the principle of data quality and integri- 1. Evidence-Based Policymaking Commission Act of 2016 (Public ty and the principle of security, and also relate to Law 114–140, Section 3(a), March 30, 2016). other principles, especially transparency. The Promise of Evidence-Based Policymaking 49

Fair Information Practice Principles1 • Transparency: notify individuals regard- • Use Limitation: use PII solely for the pur- ing collection, use, dissemination, and pose(s) specified maintenance of personally identifiable • Data Quality and Integrity: ensure that PII information (PII) is accurate, relevant, timely, and complete involve the • Individual Participation: • Security: protect PII through appropriate individual in the process of using PII and, security safeguards to the extent practicable, seek individual consent for the collection, use, dissemina- • Accountability and Auditing: be account- tion, and maintenance of PII able for complying with these principles and audit the actual use of PII to demon- • Purpose Specification: articulate the strate compliance authority that permits the collection of PII and specifically articulate the purpose(s) for which the PII is intended to be used 1. U.S. Department of Health, Education, and Welfare, Records, Computers, and the Rights of Citizens, Report of the Secretary’s • Data Minimization: collect PII that is Advisory Committee on Automated Personal Data Systems directly relevant and necessary to accom- (Washington, D.C., July 1973); https://www.justice.gov/opcl/docs/ rec-com-rights.pdf (accessed August 10, 2017); The White House, plish the specified purpose(s) and only re- “Appendix A. Fair Information Practice Principles,” National tain PII for as long as is necessary to fulfill Strategy for Trusted Entities in Cyberspace: Enhancing Online Choice the specified purpose(s) Efficiency, Security and Privacy (2011); https://obamawhitehouse. archives.gov/sites/default/files/rss_viewer/NSTICstrategy_041511. pdf (accessed August 10, 2017).

While privacy harms can come from many evidence means minimizing the risk of such pri- sources, the Commission’s recommendations fo- vacy harms. cus on establishing laws, policies, and procedures Risk is “a measure of the extent to which an to minimize the risk that confidential data used entity is threatened by a potential circumstance for evidence building may be misused to cause or event, and typically is a function of: (i) the ad- privacy harm. Privacy scholar Ryan Calo notes verse impact, or magnitude of harm, that would that privacy harms can be both subjective and ob- arise if the circumstance or event occurs; and jective: “The subjective category of privacy harm is (ii) the likelihood of occurrence.” 3 Risk can come the perception of unwanted observation. This cat- from bad actors who actively attempt to break egory describes unwelcome mental states—anxi- into confidential data that they know they are ety, embarrassment, fear—that stem from the be- not authorized to use and from well-intentioned lief that one is being watched or monitored…The people who inadvertently violate the procedures objective category of privacy harm is the unantic- established to protect privacy. Risk also can come ipated or coerced use of information concerning from outdated data encryption protocols and un- a person against that person. These are negative, even capacity to apply up-to-date data protection external actions justified by reference to personal techniques. An increasingly important source of information.”2 Privacy harm occurs when an indi- risk is the growing number of publicly available vidual is adversely affected because of misuse of data sources containing personal information that data; aspects of this harm can also affect business- can be combined with publicly available datasets es or other entities if, for example, their confiden- designed to be de-identified, thus permitting the tial data are inappropriately accessed and used. Respecting privacy in the generation and use of 3. Office of Management and Budget, “Managing Information as a Strategic Resource,” Circular A–130 (updated July 27, 2016); 2. Ryan Calo, “The Boundaries of Privacy Harm,”Indiana Law Jour- https://obamawhitehouse.archives.gov/sites/default/files/omb/ nal 86, no. 3 (2011); https://ssrn.com/abstract=1641487 (accessed assets/OMB/circulars/a130/a130revised.pdf (accessed August 10, August 10, 2017). 2017). 50 The Promise of Evidence-Based Policymaking

connection of identities with previously private, are respectful of privacy.5 Not surprisingly, public sensitive information. This risk must be managed. trust in the accuracy and validity of statistical data This chapter describes a need to better protect reflect the public’s trust in the statistical agencies individuals from privacy harms caused by unau- that produced them.6 thorized use or disclosure of Federal confidential The laws governing PSAs give them the author- data. The Commission developed specific recom- ity to protect confidential data by allowing use of mendations about the quality, integrity, and se- the data for exclusively statistical purposes. Sta- curity of confidential data to prevent individuals tistical purposes include analyzing individual re- and organizations from being identified without cords along with many other individual records to authorization. Private sector research has contrib- produce descriptions of groups of people or cre- uted to rapid advances in computer science that ating averages about groups within society, the make data safer and easier to access. Adoption of economy, or the environment. PSAs cannot and these emerging technologies can accelerate se- do not use data records about individual people, cure access to the government’s confidential data businesses, or other entities for non-statistical resources for evidence building. Federal depart- purposes, such as administering a program, deter- ments need leaders with the skills and vision to mining benefits, or enforcing laws. fully harness the promise of privacy-protective PSAs pioneered and continue to advance “sta- data stewardship. tistical disclosure limitation” techniques, discussed below, with the purpose “to ensure that the “What people care most about is risk of disclosing confidential information about not simply restricting the flow of identifiable persons, businesses or other units will be very small.”7 PSAs have a long history of pro- information but ensuring that it viding secure access to the least amount of data 4 flows appropriately.” needed for approved statistical purposes and of – Helen Nissenbaum, Professor of ensuring that researchers abide by strict rules to Information Science at Cornell Tech protect confidentiality. Reducing barriers to accessing and using administrative data for evidence building requires Findings the same kind of attention to confidentiality that is embedded in the culture and history of the PSAs. Protecting privacy has been a priority for Federal Through the establishment of disclosure review evidence building for many decades. Many Feder- boards, PSAs apply best practice statistical disclo- al departments use modern methods to keep data sure limitation techniques and assess the risk of secure and confidential, but government practices public data releases. Disclosure review boards as- must adapt to new threats and take advantage of sess proposed public data releases to ensure that technologies that better protect data. enough identifying information has been removed For decades, Principal Statistical Agencies or masked to protect confidentiality. All Federal (PSAs) such as the U.S. Census Bureau have demonstrated responsible stewardship of data collected through censuses and surveys. These 5. Jennifer Hunter Childs, “Understanding Trust in Official Sta- tistics in the U.S. and Implications For Administrative Record data are used to create important national statis- Use” (prepared for the Commission, Washington, D.C., December tical indicators, such as the unemployment rate, 2016). gross domestic product, health outcome statis- 6. Melissa Mitchell, Jennifer Hunter Childs, and Morgan Earp, tics, and crime statistics. Public trust in the Fed- “Monitoring and Detecting Shocks That Influence Public Trust To- eral Statistical System historically has been high, wards the Federal Statistical System” (paper presented at the 68th Annual Conference of the American Association of Public Opinion with a majority of Americans concluding that gov- Research, Boston, MA, May 2013); http://www.aapor.org/AAPOR_ ernment can be trusted to produce statistics that Main/media/AnnualMeetingProceedings/2013/Session_H-5-2- Mitchell.pdf (accessed August 10, 2017). 7. Federal Committee on Statistical Methodology (FCSM), “Re- port on Statistical Disclosure Limitation Methodology,” Statisti- 4. Helen Nissenbaum, Privacy in Context: Technology, Policy, and cal Policy Working Paper 22 (Washington, D.C.: FCSM, December the Integrity of Social Life (Stanford, CA: Stanford University Press, 2005); https://s3.amazonaws.com/sitesusa/wp-content/uploads/sites/ 2010). 242/2014/04/spwp22.pdf (accessed August 10, 2017). The Promise of Evidence-Based Policymaking 51

departments should do the same. When govern- and Confidential Information Protection and Sta- ment pledges to keep data confidential, the data tistical Efficiency Act of 2002 (CIPSEA).10 These should have strong protections, and data use should laws seek to protect confidentiality by requiring generally be made known to the American public. that released records be in a form that is “not indi- The Federal government’s Open Data initiative vidually identifiable”11 (Privacy Act) or “prevents has made more government data publicly avail- the identity of the respondent…[from being] able than ever before. Even so, “agencies must reasonably inferred by either direct or indirect review the information collected or created for means”12 (CIPSEA). The Privacy Act and CIPSEA, valid restrictions to release to determine whether as well as other Federal laws governing confiden- it can be made publicly available, consistent with tial data, share a basic goal: to ensure that infor- the Open Government Directive’s presumption in mation given to the Federal government under a favor of openness, and to the extent permitted by promise of confidentiality is not released to any- law and subject to privacy, confidentiality pledge, one in identifiable form, except as allowed by law. security, trade secret, contractual, or other valid As described in Chapter 2, CIPSEA authorizes ac- restrictions to release.”8 cess to identifiable confidential data for exclusive- Throughout its fact-finding phase, the Commis- ly statistical purposes by (1) trained staff in PSAs sion learned about potential threats to the confi- and recognized statistical units and (2) by external dentiality of Federal data used for evidence building researchers who are trained and qualified by PSAs. and heard from experts about new and emerging Even then, direct identifiers are removed from methods to address such threats. The findings pre- analysis files and their access protected and limit- sented in this chapter highlight three challenges ed to a few technical staff. Several Federal agencies that keep the Federal government’s confidentiality that collect confidential administrative data under protections from being as strong as possible. First, the Privacy Act also provide secure access to identi- laws governing Federal data sometimes do not re- fiable information for research purposes. quire a risk assessment or disclosure review prior PSAs release “de-identified” confidential data to the public release of de-identified confidential collected under CIPSEA as part of their mission and data. Second, Federal departments must adapt to legal authority to disseminate data in their topical new threats to information security and privacy and domains. Many program agencies that collect con- take advantage of emerging technologies that better fidential information under the Privacy Act also re- protect data. Third, many Federal departments lack lease “de-identified” data under their own statutory senior leadership focused on or experienced in data authorities. These public-use files are easily acces- stewardship for evidence building, and the PSAs, sible and widely available; many public-use files while often best positioned to provide such leader- can be downloaded from department websites. ship, have uneven independence and authority. In the Federal government today, however, there is great variation in the meaning of “de-identified” Challenge #1: Key laws governing Federal data. Following CIPSEA implementation guidance data do not require the application of best from OMB, statistical agencies routinely go beyond practice statistical disclosure limitation removing direct identifiers to apply appropriate techniques or a risk assessment prior to statistical disclosure limitation techniques before the public release of de-identified confi- seeking disclosure review board approval to re- 13 dential data. lease de-identified datasets. In contrast, program As discussed in Chapters 1 and 2, two laws that govern much of the confidential Federal data used 10. Confidential Information Protection and Statistical Efficiency for evidence building are the Privacy Act of 19749 Act of 2002,” (CIPSEA), (Public Law 107–347, Title V, 2002). 11. Privacy Act, 5 USC § 552a (b)(5) (1974). 12. Confidential Information Protection and Statistical Efficiency 8. Office of Management and Budget (OMB), “Open Data Policy: Act of 2002,” (CIPSEA), (Public Law 107–347, Section 502, 2002). Managing Information as an Asset” (memorandum M–13–13, 13. “Implementation Guidance for Title V of the E-Government Washington, D.C.: OMB, Executive Office of the President, Novem- Act, Confidential Information Protection and Statistical Efficien- ber 8, 2016); https://project-open-data.cio.gov/policy-memo (accessed cy Act of 2002 (CIPSEA),” Federal Register 72, no. 115 (June 15, August 10, 2017). 2007): 33361; https://www.gpo.gov/fdsys/granule/FR-2007-06-15/ 9. Privacy Act, 5 USC § 552a et seq.(1974). E7-11542/content-detail.html (accessed August 10, 2017). 52 The Promise of Evidence-Based Policymaking

agencies routinely remove direct identifiers follow- Federal departments that collect and store confi- ing legal guidance from general counsels about im- dential data are increasingly concerned about the plementing the Privacy Act, but only some apply risk of privacy harm that comes from data security rigorous statistical disclosure limitation techniques breaches and unauthorized re-identification of in- before releasing de-identified data. According to dividual identities in confidential data. The Federal the Federal Committee on Statistical Methodolo- government has an obligation to develop strategies gy’s Working Paper 22, “agencies that need to pro- to assess and minimize the impact of these risks tect data should move as far as possible toward the when it uses government data for evidence build- use of a small number of standardized disclosure ing. The Commission’s witnesses indicated that limitation methods whose effectiveness has been new technologies can be part of the solution. demonstrated.”14 Federal departments run the continuum of Preventing Unauthorized capacity and expertise to apply state-of-the-art Re-Identification methods for creating de-identified data that min- imizes the risk of re-identification—that is, the Early laws on privacy in the United States gener- chance that any individual in a publicly released ally described a relatively short and finite list of dataset could be uniquely identified. Minimizing direct identifiers that could easily be removed to the risk of re-identification is important because create de-identified data for broad distribution. of the potential that a bad actor could inappropri- Accepted practice for many years meant simply re- ately figure out which record belongs to an indi- moving direct identifiers such as names, address- vidual and inflict harm as a result. es, and dates of birth. By the late 1970s, however, Statisticians, computer scientists, and the the Federal Committee on Statistical Methodology recognized that de-identification of data required far more than simply removing elements from a 14. FCSM, Report on Statistical Disclosure Limitation Methodology: 6. dataset and advocated the adoption of more formal

Statistical Disclosure Limitation Techniques from Working Paper 22

“To reduce the potential for disclosure… 3. Adding or multiplying by random num- public use data files: bers (noise), 1. Include data from only a sample of the 4. Swapping or rank swapping (also called population, switching), 2. Do not include obvious identifiers, 5. Selecting records at random, blanking out selected variables and imputing for 3. Limit geographic detail, and them (also called blank and impute), 4. Limit the number and detailed break- and down of categories within variables on 6. Aggregating across small groups of the file. respondents and replacing one individ- Additional methods typically used to ual’s reported value with the average 1 disguise high risk variables include: (also called blurring).” 1. Truncation of extreme codes for certain variables (top or bottom-coding), 1. FCSM, Report on Statistical Disclosure Limitation Methodology: 2. Recoding into intervals or rounding, 24–25. The Promise of Evidence-Based Policymaking 53

statistical disclosure limitation methods to protect effort… In general, de-identification individuals’ privacy in public data releases.15 approaches are designed to push The potential danger of re-identification from data to the left [towards not being indirect identifiers and contextual information related to individuals] while retain- in datasets stripped of direct identifiers became ing some desired utility, lowering more widely recognized in the 1990s. As a result, the risk of distributing de-identified PSAs increasingly began to rely on statistical dis- data to a broader population or the closure limitation techniques to mask the data. general public.”18 The Federal Committee on Statistical Methodolo- gy’s Working Paper 22, issued in 2005, stated: “Ev- Current best practice statistical disclosure lim- ery agency or unit within an agency that releases itation methods may become less effective in the statistical data should be capable of selecting and future because of the growing availability of pub- applying suitable disclosure limitation procedures lic and private information about individuals and to all the data it releases.”16 By the early 2000s, the computing techniques to exploit them. Adding to prevailing way to implement strong de-identifica- currently available information, Federal depart- tion protocols was to use statistical disclosure lim- ments are now directed to release as much data itation techniques on a release-by-release basis to as possible to the public under the Open Data ini- reduce the risk of re-identification that could po- tiative. However, advances in computer science tentially cause privacy harm. and technology have produced promising new Today, a variety of statistical disclosure lim- enhanced statistical disclosure limitation tech- itation techniques, methods, and technologies niques that have the potential to provide stronger are used to reduce the identifiability of a dataset. confidentiality protections by minimizing the risk One relatively new technique, k-anonymity, pro- of re-identification. tects confidentiality by adjusting the data to make sure that all combinations of potentially identi- Assessing the Risk of Re-Identification fiable information reflect a minimum number of individuals (k).17 No existing statistical disclosure The Privacy Act and CIPSEA do not currentlyre - limitation method, however, is able to complete- quire agencies to assess the risk of unauthorized ly eliminate the risk of re-identification. As Sim- re-identification for de-identified confidential son Garfinkel, formerly of the National Institute data; however, the CIPSEA implementation guid- of Standards and Technology, the Federal gov- ance includes the requirement that “for CIPSEA ernment’s standard-setting body for information protected information, the agency as well as any technology, explains: agent accessing the information shall ensure that any dissemination of information based on con- “…all data exist on an identifiability fidential information is done in a manner that spectrum. At one end…are data that preserves the confidentiality of the informa- are not related to individuals (for tion.”19 Today, each agency that releases de-iden- example, historical weather records) tified confidential data has to decide for itself how and therefore pose no privacy risk. it will assess the risk of re-identification and the At the other end…are data that are resulting potential for harm to individuals from linked directly to specific individu- misuse of individual information. als. Between these two endpoints One demonstrated approach for carrying out the are data that can be linked with responsibility to assess risk is to establish a disclosure review board. Most of the 13 PSAs and a few other Federal agencies have established formal disclosure 15. FCSM, “Report on Statistical Disclosure and Disclosure Avoid- ance Techniques,” Statistical Policy Working Paper 2 (Washington, D.C.: FCSM, 1978); https://s3.amazonaws.com/sitesusa/wp-content/ 18. Simson L. Garfinkel, “De-Identification of Personal Informa- uploads/sites/242/2014/04/spwp2.pdf (accessed August 10, 2017). tion,” National Institute of Standards and Technology (NIST) In- 16. FCSM, Report on Statistical Disclosure Limitation Methodology: 2. ternal Report 8053 (Gaithersburg, MD: NIST, October 2015): 5; https://dx.doi.org/10.6028/NIST.IR.8053 (accessed August 10, 2017). 17. Latanya Sweeney, “K-Anonymity: A Model for Protecting Pri- vacy,” International Journal on Uncertainty, Fuzziness and Knowl- 19. “Implementation Guidance for Title V of the E-Government edge-Based Systems 10, no. 5 (2002): 557–570. Act, CIPSEA.” 54 The Promise of Evidence-Based Policymaking

review boards that assess the risk of re-identifica- Data collected and released by state and local tion in the agencies’ public data releases.20 Having governments, private companies, non-profit or- a disclosure review board allows a department to ganizations, and researchers contribute to the cu- assess the underlying risk of re-identification prior mulative amount of information available about to public data release and assess whether the sta- individuals and businesses that could be used for tistical disclosure limitation methods applied to the re-identification. Researchers recently demon- data have effectively mitigated the risk. All PSAs strated how data on air and dust samples from 50 conduct disclosure review and typically document homes in two communities in California could be the results in “safe to release” memos or disclosure combined with data released under the Safe Har- review reports for public data releases. bor provisions of the Health Insurance Portabil- The establishment of a centralized disclosure ity and Accountability Act (HIPAA) to “uniquely review board within a department has added ben- and correctly identify [in one community] 8 of 32 efits, in that each data release can be considered (25 percent) by name and 9 of 32 (28 percent) by within the context of all other planned data re- address.”21 leases across the department (see the box “Estab- Many programs have released de-identified lishing a Centralized Disclosure Review Board"). public-use data files for decades without formally In addition, a centralized disclosure review board assessing risk. However, the broader context with- can leverage expertise from across the depart- in which these releases are occurring has changed ment, bringing together staff that have technical as the amount of information about individuals skills, such as mastery of statistical disclosure limitation methods, and specialized program knowledge about the contents of a particular dataset. 21. Latanya Sweeney, Ji Su Yon, Laura Perovich, Katherine E Bor- onow, Phil Brown, and Julia Green Brody, “Re-identification Risks in HIPAA Safe Harbor Data: A Study of Data From One Environ- 20. FCSM, Report on Statistical Disclosure Limitation Methodology. mental Health Study,” Technology Science (August 28, 2017).

Establishing a Centralized Disclosure Review Board

In 2013, the U.S. Department of Education es- essary to ensure that the department releases tablished a centralized disclosure review board as much usable data as possible while protect- (the ED-DRB) to review and approve privacy ing privacy. The ED-DRB consists of staff with protections for public releases of the depart- expertise in privacy law, policy priorities, dis- ment’s administrative data. Prior to the ED- closure risk assessment, and the application DRB’s creation, each of the department’s prin- of statistical disclosure limitation techniques, cipal offices was responsible for selecting and as well as subject matter experts from data-re- applying privacy-protection methods for their leasing offices in the department. Education own data releases. This decentralized approach Department staff participate in the ED-DRB as was problematic, as data releases by one office one part of their regular duties. Since its cre- could affect the re-identification risk of related ation, the ED-DRB has been a model for other data released by other offices. Senior depart- agencies seeking to improve the consistency ment leaders recognized that decentralization and efficacy of the disclosure review process for of risk assessments resulted in substantial vari- administrative data releases.1 ation across offices about the acceptable level of re-identification risk in public data releases. 1. Federal CIO Council Innovation Committee, The Data The Education Department chartered the Disclosure Decision: Use Case Study (Washington, D.C.: Gov- ED-DRB to be the central point for review and ernment Printing Office, 2015);https://s3.amazonaws.com/ approval of privacy protections for the majority sitesusa/wp-content/uploads/sites/1151/downloads/2015/03/ The%20Data%20Disclosure%20Decision%20-%20Depart- of the department’s administrative data releas- ment%20of%20Education%20Case%20Study_Mar%202015. es. The board has the technical expertise nec- pdf (accessed August 10, 2017). The Promise of Evidence-Based Policymaking 55

that is publicly available has grown and the tech- record. In the absence of strict protocols for using nology that can permit unauthorized re-identifi- confidential data, an inexperienced researcher cation has improved. Within the Federal govern- may attempt to take a screen shot of findings that ment alone, the Open Data initiative made over have not been through the required disclosure re- 150,000 datasets accessible through a single web- view or use other means to remove information site, including many administrative datasets nev- not approved for release. Federal departments are er before released to the public.22 While releasing required to have internal controls to prevent and these data can generate tremendous value, en- address poor data handling practices and to im- abling entrepreneurs to produce better products pose penalties for any procedural violations. Fed- and departments to understand their work better, eral guidance, however, “does not prescribe how it is important that the risks that may accompany management designs, implements, and operates data releases also be assessed.23 The Commission an internal control system.”25 learned that a robust approach to risk assessment, To keep data safe from misuse, the Federal gov- one that considers the availability of external data ernment must keep data secure and ensure appro- in deciding what is safe to release, is paramount to priate training, controls, and accountability. Data the success of all future evidence building. security is the development and implementation Agencies that link electronic records for inter- of “management, operational, and technical as- nal government administrative purposes related pects of protecting the confidentiality, integrity, to financial benefits are required today to assess and availability of federal information and infor- the risk of the linkage and develop procedures for mation systems.”26 Many Federal agencies collect safeguarding the data.24 The system used for the identifiable information in accordance with their assessment is cumbersome and could be simpli- statutory missions and store those data on govern- fied, and statistical activities are exempt. However, ment servers. A breach of a data system, or a mis- the transparent accountability portion of the pro- use of data, resulting in the release of identifiable cess is instructive for evidence building. Agencies information could lead to an increased risk of indi- must issue a Federal Register notice and post the viduals being re-identified in a dataset released to written agreement between the relevant agencies the public under CIPSEA or the Privacy Act. on each agency’s website. For example, the U.S. Data security in Federal departments today Department of Education and the U.S. Department is generally the responsibility of Chief Infor- of Veteran’s Affairs have a publicly available agree- mation Officers who implement standards pro- ment to link administrative data to determine eli- mulgated by OMB in consultation with the Na- gibility for veteran’s educational benefits. tional Institute of Standards and Technology. A recent report from the Commission on Enhanc- Challenge #2: Federal departments must ing National Cybersecurity sought to develop adapt to new threats to information se- “actionable recommendations for securing and curity and privacy and take advantage of growing the digital economy by strengthening emerging technologies that better protect cybersecurity in the public and private sectors.”27 data. Cyberattacks on confidential data increase the risk of privacy harm and may erode public trust. The risk of re-identification is not the only source of potential privacy harm from personal informa- 25. Government Accountability Office (GAO),Standards for Inter- tion collected by the Federal government. Staff nal Control in the Federal Government (Washington, D.C.: GAO, Sep- with otherwise legitimate reasons to access per- tember 2014): 8; http://www.gao.gov/assets/670/665712.pdf. sonal information may misuse such access, for ex- 26. Marianne Swanson, Joan Hash, and Pauline Bowen, Guide for ample, by browsing a celebrity’s or acquaintance’s Developing Security Plans for Federal Information Systems, National Institute of Standards and Technology (NIST) Special Publication 800–18 (Gaithersburg, MD: NIST, February 2006): 13; http:// nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-18r1. 22. DATA.gov, “Federal Agency Participation;” https://www.data. pdf (accessed August 10, 2017). gov/metrics (accessed August 10, 2017). 27. Commission on Enhancing National Cybersecurity, “Report on 23. DATA.gov, “Federal Agency Participation.” Securing and Growing the Digital Economy” (December 1, 2016): 24. Computer Matching and Privacy Protection Act of 1988 (Pub- i; https://www.nist.gov/sites/default/files/documents/2016/12/02/ lic Law 100–503); https://www.gpo.gov/fdsys/pkg/STATUTE-102/ cybersecurity-commission-report-final-post.pdf (accessed August 10, pdf/STATUTE-102-Pg2507.pdf (accessed August 10, 2017). 2017). 56 The Promise of Evidence-Based Policymaking

Well-publicized cybersecurity incidents in the of re-identification, guard against other forms of past have affected Federal systems and resulted privacy harm, and allow for increased use of data in the unintentional release of millions of confi- to support evidence-based policymaking. In par- dential records, raising concerns about vulner- ticular, privacy-protective technologies and proto- abilities in the level of data security of Federal cols can expand access to data while potentially systems. Nearly all Federal agencies must assess improving, rather than adversely affecting, pri- and secure information systems under the Federal vacy. Because technology is always evolving, the Information Security Management Act (FISMA).28 Commission does not endorse the adoption of any PSAs and the three other statistical units autho- one specific approach. Rather, the Commission rized to protect data under CIPSEA have an explicit acknowledges the critical need for leadership and requirement to “ensure the physical security and partnerships that will advance and accelerate the information systems security where data protect- adoption of technologies that improve authorized ed under CIPSEA are accessed and stored.”29 The access and protect privacy. National Institute of Standards and Technology is As the Commission learned, new approaches currently collaborating with Federal departments hold the potential for enhancing privacy protec- and the private sector to establish new engineer- tion. Examples include differential privacy, Secure ing standards and privacy risk assessments intend- Multiparty Computation, and synthetic data (see ed to minimize security risk in Federal information the box “Emerging Approaches That Enhance Pri- technology systems.30 vacy Protections”). PSAs respond to data security mandates under CIPSEA and individual authorizing statutes by re- “Differential privacy is a quiring computer systems that store confidential mathematical guarantee that data be kept completely separate from other systems. In practice, this means that PSAs store confi- an individual data contributor dential data used for evidence building on servers will not be affected, adversely or that are not connected to other data systems with- otherwise, by allowing her data to in their departments. Data can be extracted from be used in any study or analysis, these servers only by authorized individuals for exclusively statistical purposes such as to create the no matter what other studies, PSA’s own production statistics or to combine data datasets, or information sources, across authorized agencies for approved purposes. are—or will become—available.” The Federal evidence-building community – Cynthia Dwork, Microsoft Research (former), must keep pace with evolving technology to main- Commission Meeting, Washington, D.C., tain the security of data and the trust of the Ameri- September 9, 2016 can public. While there are multiple approaches to protecting data against misuse, the Commission believes that technology offers part of the solution The Census Bureau has led efforts among for enhancing privacy while allowing more use of PSAs to use enhanced statistical disclosure lim- data to satisfy the growing demand for evidence itation tools now made possible by cutting-edge to support policymaking. technologies. For example, the Census Bureau New and evolving technologies have the po- is beginning to use synthetic data to create data tential to improve data security, minimize risk tools that answer interesting policy questions while using differential privacy to measure the risk of re-identification and keep it below a de- 28. E-Government Act of 2002 (Public Law 107–347, Title III). fined threshold (see the box “OnTheMap: Differ- 29. “Implementation Guidance for Title V of the E-Government ential Privacy in Practice at the Census Bureau”). Act, CIPSEA.” Adopting new privacy-protective technologies 30. Sean Brooks, Michael Garcia, Naomi Lefkovitz, Suzanne Light- will pose implementation challenges even once man, and Ellen Nadeau, An Introduction to Privacy Engineering and the methods have been sufficiently demonstrat- Risk Management in Federal Systems, NIST Internal Report 8062 ed in practice. One substantial challenge, for ex- (Gaithersburg, MD: NIST, January 2017); http://nvlpubs.nist.gov/ nistpubs/ir/2017/NIST.IR.8062.pdf (accessed August 10, 2017). ample, is that many departments currently use The Promise of Evidence-Based Policymaking 57

Emerging Approaches That Enhance Privacy Protections

Differential privacy is a formal privacy con- SMC in the real world.3 The technique allows cept that applies to analyses performed over one party to jointly compute a function with collections of sensitive personal information.1 another party, while never revealing one an- It provides a quantifiable measure of the ex- other’s private data. SMC and its variations rely cess privacy risk any individual may incur due on mathematical principles based on cryptog- to their data being included in an analysis, as raphy to compute answers over databases from compared with their data not being included. different locations and controlled by different Because the application of differential pri- organizations without either side needing to vacy includes the mathematical computa- see individual records of the other. In short, tion of the risk to privacy loss in any given SMC enables distributive computing that al- analysis, parameters are established that set lows data owners to maintain their data “in bounds on what amount of privacy loss is ac- their own silos” and yet compute results on the ceptable. These parameters are then applied combined data. to control the tradeoff between protection of Synthetic Data: Verification Model: A individual privacy on the one hand and accu- synthetic dataset mimics an original dataset, racy of the performed analyses on the other. by replacing actual values from the original Research over the last decade has provid- data with altered values that still retain many ed solid theoretical grounds for differential (but not necessarily all) important statistical privacy. Algorithms realizing optimal priva- properties of the real dataset.4 Synthetic data- cy-accuracy tradeoffs are emerging for a vari- sets include more detailed data with lower ety of statistical and machine learning tasks. disclosure risk because the original data are In particular, these algorithms can enable pri- changed. But for the same reason, analysts may vacy-preserving computations using data that be uncertain about the precision of the results. more traditional techniques would need to re- Using a verification server is one way to ad- dact. Notably, differential privacy is currently dress this uncertainty. The verification server the only framework with mathematically prov- runs the analysis on the original data and then en bounds on the accumulated privacy risk re- tells the analyst how different the results are sulting from the composition of several analy- between the two datasets. If the analyst feels ses. The first real-world uses now exist (by the that the result using synthetic data is “good U.S. Census Bureau and Internet companies enough,” then they may choose to work solely such as Apple, Google, and Uber). with the synthetic data; but if the analyst feels Secure Multiparty Computation, or SMC, that the result using synthetic data is not “good is a method for conducting privacy-preserv- enough,” they may choose to pursue access to ing data analysis.2 The concept of SMC has the original dataset. been in existence for over 40 years. However, recent advances in algorithms and computational power have made it feasible to achieve 3. Dan Bogdanov, Liina Kamm, Baldur Kubo, Reimo Rebane, Ville Sokk, and Riivo Talviste, “Students and Taxes: a Priva- cy-Preserving Study Using Secure Computation,” Proceedings 1. National Academies of Sciences, Engineering, and Medi- on Privacy Enhancing Technologies no. 3 (July 2016); https:// cine. Innovations in Federal Statistics: Combining Data Sourc- doi:10.1515/popets-2016-0019 (accessed August 10, 2017). es While Protecting Privacy (Washington, D.C.: National Acad- 4. National Institutes of Health Office of Behavioral and Social emies Press, 2017). Science Research, “Synthetic Data: Protecting Data Privacy in 2. Yehuda Lindell and Benny Pinkas, “Secure Multiparty Com- an Era of Big Data” (October 15, 2015); https://obssr.od.nih.gov/ putation for Privacy-Preserving Data Mining,” in Encyclopedia synthetic-data-protecting-data-privacy-in-an-era-of-big-data of Data Warehousing and Mining. (accessed August 10, 2017). 58 The Promise of Evidence-Based Policymaking

techniques that assess and seek to minimize the technologies and techniques to provide secure ac- risk of re-identification of a particular data release cess to data, the government must consider the as a stand-alone matter. In contrast, full imple- need for reproducibility. mentation of differential privacy requires determining the cumulative re-identification risk of Challenge #3: Many Federal departments data releases over the entire life-span of the data. lack senior leadership focused on data The widespread adoption of techniques such as differential privacy would require departments to stewardship, and PSAs have uneven inde- consider and manage the myriad planned and un- pendence and authority to protect confi- planned future uses of data. dential data used for evidence building. Even with these limitations, the promise of “provable privacy” methods such as differential The Commission found that evidence-based pol- privacy is that the level of risk of re-identification icymaking requires leaders who recognize the can be quantified, thus allowing any dataset to be value of generating more and better evidence as used for evidence building while minimizing pri- a mechanism for improving government services vacy risk. and holding government accountable. Feder- Another implementation challenge is ensuring al leadership that acknowledges and values the that enhanced statistical disclosure methods do importance of evidence building must support not change the data in ways that increase the dif- the use of Federal data assets for statistical pur- ficulty of reproducing research results. The Com- poses. Without sustained leadership and support mission’s guiding principle on humility points within Federal departments for data stewardship, to the need for multiple researchers to replicate opportunities to improve and appropriately lever- results to verify the credibility of evidence. Fur- age data resources for evidence building will be ther, professional journals sometimes require missed. researchers to demonstrate that analyses can be Meaningful collaboration within departments reproduced prior to publication.31 In adopting new is essential if administrative data are to be har- nessed as an integral part of evidence building. 31. For example, the American Journal of Political Science’s For example, the Commission to Eliminate Child “Replication and Verification Policy;” https://ajps.org/ajps- Abuse and Neglect Fatalities concluded in 2016 replication-policy (accessed August 10, 2017). Additionally, the that the inability to coordinate data collection and American Economic Review only publishes papers if the data used in the analysis are documented and available for replication; https:// analysis activities across child protection programs

www.aeaweb.org/journals/policies/data-availability-policy (accessed posed a real risk to children’s lives. One of the August 10, 2017).

OnTheMap: Differential Privacy in Practice at the Census Bureau

OnTheMap is an online mapping and report- which is becoming popular in the statistical ing application that shows areas where peo- disclosure limitation community.”1 Without ple work and where workers live. OnTheMap the use of privacy-preserving technologies, was developed through a partnership between OnTheMap would not be able to provide the the Census Bureau and its Local Employment detail it does. Dynamics partner states. OnTheMap is an example of differential privacy in practice. In “Privacy: Theory meets Practice on the Map,” 1. Ashwin Machanavajjhala, Daniel Kifer, John Abowd, researchers describe how technology was used Johannes Gehrke, and Lars Vilhuber, “Privacy: Theory Meets to create OnTheMap. “The algorithm used to Practice on the Map” (paper presented at the IEEE 24th International Conference on Data Engineering, Cancun, anonymize the data for the… mapping applica- Mexico, April 2008); http://www.cse.psu.edu/~duk17/papers/ tion is known as the synthetic data generation, PrivacyOnTheMap.pdf (accessed August 10, 2017). The Promise of Evidence-Based Policymaking 59

Commission’s recommendations was to “improve evidence about programs and policies. and support data collection about child abuse and The Commission found that PSAs are well posi- neglect fatalities of…children, and integrate the tioned to become department leaders to facilitate data into national databases for analysis, research, the transformation and analysis of administrative and the development of effective prevention strat- data for evidence building. Currently, however, egies.” 32 Coordinating evidence-building activities PSAs vary in readiness to take on greater respon- and supporting use of administrative data across sibility for supporting the larger evidence-build- departments leads to useable inventories, data re- ing community. Some PSAs have strong statutes sources, and analytic capabilities for responding that ensure their independence, place them high to specific policy and research questions. in department hierarchies, and authorize them All Federal departments have a Chief Informa- to protect their confidential data. Other PSAs as- tion Officer (CIO) and a Senior Agency Official for sert their need for independence and authority Privacy (sometimes called a Chief Privacy Offi- through policy and practice but may lack strong cer). According to OMB Circular A-130, CIOs play legal protections. an important role in the management of Federal Lack of independence can be a challenge for information resources. In addition to advising some PSAs. The Federal Information Technology department heads on “the design, development, Acquisition Act (FITARA) of 2014 led department and implementation of information resources,” information technology leadership to consolidate CIOs help prioritize department expenditures on Federal data centers and coordinate information information resources.33 The Senior Agency Offi- technology acquisitions to promote consisten- cial for Privacy has responsibility for “implemen- cy and efficiency.35 These well-intentioned efforts tation of privacy protections; compliance with could hamper the ability of the evidence-building Federal laws, regulations, and policies relating to community to limit access to confidential data. As privacy; [and] management of privacy risks at the Katherine (Kitty) Smith of the Council of Profes- agency.”34 Other senior leaders with influence on sional Associations on Federal Statistics (COPAFS) department information resources include Chief noted in comments to OMB about the roles and Financial Officers and Chief Performance Officers. responsibilities of Federal statistical agencies, “in- The Commission also recommends that depart- formation technology systems that are out of the ments identify a senior official to serve as Chief direct control of the statistical agency can: result in Evaluation Officer (Recommendation 5-1). delays in the retrieval of and dissemination of sta- While each of these officials has a role to play tistical data; impose restrictions on the accompani- in data stewardship, each also has an expansive ment of transparent explanations of methodology portfolio of responsibilities that prevents their with the data; violate the integrity of statistical in- prioritizing data management and secure access formation; and, very importantly, endanger the sta- to data resources for evidence-building activities. tistical agencies’ ability to follow through on their The Commission learned that none of these senior pledges of confidentiality and non-disclosure.”36 leaders is explicitly charged with managing the In public comment, witnesses expressed con- portfolio of department data resources in support cern about independence for PSAs and recommend- of Federal evidence building. Because senior lead- ed that PSAs should have sufficient legal authority ers with a focus on data stewardship for evidence implemented or restored. Margaret Levenstein of building are not at the table when priorities are ICPSR, a consortium of more than 750 academic discussed and resources allocated, departments institutions and research organizations, said that cannot take full advantage of Federal data to build “undermining [the public’s trust in Federal statistical agencies] undermines statistical measurement

32. Commission to Eliminate Child Abuse and Neglect Fatalities, Within Our Reach: A National Strategy To Eliminate Child Abuse and 35. Federal Information Technology Acquisition Reform Act, 40 Neglect Fatalities (Washington, D.C.: Government Printing Office, USC § 11319(b)(1)(A). 2016): 14. 36. Katherine R. Smith (COPAFS) to Katherine K. Wallman, 33. U.S. Office of Management and Budget (OMB), “Managing In- Comment on OMB Statistical Policy Directive, June 18, 2014; formation as a Strategic Resource:” 14. http://www.copafs.org/UserFiles/file/handouts/CommentonOMBSta- 34. OMB, “Managing Information as a Strategic Resource:” 36. tisticalDirective%20.pdf (accessed August 10, 2017). 60 The Promise of Evidence-Based Policymaking

as well as the effectiveness of the programs upon elements of Statistical Policy Directive #1 would which statistics are based.”37 Clyde Tucker of the help PSAs further protect confidential data used American Statistical Association cited a 2015 letter in evidence building. from 20 former PSA heads that said “the independence of a federal statistical agency is a critical ele- Recommendations ment in an agency producing objective and credible statistical data….such autonomy should include Achieving the Commission’s vision for enhancing control over an agency’s planning, budget, press re- privacy while also making better use of data for leases, and information technology.”38 evidence building will require some significant Several commenters asked the Commission to changes. Departments will need to conduct risk reinforce the importance of PSAs and the laws and assessments for public releases of de-identified policies that undergird them. The public’s willing- confidential data and to adopt cutting-edge tech- ness to trust PSAs to protect confidential informa- nology for data security, integrity, and confiden- tion must be maintained and strengthened. tiality. This vision also calls for data stewardship OMB Statistical Policy Directive #1 encourag- coordinated among senior leaders within Federal es Federal departments to recognize the need for departments and the independence of PSAs to integrity and objectivity in Federal statistics (see protect confidential data. The Commission con- box, “Four Responsibilities of Statistical Agencies cluded that the Federal government must improve from OMB Statistical Policy Directive #1”). Poli- data quality, integrity, and security by updating laws cy Directives do not have the same force as stat- to require risk assessments for data releases, embrac- utes, however, so legislation to codify certain key ing cutting-edge technologies, and prioritizing data stewardship and the public trust.

37. Margaret Levenstein, ICPSR, submission to the Commission’s The recommendations below support the Request for Comments. Commission’s vision by allowing secure access to 38. Clyde Tucker, American Statistical Association, Commission confidential data in a privacy-protective manner, Public Hearing, Washington, D.C., October 21, 2016. enabling the accountable production of evidence.

Four Responsibilities of Statistical Agencies from OMB Statistical Policy Directive #1 1

1. Relevance requires “communication tical units must function in an environment across and within departments with plan- that is clearly separate and autonomous ning information collection and dissemina- from the other administrative, regulatory, tion activities” and that statistical agen- law enforcement, or policymaking activities cies collect program and policy-relevant within their respective Departments.” information from “administrative records 4. Thetrust of information providers requires collected and maintained by the agency, or agencies to ensure “confidentiality and other government agencies.” exclusive statistical use” of their data. 2. Credibility requires that agencies “apply sound statistical methods to ensure statistical products are accurate.” 3. Objectivity requires that agencies “pro- 1. OMB, “Statistical Policy Directive No. 1: Fundamental duce data that are impartial, clear, and com- Responsibilities of Federal Statistical Agencies and Recog- nized Statistical Units,” Federal Register 79 (December 2, plete and are readily perceived as such by 2014): 71610; https://www.gpo.gov/fdsys/pkg/FR-2014-12-02/ the public” and further requires that “statis- pdf/2014-28326.pdf (accessed August 10, 2017). The Promise of Evidence-Based Policymaking 61

When taken together with the privacy-protective Figure 9. Improving Confidentiality data access and use recommendations proposed Protections During Public Release of in Chapters 2 and 4, these recommendations seek De-Identified Data to safely harness the potential of data already collected by government to produce more and better evidence in the future. Data Acquired Under Pledge of Confidentiality REC. 3-1: The Congress and the Presi- dent should amend the Privacy Act and the Confidential Information Protec- Statistical Disclosure Limitation Techniques Applied tion and Statistical Efficiency Act (CIPSEA) to require Federal departments to conduct a comprehensive risk assessment on Formal Comprehensive de-identified confidential data intended Risk Assessment Conducted for public release. De-identified confidential data subject to the Privacy Act and CIP- Data Release Approved SEA should be made available only after a disclosure review board (1) approves the release and (2) publicly provides the risk Risk Assessment Posted to assessment and a description of steps tak- Transparency Portal en to mitigate risk.

Public Release Cleared The Commission concludes that existing laws such as the Privacy Act and CIPSEA should require a comprehensive risk assessment and disclo- De-Identiﬁed Conﬁdential sure review board approval before public release Data Released to Public of de-identified confidential data. With a small change in how government approaches the management of confidential data, public data releases can be made much more secure. Many PSAs have of all known sources of risk (see Figure 9). Depart- the expertise to do this now and are a valuable ments should consider the risk of unauthorized resource for program agencies that want to make re-identification in the context of other publicly their data releases more secure. available data. As a starting point, departments Establishing a requirement in law for risk as- should prioritize risk assessments for de-identified sessment provides a basis for the entire Execu- confidential datasets planned for public release tive Branch to more seriously and deliberately and expand risk assessment requirements over approach risk management for publicly releasing time to include all public data releases. data. OMB should collaborate with Federal depart- Departments may choose to go through exist- ments and other stakeholders to establish imple- ing disclosure review boards, such as those in the mentation guidance that requires a risk assess- PSAs, for approval to release de-identified con- ment and approval by a disclosure review board fidential data under the Privacy Act or CIPSEA. of any de-identified data released under the Pri- Other departments may choose to establish a sep- vacy Act or CIPSEA. The Congress should consider arate disclosure review board for this purpose or whether other statutes governing privacy for Fed- establish an entirely new disclosure review board eral confidential data should be subject to a simi- if no such centralized function currently exists. In lar requirement. addition, the National Secure Data Service (NSDS) Under the approach envisioned by the Commis- will establish a disclosure review board to provide sion, Federal departments will conduct compre- review services, extending the capacity of Federal hensive risk assessments that include an analysis departments. 62 The Promise of Evidence-Based Policymaking

Departments should be required to public- Federal government’s lead on the implementa- ly post risk assessments as well as the general tion and application of cutting-edge data manage- steps taken to mitigate risk from public releases ment technologies. OMB should review publicly of de-identified confidential data covered by the available risk assessments to determine if Federal Privacy Act and CIPSEA. Risk assessments should departments are adopting the most advanced dis- be posted to the NSDS transparency portal and in- closure limitation methods, techniques, and tech- clude a mechanism for the public to report infor- nologies. OMB should set standards that require all mation about additional sources of risk (see Rec- agencies to adopt new methods as they are shown ommendation 4-3). Either OMB or a department to be feasible in the Federal context. The National could request the initiation of a new risk assess- Institute of Standards and Technology should con- ment when new information comes to light. tinue its efforts to ensure security protection for Transparency about risk assessments means data exchange, transmission, and storage. that the public will know what the government The evidence-building community should be considered in its assessment of risk when it as- at the forefront of applying cutting-edge tech- sesses risk and why the government now believes nologies to improve data security and privacy. the data can be released as de-identified. Public A National Academies of Sciences, Engineering, risk assessments also make it easy for agencies to and Medicine report recently recommended that learn from one another about the implementation “Federal statistical agencies should adopt modern of enhanced statistical disclosure limitation tech- database, cryptography, privacy-preserving, and niques. privacy-enhancing technologies.”39 The Commis- sion’s Recommendation 3-2 echoes and extends REC. 3-2: The President should direct the report’s recommendation. Standards govern- Federal departments, in coordination ing data security for all Federal agencies should with the National Secure Data Service, to be adapted and strengthened as new technologies emerge that provide greater privacy protections. adopt state-of-the-art database, cryptog- Leaders within Federal departments should ac- raphy, privacy-preserving, and privacy-en- knowledge and adhere to special data stewardship hancing technologies for confidential data requirements for confidential data used for evi- used for evidence building. dence building.

To implement this recommendation, the Com- REC. 3-3: The President should direct mission proposes that the NSDS should have the Federal departments to assign a initial and generative role in this enhancement of senior official the responsibility for coor- privacy protections. The NSDS should lead the implementation and application of new privacy-re- dinating access to and stewardship of the lated technologies in the Federal evidence-build- department’s data resources for evidence ing community. OMB should consult with the building in collaboration with senior National Institute of Standards and Technology department information technology, pri- on the most appropriate privacy-protective data vacy, and other leaders. A Principal Statis- exchange standard for use across Federal depart- tical Agency head, or other appropriately ments and Federally funded projects in support of qualified senior official, should serve this evidence-building, consistent with or useful for function. operational or administrative data exchange standards. Finally, this recommendation requires the Each department should assign to a senior offi- active participation of all Federal departments, as cial the duties of Senior Agency Official for Data they will need to periodically review their compli- Policy and the responsibility for coordinating the ance with the National Institute of Standards and Technology standards for the security of confidential data in transmission and storage. 39. National Academies of Sciences, Engineering, and Medicine. Innovations in Federal Statistics: Combining Data Sources While The proposal to establish the NSDS in statute Protecting Privacy (Washington, D.C.: National Academies Press, should include authority for the NSDS to be the 2017): 93. The Promise of Evidence-Based Policymaking 63

department’s statistical needs and information serve as the department’s liaison to the NSDS and policy in support of evidence building. advise the NSDS, other Federal departments, and The person assigned this responsibility would OMB about the best way to use existing depart- need to have the technical and substantive exper- ment data beneficially for exclusively statistical tise to be effective in promoting secure evidence purposes. building within the department. Because of the In many cases, departments would not have imperative to protect confidential data, the desig- to create a new position or hire a new person to nated senior leader would need extensive knowl- fill the role. Because of the expertise required, in edge of data confidentiality laws, policies, and most cases the head of a PSA should fill the po- practices including experience with applying sta- sition. In other cases, or in departments that do tistical disclosure limitation techniques in public not have a PSA, a senior official with expertise releases of data. Other necessary technical skills in evaluation or policy-relevant research may be would include knowledge of strategies to collect qualified to provide such leadership. Regardless, and manage data used for evidence building, data collaborative leadership around the coordination analysis methods appropriate to data used for ev- and management of data resources within a de- idence building, and emerging strategies to pro- partment is important for the efficient production vide secure access to confidential data in a manner of evidence. useful for analysis. In addition, this senior official would need a connection to and understanding REC. 3-4: The Congress and the Presi- of the broader evidence-building community, in- dent should enact legislation to codi- cluding a deep appreciation for the importance fy relevant portions of Office of Manage- of keeping Federal data authorized for evidence building on the exclusively statistical purposes ment and Budget Statistical Policy Directive side of the functional wall of separation described #1 to protect public trust by ensuring that in Chapter 1. data acquired under a pledge of confidenti- The Commission believes that this function is ality are kept confidential and used exclu- essential as the Federal government increasingly sively for statistical purposes. uses its existing data resources to generate valid evidence. Today, leadership for evidence building The Congress and the President should enact is fragmented in or absent from many depart- legislation to codify those elements of Statisti- ments. The senior data policy official would be cal Policy Directive #1 related to supporting “the charged to anticipate that all of the department’s quality and objectivity of Federal statistical in- data may be assets for evidence building. They also formation.”40 OMB issued the directive in 2014 would be charged to ensure that legal, policy, and to affirm the long-acknowledged, fundamental technical data stewardship requirements are met responsibilities of Federal statistical agencies and when using data for evidence building. recognized statistical units in the design, collec- Senior Agency Officials for Data Policy would tion, processing, editing, compilation, storage, have both inward- and outward-facing roles. analysis, release, and dissemination of statistical Within their own departments, the senior data data and information. The Commission recom- policy officials would: (1) collaborate with oth- mends that the requirements embodied in the er senior department leaders in information directive be embedded in law. The Congress and technology, finance, privacy, and evaluation to the President should provide each PSA with the prioritize the use of data for evidence building; authority and institutional support necessary for (2) ensure that program and statistical agencies ensuring confidentiality and maintaining the in- determine the sensitivity of data resources and tegrity and objectivity of Federal statistics. establish appropriate access controls (see Rec- Recommendation 3-4 further strengthens ommendation 2-8); and (3) promote the use of statistical institutions by placing in law the ex- enhanced statistical disclosure limitations tech- pectations of independence outlined in OMB’s niques to protect confidentiality and expand access for evidence building. In their external coordination role, senior data policy officials would 40. OMB, “Statistical Policy Directive No. 1.” 64 The Promise of Evidence-Based Policymaking

Statistical Policy Directive #1. The placement employees established in Title 5 of the U.S. Code.41 of these responsibilities explicitly in statute Placing the directive in law also provides PSAs a strengthens the government’s commitment to stronger basis from which to defend their need for the policies, makes the principles more difficult independent information technology resources to change, and increases the likelihood that vi- and tailored procedures to secure the confidenti- olations of the responsibilities would be report- ality of Federal data used for evidence building. ■ ed under whistleblower protections for Federal 41. Prohibited Personnel Practices, 5 USC § 2302. The Promise of Evidence-Based Policymaking 65

4 Modernizing America’s Data Infrastructure for Accountability and Privacy

Empower government to develop state-of-the-art capacity to securely combine existing data and provide secure data access for exclusively statistical purposes in a privacy-protective and transparent way.

Recommendations

4-1: The National Secure Data Service the NSDS should maintain a searchable in- (NSDS) should be established as a sepa- ventory of approved projects using confiden- rate entity in the Department of Commerce tial data and undergo regular auditing of that builds upon and enhances existing ex- compliance with rules governing privacy, pertise and infrastructure in the Federal gov- confidentiality, and access. ernment, especially at the Census Bureau, to ensure sufficient capacity in secure record 4-4: The NSDS should have specific ad- linkage and data access for evidence build- ministrative and implementation flexi- ing. bilities including the ability to leverage public-private partnerships and to collect and 4-2: The NSDS should establish a Steer- retain user fees. ing Committee that includes representatives of the public, Federal departments, 4-5: The Office of Management and state agencies, and academia. Budget should increase efforts to make information available on existing Federal 4-3: To ensure exemplary transparency datasets including data inventories, metada- and accountability for the Federal gov- ta, and data documentation in a searchable ernment’s use of data for evidence building, format. 66 The Promise of Evidence-Based Policymaking

he Federal evidence-building community • A developer and implementer of new and Tneeds a modern institutional infrastructure secure approaches to data access and analysis. that facilitates evidence building and incorporates The NSDS will be designated as a Principal Sta- strong privacy protections. Today, some elements tistical Agency (PSA) with an exclusively sta- of that infrastructure exist within individual si- tistical mission, with the authority to protect los, focused on particular agency topics or tasks. data under the Confidential Information Pro- Achieving the Commission’s vision of routine, se- tection and Statistical Efficiency Act (CIPSEA) cure evidence production for the public good re- and the resources and capacity to implement quires a new infrastructure aiding Federal depart- emerging privacy-protective technologies. ments and outside researchers by improving the amount and quality of evidence while enhancing The Congress and the President specifically privacy. charged the Commission to consider whether the While the risk assessment described in Chapter country needed a data “clearinghouse.”1 While the 3 (Recommendation 3-1) was designed for pub- term was not defined in the Commission’s statute, lic releases of de-identified confidential data, the “clearinghouse” evokes images of past proposals recommendations in this chapter seek to promote to consolidate large amounts of confidential data evidence generation possible only by securely har- into a central location. For example, in the 1960s, nessing the value of detailed data that cannot be the Kaysen Committee proposed creating a na- made publicly available, and only for exclusively tional Data Bank to address the decentralized na- statistical purposes. ture of the Federal Statistical System and the Fed- This chapter expands on Recommendation eral government’s data infrastructure. The Kaysen 2-1 to establish the National Secure Data Service Committee determined that decentralization (NSDS) by describing several of the implementa- prohibited effective use of socioeconomic data by tion steps necessary to make the envisioned ser- researchers both inside and outside government, vice possible. The NSDS described in Chapter 2 is and recommended the creation of a large ware- intended to be: house of data about the American public.2 While the Commission on Evidence-Based Policymaking • A service for qualified researchers seeking came to a similar conclusion about the value of approval to securely analyze confidential data, it concluded that setting up a data warehouse government data for exclusively statistical of the sort envisioned by the Kaysen Committee purposes. The NSDS will apply data minimi- would create an attractive target for misuse of zation techniques to protect combined data, private data. The Commission on Evidence-Based require researchers to complete training, and Policymaking opposes the creation of any such subject researchers to stringent monitoring for “clearinghouse” and proposes an entirely differ- adherence to privacy-protective protocols. ent solution for addressing the challenges of data access. (See the box “How is the National Secure • A center for unparalleled transparency about Data Service Different from the Data Bank?”.) government uses of confidential data for Technology and privacy-protective approaches exclusively statistical purposes. The NSDS will have advanced greatly over the past 50 years since make information about data sensitivity, risk the Data Bank proposal was rejected, and they assessments for public release of de-iden- will continue to evolve. New approaches to secure tified confidential data, and individual evi- access to data for evidence building feature stron- dence-building projects available in ways that ger privacy protections than at any point in histo- exceed previous transparency and accountabil- ry. The Commission’s approach to implementing ity for evidence building in the United States.

• A developer and implementer of state-of-the- 1. Evidence-Based Policymaking Commission Act of 2016 (Public art methods to safely combine confidential Law 114–140, Section 4(b), March 30, 2016). data. The NSDS will temporarily and securely 2. Rebecca S. Kraus, “Statistical Déjà Vu: The National Data Cen- combine data and create analysis files with the ter Proposal of 1965 and Its Descendants” (paper presented at the Joint Statistical Meetings, Miami Beach, FL, August 1, 2011); minimum amount of data needed for approved https://www.census.gov/history/pdf/kraus-natdatacenter.pdf (ac- projects with exclusively statistical purposes. cessed August 10, 2017). The Promise of Evidence-Based Policymaking 67

How is the National Secure Data Service Different from the Data Bank?

While both the 1960s Data Bank and the Na- remove personal identifiers before the data can tional Secure Data Service (NSDS) proposals be analyzed for exclusively statistical purposes. seek better use of government data for statis- The NSDS takes privacy to a new level by apply- tical activities, that is where their similarity ing state-of-the-art protections and ensuring ends. The Data Bank was designed as a massive that, at every step, the American public is noti- warehouse of constantly cumulating govern- fied about the uses of data to hold government ment-held data about the American public that accountable on a project-by-project basis. The could have been used for diverse purposes. The NSDS proposal requires a privacy audit for pub- NSDS instead is a service that brings together lic transparency and accountability. as little data as possible for as little time as pos- In short, the Commission does not recom- sible for exclusively statistical purposes. The mend a single “clearinghouse” that collects NSDS is not intended to store data and users of massive stores of data to be warehoused in- the service would not be permitted to conduct definitely, as was envisioned in the 1960s Data analyses for non-statistical purposes under the Bank proposal. In fact, the Commission con- Commission’s proposed legal and operational cluded that a single national “clearinghouse” framework. with all types of data is simply not necessary, While the Data Bank proposal would have nor does it represent best practices in privacy stored individual-level data, the proposed protection. NSDS is designed to temporarily link data and

Key Similarities and Differences Between the Data Bank and the NSDS

1960s 2017 Proposal Feature Data NSDS Bank

Limits data uses to exclusively statistical purposes

Allows public participation in setting policies and priorities

Intended to improve researcher access to confidential data Restricts access to data with personally identifiable information (PII) and keeps access temporary Approves projects individually

Includes an appeals process for project approval

Includes a publicly available inventory of approved projects

Implements state-of-the-art privacy protection techniques

Makes transparent all data access for statistical purposes

Includes an auditing function 68 The Promise of Evidence-Based Policymaking

Recommendation 2-1, elaborated upon in this cies abide by legal requirements for transparency chapter, aims to take full advantage of the oppor- about data systems, but have made only limited tunity created by these new approaches. attempts at public transparency about individu- However, the Federal government currently al projects using confidential data for exclusive- lacks the complete infrastructure needed to rou- ly statistical purposes. Finally, the lack of a com- tinely, efficiently, and securely generate evidence prehensive inventory of data useful for evidence about government policies and programs using building, lack of complete and accurate technical confidential data. The infrastructure that does documentation for many government datasets, exist is not designed to meet government-wide and inconsistent definitions of key concepts needs; does not have the capacity to meet expand- across datasets inhibits routine evidence building. ed demand from inside and outside government; is uneven in its application of best practice tech- Challenge #1: Existing Federal government nologies and techniques for secure data access, privacy-protective data linkage, access, linkage, and analysis; and does not always prioritize public transparency and accountability. and analysis capabilities do not meet gov- This chapter includes recommendations that ernment-wide needs for evidence. elaborate on aspects of NSDS’s implementation and build on a vision of the NSDS as a source of The Commission received testimony from Federal important shared services to improve govern- agencies using state-of-the-art methods to pro- ment-wide data infrastructure for evidence build- vide privacy-protective access to securely linked ing. The Commission envisions that NSDS will confidential data for exclusively statistical pur- help government increase the appropriate use poses. Within the Federal government, pockets of of data for exclusively statistical purposes, while excellence can be found for most of the statistical keeping confidential data safe and secure. NSDS activities needed to routinely generate evidence. will have the capacity and capability to support Some Federal departments are using state-of- secure analysis of single or combined confiden- the-art methods for secure data linkage, access, tial, restricted-use datasets while minimizing the and analysis in response to their specific statuto- risks of privacy harm from evidence-building ac- ry missions. These departments have expert staff tivities. Given the breadth of the existing gaps in with experience implementing privacy-protective infrastructure, the Commission recommends the techniques in a Federal context. This expertise establishment of NSDS to address multiple capac- constitutes a valuable resource for the Federal ity needs simultaneously to usher in a new era of evidence-building community, but it is currently evidence building in the United States. concentrated in different agency silos and focused on specific agency purposes. There are pockets of Findings excellence; there is no center of excellence capable of serving government-wide needs for priva- The Commission learned from witnesses and sub- cy-protective data linkage, access, and analysis for mitted statements about gaps and unevenness in evidence building. infrastructure supporting evidence building. The A great deal of expertise relevant to Federal findings presented in this chapter highlight four evidence building resides in the U.S. Department challenges to routine and secure evidence gen- of Commerce. The U.S. Census Bureau has built eration to improve government policies and pro- the capacity to securely link demographic data grams. These findings relate specifically to the use for exclusively statistical purposes in support of of confidential government data for exclusively Census Bureau operations and demographic and statistical purposes. First, the Federal government socioeconomic research (see the box “Center for currently has pockets of expertise in privacy-pro- Administrative Records Research and Applica- tective data linkage, access, and analysis, but this tions”). The Census Bureau’s Center for Econom- capacity does not meet government-wide needs ic Studies (CES) links and archives business and for evidence. Second, the identification and adop- economic data used for research and analysis. The tion of new technologies that can increase data Census Bureau also coordinates a network of Fed- access while enhancing privacy is slow and un- eral Statistical Research Data Centers (FSRDCs) to even across Federal agencies. Third, Federal agen- provide secure access to confidential data (see the The Promise of Evidence-Based Policymaking 69

Center for Administrative Records Research and Applications

The Center for Administrative Records Research organizations; and directs Census to seek out and Applications (CARRA) at the U.S. Census this information instead of conducting direct Bureau strategically uses administrative data inquiries. Further, the Census Bureau has a from Federal, state, and third party providers specific exemption under the Privacy Act that for exclusively statistical activities in support of allows Federal agencies to disclose records to the Census Bureau’s operations and demograph- the Census Bureau without prior written con- ic and socioeconomic research. Linked data are sent of the individual if it is for the purposes of also used by researchers at the Census Bureau’s planning or carrying out a census or survey or Center for Economic Studies and through the related activity. Federal Statistical Research Data Centers. Census is authorized to access Federal Tax As of 2017, CARRA’s portfolio includes 12 Information for individuals and businesses pilot research and evaluation projects span- under Title 26 (IRS statute). CIPSEA authoriz- ning housing, health, welfare, education, and es the Census Bureau, the U.S. Bureau of La- labor. The initiative also includes six pilot proj- bor Statistics, and the U.S. Bureau of Economic ects in partnership with Chapin Hall at the Analysis to share business data for exclusively University of Chicago that combine state and statistical purposes. While these authorities local administrative data with information ac- permit the Census Bureau to access many dif- cessible to the Census Bureau. Existing CARRA ferent datasets, in practice data sharing agree- partnerships with experts in academia support ments are negotiated separately and governed CARRA’s ongoing efforts to improve capabili- by applicable laws and policies. ties for securely combining confidential data. Under Title 13, CARRA can establish agree- CARRA benefits from a number of legal ments to access data held by Federal agencies, authorities applicable to the Census Bureau. states, and some other jurisdictions for exclu- Foremost, the Census Bureau has broad au- sively statistical purposes. The list below high- thority under Title 13 (Census statute) to ac- lights a portion of the data CARRA currently acquire administrative records. Title 13 authoriz- quires to support the Census Bureau’s mission.1 es the Census Bureau to use records previously State administrative records not only vary on collected by other Federal agencies and state, the number of states participating, but also in tribal, or local governments, as well as private the years of data available.

Federal administrative data State administrative data • Medicaid Statistical Information System • Child Care Development Fund (1 state) • Office of Personnel Management • Low Income Home Energy Assistance Personnel System Program (1 state) • Public and Indian Housing Information Center • Special Supplemental Nutrition Program • Selective Service System for Women, Infants, and Children (6 states) • Small Business Administration Loan • Supplemental Nutrition Assistance Guarantee Program Program (15 states) • Social Security Old Age, Survivors, and • Temporary Assistance for Needy Families Disability Insurance (9 states) • Tax Information • Unemployment Insurance quarterly wages (48 states) • United States Postal Service Change of Address

1. U.S. Census Bureau, “Administrative Data Inventory” as of July 18, 2017; https://www2.census.gov/about/linkage/data-file- inventory.pdf (accessed August 10, 2017). 70 The Promise of Evidence-Based Policymaking

box “Federal Examples of Secure Access to Con- 49) of other responding offices have provided oth- fidential Data”). In addition, the U.S. Bureau of er Federal agencies access to data they collect.3 All Economic Analysis, a PSA that produces a broad PSAs have external researchers who access data, range of economic data; the U.S. National Oceanic while the same is true of 75 percent of respond- and Atmospheric Administration, an agency with ing evaluation offices and 31 percent of other re- many scientific and technical data resources; and sponding offices. the National Institute of Standards and Technolo- The Commission considered whether this gy are all part of Commerce. existing infrastructure could, with little new PSAs, and some program agencies, provide se- additional authority or resources, meet govern- cure access to their confidential data for statistical ment-wide needs for evidence building. Much of purposes authorized by law (see examples in the the government’s staff expertise on securely link- box “Federal Examples of Secure Access to Confi- ing confidential data resides at the Census Bureau, dential Data”). The CEP Survey of Federal Offices found that 10 out of 13 PSAs, 5 out of 8 respond- 3. Two of the 10 responding evaluation offices and 18 of the 67 ing evaluation offices, and 39 percent (19 out of responding other offices reported that they do not collect data.

Federal Examples of Secure Access to Confidential Data

Administrative Data Research Facility: The data related to education in the United States Administrative Data Research Facility is a pilot and other nations. NCES makes detailed data project that enables secure access to analytical available to external researchers through the tools, data storage and discovery services, and IES restricted-use data licensing system. NCES general computing resources for users, includ- loans researchers restricted-use data through a ing Federal, state, and local government ana- license (contract) among IES, the user, and the lysts and academic researchers. The Census user’s organization (e.g., university, research Bureau and academic partners developed the institution, or company). Licensed users agree project as part of the collaborative Training to the terms of the IES confidentiality law; IES Program in Applied Data Analytics sponsored performs regular audits to confirm compliance by the University of Chicago, New York Univer- with required procedures. Because of this audit sity, and the University of Maryland.1 It is cur- requirement, IES will only loan restricted-use rently operating as a pilot with users accessing data to qualified organizations in the 50 States the Facility as part of the training program. The and Washington, D.C. Researchers seeking a Facility operates as a cloud-based computing data license must complete an online formal environment, with Federal security approvals, request and research proposal.2 which currently hosts selected confidential Federal Statistical Research Data Cen- data from the U.S. Department of Housing and ters: The Federal Statistical Research Data Urban Development and the Census Bureau, as Centers (FSRDCs) are partnerships between well as state, city, and county agencies, and an Federal statistical agencies and leading re- array of public use data. search institutions. They are secure facilities Data Licensing System: The National Cen- housed in partner institutions and managed ter for Education Statistics (NCES), one of the by the U.S. Census Bureau to provide secure Principal Statistical Agencies (PSAs), is part of access to a range of Federal restricted-use mi- the Institute of Education Sciences (IES) in the crodata for statistical purposes only. All FSRDC U.S. Department of Education. It is the prima- –continues ry Federal entity for collecting and analyzing

2. U.S. Department of Education, National Center for Educa- tion Statistics, “Learning Procedures: Frequently Asked Ques- 1. “Training Program in Applied Data Analytics;” http://www. tions;” http://nces.ed.gov/statprog/instruct_licensing_faq.asp (ac- applieddataanalytics.org (accessed August 10, 2017). cessed August 10, 2017). The Promise of Evidence-Based Policymaking 71

and particularly in the Center for Administrative at the Census Bureau” in Chapter 3), but no one Records Research and Applications (CARRA). As agency today serves as a model of best-practice currently constituted, however, CARRA cannot application of cutting edge privacy-protecting meet all of the project linkage needs of other Fed- technologies. eral departments and qualified external research- The Commission’s public meetings, hearings, ers. It has neither the authority in law nor the Request for Comments, and Survey of Federal Of- capacity to do so. The FSRDCs and PSAs provide fices yielded numerous examples of unintentional secure access to confidential data; none of them barriers to evidence building erected due to unclear has the authority or capacity to provide expand- legal frameworks as described in Chapter 2. Other ed access for increased evidence generation gov- barriers arise because of uneven capacity across ernment wide. Some individual Federal projects Federal departments. Some Federal departments at the Census Bureau and elsewhere are piloting lack PSAs; some evaluation and policy research new technologies for secure data analysis (see the offices lack technical capacity; and some program box “OnTheMap: Differential Privacy in Practice agencies fail to prioritize evidence building.

Federal Examples of Secure Access to Confidential Data— continued

researchers must obtain Census Bureau Special years, resources permitting.3 Sworn Status. FSRDC data enclaves meet the Virtual Research Data Center: The Cen- physical and environmental protections re- ters for Medicare and Medicaid Services (CMS) quired by Federal statistical agencies for use of is responsible for administering the Medicare, restricted-access microdata, including Federal Medicaid, and Children’s Health Insurance Tax Information. FSRDC researchers can col- programs. To support these functions, CMS laborate with other FSRDC researchers across collects, generates, and stores financial, health the United States through the secure comput- care, and other sensitive information. With ap- ing environment. propriate safeguards, CMS provides access to Joint Statistical Research Program: The microdata for statistical and research uses to Statistics of Income Division at the Internal Federal agencies, Federal grantees, and other Revenue Service collects and processes tax data approved researchers on encrypted data files and provides access to tax microdata to those provided on external media or through a Vir- users authorized under statute—including tual Research Data Center. Through the virtual certain government agencies—for tax policy data center, approved researchers can access analysis, research, and statistical purposes. and perform their own analysis and manip- In addition, the Statistics of Income Division ulation of CMS data virtually from their own makes microdata available to researchers workstations. The virtual data center helps through its Joint Statistical Research Program. CMS meet additional demand from data users The goals of the program are to (1) provide new while also ensuring data privacy and security. insights and advance the understanding of the It also provides researchers with more timely ways existing tax policies affect individuals, access to data in a more cost-effective manner.4 businesses, and the economy, (2) suggest tax policy solutions to advance the common good, and (3) provide new understanding of taxpayer 3. Internal Revenue Service, Statistics of Income, “Joint Statis- behavior that could impact the administration tical Research Program Call for Proposals;” https://www.irs.gov/ of the U.S. tax system. Applications for the pub/irs-soi/16jsrpapplication.pdf (accessed August 10, 2017). program are accepted approximately every two 4. Centers for Medicare and Medicaid Services (CMS), “Virtual Research Data Center (VRDC);” http://www. resdac.org/cms-data/request/cms-virtual-research-data-center (accessed August 10, 2017). 72 The Promise of Evidence-Based Policymaking

Some states and local jurisdictions told the and burden for local entities due to the expertise Commission that basic technical services could required and may [disincentivize] research part- substantially improve their capacity to support ev- nerships. A central service could lend consistent idence-based policymaking about their own pro- technical expertise to the task of data matching.”5 grams and policies as well as those implemented Such technical services could be useful to Federal in partnership with the Federal government.4 At a departments facing capacity challenges as well. Commission public hearing in February of 2017, CARRA’s approach to using statistical and ad- Joy Bonaguro, Chief Data Officer for the City and ministrative data for statistical activities over the County of San Francisco, testified that “matching past several years demonstrates the possibility and and linking data across systems can be a challenge potential of making better use of these data for

4. Communication with National Association of State Workforce Agencies Labor Market Information Committee, Chicago, IL, April 5. Joy Bonaguro, City and County of San Francisco, Commission 20, 2017. Public Hearing, Washington, D.C., February 9, 2017.

Examples of Secure Access to Confidential Data in Other Countries

Administrative Data Research Network Research Data and Service Centre (United Kingdom): The Administrative Data (Germany): The German Bundesbank compiles Research Network is a national partnership data for monetary, financial, and external sta- in the United Kingdom that provides trained tistics and produces a comprehensive set of social and economic researchers access to indicators under its legal mandate to develop de-identified, linked administrative data in a a broad spectrum of user-oriented economic secure environment. The Network was founded data. The Bundesbank established the Research in 2013 by the Economic and Social Research Data and Service Centre to facilitate data link- Council—an independent, publicly funded en- age and researcher access. The Bundesbank’s tity—to allow more use of government data program provides visiting researchers with in social and economic research, with safety on-site access to microdata on banks, securi- measures in place to protect sensitive data. A ties, investment funds, enterprises, and house- goal of the Network’s work is to develop high holds. Researchers seeking to access Bundes- standards for sharing, linking, and matching bank data must submit an application that records securely and consistently. The Network includes a description of the project, its hy- continues to research data linkage methodol- potheses and methods, and a justification for ogies and evaluate new technologies. It also the datasets required. Staff at the Bundesbank’s recognizes that public trust through public program work with researchers in advance of engagement is the most important element their application to help them refine their re- for success. The Network’s processes were de- search questions and identify the data sources signed with a great deal of public consultation. they will need.2 The Network’s public engagement strategy is Research Services Division (Denmark): designed to promote sustainability through Statistics Denmark, under the Ministry for Eco- transparency and public understanding.1 nomic and Interior Affairs, is responsible for creating official statistics on Danish society. The –continues 1. United Kingdom, Administrative Data Research Network, “Communication and Public Engagement;” https://adrn.ac.uk/ 2. Germany, Bundesbank, “Research Data and Service about/network/ads/communication-and-public-engagement (ac- Centre;” http://www.bundesbank.de/Navigation/EN/Bundesbank/ cessed August 10, 2017). Research/RDSC/rdsc.html (accessed August 10, 2017). The Promise of Evidence-Based Policymaking 73

government-wide evidence building. That CARRA sure limitation techniques to protect confidential safely and confidentially links data for exclusively data and provide highly restricted access to con- statistical purposes under the law without priva- fidential data for exclusively statistical purposes cy breaches demonstrates this can be done well (see Chapter 3). when appropriate authorities and privacy protec- Some of today’s most innovative and informa- tions are in place. tive evidence-building projects rely on highly restricted access for researchers to confidential data. Challenge #2: Identification and adoption Examples include (1) research on intergeneration- al mobility by race and ethnicity using tax and of emerging technologies and techniques survey data,6 (2) assessments of the long-term ef- for privacy-protective data access, link- fects of antipoverty policies that connect welfare age, and analysis are too slow and uneven data with earnings records,7 and (3) development in government to effectively support evi-

dence building. 6. Raj Chetty, Nathaniel Hendren, Patrick Kline, and Emmanuel Saez, “Where is the Land of Opportunity? The Geography of In- As described in Challenge #1, few Federal agencies tergenerational Mobility in the United States,” National Bureau of are providing privacy-protective access to confi- Economic Research (NBER) Working Paper 19843 (January 2014); dential data for exclusively statistical purposes, http://www.nber.org/papers/w19843 (accessed August 10, 2017). and even fewer are securely linking confidential 7. For example, “Experimental Estimates of the Long-Run Impacts of Welfare Reform on Participants and Their Children” and on- data for evidence building. All PSAs and a few oth- going work by Hilary Hoynes, Jordan Matsudaira, Pauline Leung, er Federal agencies use rigorous statistical disclo- and Zhuan Pei.

Examples of Secure Access to Confidential Data in Other Countries—continued Research Services Division of Statistics Den- d’Accès Sécurisé aux Données, or CASD) was mark provides researcher access to linked data founded in 2009. One distinct feature of this with identifiers removed for statistical purposes system is a secure device, the SD-Box, that al- only. Since 1968, every person in Denmark has lows researchers to remotely access and analyze received a Personal Identification Number. Each confidential data from CASD’s servers. The SD- person is also associated with a dwelling identi- Box is installed at approved researchers’ own in- fication and possibly an enterprise or business stitutions. With their access card and fingerprint identification. These three keys can be used to confirmation, researchers access everything link multiple sources of administrative and sur- they need at their local workstation—including vey data for individuals and family units over data, statistical software, storage, and comput- time. Researchers affiliated with an approved ing power—when, in fact, all of these are on Danish research environment may submit an the secure servers inside CASD. CASD remote- application to the Research Services Division ly authenticates, monitors, and configures the to access Statistics Denmark data. Data may be SD-Box and all communications between it and accessed only through authorized research en- CASD servers take place through encrypted tun- vironments, which include universities, sector nels. Because there is centralized security man- research institutes, ministries, and non-profit agement, CASD can push upgrades and patches foundations.3 to each device. SD-Boxes can also be remotely Secure Data Access Centre (France): disabled. All SD-Boxes are identical and easily France’s Secure Data Access Centre (the Centre replaced in the event of an equipment failure.4

3. Denmark, Statistics Denmark Research Services Division, “Data for Research;” http://www.dst.dk/en/TilSalg/ 4. France, “Secure Data Access Centre;” https://casd.eu/en Forskningsservice (accessed August 10, 2017). (accessed August 10, 2017). 74 The Promise of Evidence-Based Policymaking

of improved measures of poverty and the income However, the absence of past breaches does not distribution that connect survey, tax, and pro- mean future breaches are impossible, particularly gram data.8 Such projects can only be conducted if as technologies evolve or bad actors focus on linkages among datasets can be made in an envi- different types of data. ronment in which privacy and confidentiality are As described in Chapters 2 and 3, a legal in- protected. frastructure must exist to enable secure data access for evidence building and to take advantage “The collection of accurate of new privacy-protective technologies. Among and complete information… those agencies providing secure access to confidential data, some have the expertise and resourc- is predicated on trust…Without es to identify and implement feasible new tech- that trust, data collection will nologies that hold the promise of allowing more be undermined before it even evidence generation in more secure ways (see the begins…” 9 box “Emerging Approaches That Enhance Privacy Protections” in Chapter 3). While these priva- – Thomas L. Mesenbourg, Former U.S. Census cy-protective approaches are in use today, they Bureau Deputy Director are not available at the scale necessary to enhance Today, PSAs and a few other Federal agencies evidence production across government. The Cen- that provide secure access to confidential data for sus Bureau’s capacity, for example, is limited to a statistical purposes have a number of protections handful of projects, with new cutting-edge tech- in place as required by statute and policy. nologies limited to even fewer applications. Federal staff and outside researchers accessing The Commission heard from several other coun- confidential data must complete annual training tries about models for using new technologies for in information technology security and data secure data access (see the box “Examples of Secure stewardship. Agencies restrict access to person Access to Confidential Data in Other Countries”). identifiers such as name and date of birth to a Today, the Federal evidence-building commu- very few technical staff under tightly controlled nity lacks a coordinated focus on harnessing new conditions. Other staff and outside researchers technologies to provide more secure data access for almost never receive approval to access personal exclusively statistical purposes. The government’s identifiers. Any statistical research output that capacity to take advantage of emerging privacy-pro- uses confidential data undergoes statistical tective technologies is spread across a number of disclosure review before public release to ensure different agencies. The focus of individual agencies confidentiality is not compromised. During the is rightfully on their own missions and not orient- Commission’s December meeting, government ed towards government-wide evidence building. officials, including from the Census Bureau No one agency has sufficient resources and staff and the Statistics of Income Division in the expertise to develop and implement state-of-the- Internal Revenue Service, indicated that to their art technologies or to push the evidence-building knowledge no breach of their confidential data community to continually innovate to address po- used for evidence building had ever occurred.10 tential threats and risks to the American public’s confidential data. Achieving the Commission’s vision means addressing these challenges. 8. Bruce D. Meyer and Nikolas Mittag, “Using Linked Survey and Administrative Data to Better Measure Income: Implications for Poverty, Program Effectiveness, and Holes in the Safety Net,” Challenge #3: The existing infrastructure NBER Working Paper 21676 (October 2015); http://www.nber.org/ for accessing, linking, and analyzing confi- papers/w21676 (accessed August 10, 2017). dential data for evidence building does not 9. Thomas L. Mesenbourg, “How Statutory and Regulatory Chang- es May Create New and Unintended Challenges for Statistical always prioritize state-of-the-art transpar- Agencies” (paper presented at the International Society of Scien- ency and oversight. tometrics and Informetrics Conference, Durban, South Africa, July 2011). From collecting data to generating evidence about 10. Barry Johnson, Internal Revenue Service, and Ron Jarmin, Census Bureau, Commission Meeting, Washington, D.C., Decem- government programs and policies, members of ber 12, 2016. the Federal evidence-building community should The Promise of Evidence-Based Policymaking 75

tell the American people how their information is fidential information from the American people. being used and for what benefit to society. Trans- To maintain the public’s trust, the evidence-build- parency means giving the public information ing community must take reasonable steps to en- about how the government is using their data to sure the public is aware of how data are collected, improve its effectiveness and efficiency. The Fed- safeguarded, shared, and used, and what benefits eral evidence-building community also should es- those uses may provide for society. These are long tablish oversight and accountability mechanisms recognized aspects of transparency for evidence to verify that it is using best practice procedures building. However, the Commission finds that the to protect personal information. Federal evidence-building community today does For data to be useful for evidence building, they not consistently nor effectively use best practice must be complete and accurate. The cooperation transparency processes for public accountability. of the American people is central to the quality During the Commission’s September 2016 and accuracy of survey and administrative data meeting, then-Senior Advisor for Privacy at the used for evidence building. Telling the American U.S. Office of Management and Budget (OMB) and people about how the government is using their Chair of the Federal Privacy Council, Marc Gro- data to improve accountability and serve the pub- man, emphasized the importance of transparency: lic is ethical and conforms to the Fair Information Practice Principles (FIPPs) on transparency and [As] the United States Government accountability and auditing described in Chapter we have an obligation to be trans- 3. Only with such transparency and accountabil- parent to our citizens about what ity can the public trust needed to collect and use we collect and how we use it. We personal data for evidence building be preserved. make representations about use and Further, for evidence to make a difference in we have to honor those representa- government effectiveness and efficiency, policy- tions...[by explaining] how the data makers, program administrators, and the public collect[ed] will be used. And that is must trust the results. Without trust, the results a core principle of the Privacy Act or 11 will lack credibility. Without credibility, the results other laws and of privacy generally. are more likely to be ignored, wasting the time and money used to generate them. Even worse, lack of The Privacy Act requires a System of Records credibility could jeopardize the goal of generating Notice to promote public transparency. Govern- more and better evidence to improve government ment agencies issue public notices when a “sys- and monitor the state of the nation. Trust is criti- tem of records” containing personally identifiable cal to evidence-based policymaking. information is being created or expanded. These The tradition and culture of data stewardship public notices disclose what information is col- in some parts of the Federal evidence-building lected and for what purpose. However, the notices community today should inform the efforts of all are made available in a way that can be difficult for Federal departments that provide secure access to the public to access; members of the public may confidential data. The PSAs responsible for much not know where to find them or understand the of the current data infrastructure operate within technical information they contain. While Gro- a framework of laws, regulations, policies, and man discounted the public’s interest in the notic- practices that emphasize the importance of re- es given this challenge, he added: “I can tell you specting individuals’ privacy and protecting the that advocacy groups do read it; [inspectors gen- confidentiality of their information. These histor- eral] do read it. And it provides actually a benefit, ic protections must be augmented with additional because it’s about transparency.” measures to increase transparency and account- An important limitation is that these notices ability about how data are accessed and used for are for systems of records, not for individual proj- exclusively statistical purposes. ects. Today, no notice is required for individual statistical or evidence-building projects. Some other Transparency countries provide public notice about individual

The Congress and the Executive Branch require 11. Marc Groman, U.S. Office of Management and Budget, Com- public notice when the government collects con- mission Meeting, Washington, D.C., September 9, 2016. 76 The Promise of Evidence-Based Policymaking

projects that link confidential data (see the box and improve the quality of the data collection. “Case Study: Transparency at Statistics Canada”). In sum, members of the public currently have Federal departments also are required to gain multiple opportunities to learn about and provide approval for Information Collection Requests un- feedback on government’s information collection der the Paperwork Reduction Act for new data activities before they are initiated. Such transpar- collections from 10 or more individuals. These re- ency is valuable but applies to systems of records, quests must go through two public comment peri- rather than to individual linkage and analysis ods before final approval. Some statistical agencies projects, and public notices are sometimes diffi- go even further when conducting new primary cult to find, access, and understand. data collections and engage stakeholder groups When collecting administrative information prior to data collection to incorporate feedback from program participants or beneficiaries,

Case Study: Transparency at Statistics Canada

Statistics Canada’s policies on using adminis- administrative data are combined with other trative data address how that agency protects data. It is transparent in its use of administra- privacy and provides transparency on the use tive data, and communicates benefits of using and linkage of its administrative data about Ca- existing data as well as measures taken to pro- nadians. The policy’s objective is to “maximize tect confidentiality. the efficiency and effectiveness of the use of Statistics Canada employs individuals with administrative data in Statistics Canada’s sta- dedicated roles to promote transparency. The tistical programs. Statistics Canada achieves Director General of the Communication Divi- this objective through a corporate strategy in- sion is responsible for developing and imple- tended to influence, access, use, and manage menting a plan that shows Statistics Canada’s the administrative data supplied to Statistics commitment to the transparent use of admin- Canada, and to maintain public trust while do- istrative data. Statistical and legal analysts in ing so.” 1 Statistics Canada intends the policy the Information Management Division are to communicate its statistical use of adminis- responsible for maintaining and managing an trative data to stakeholders and the public in inventory of all approved data linkages. Dedi- a proactive, coherent, and transparent manner. cated websites provide another layer of trans- Under Statistics Canada’s Principles for the parency.2 Statistical Uses of Administrative Data, statis- As part of its governance over microdata tical uses are kept functionally separate from linkages, Statistics Canada has pre-approved administrative uses through strong legal, poli- specific types of linkages. The linkages involved cy and organizational safeguards. In particular, are those in which privacy risks and situations the confidentiality of all identifiable adminis- of potential conflict of interest are low and trative data obtained by Statistics Canada for where procedures to mitigate risk to confiden- statistical purposes is protected by law, and the tiality and privacy are in place. All other mi- data are only accessed within Statistics Canada crodata linkages must undergo a prescribed re- based on demonstrated needs. Statistics Can- view and approval process, which involves the ada carefully considers the impact on privacy submission of documented proposals to senior in using identifiable administrative data for management. When such linkages include per- a purpose that was not envisaged at the time sonal information, a summary of the linkage is of original data collection, particularly when posted on Statistics Canada’s website.

2. Statistics Canada, “Linkage depository;” http://www.statcan. 1. Statistics Canada, “Policy on the Use of Administrative Data gc.ca/eng/sdle/status. See also “Studies, research papers and Obtained under the Statistics Act;” http://www.statcan.gc.ca/ technical papers;” http://www5.statcan.gc.ca/researchers- eng/about/policy/admin_data (accessed August 10, 2017). chercheurs/ (both accessed August 10, 2017). The Promise of Evidence-Based Policymaking 77

government agencies sometimes indicate that data when an independent group helped to ensure data will only be used for specific purposes, such the presence of privacy protections.13 as “only for processing a benefit application.” Such CARRA and CES operate with only the pub- promises generally mean the Federal government lic transparency and oversight that are currently cannot use those data for any other purpose, in- standard practices for Federal agencies, such as cluding statistical analysis for evidence building, through the annual appropriations process when without re-contacting the individual. However, the Congress reviews programs and activities. The when an agency uses its own administrative data public does not have meaningful opportunities to for exclusively statistical purposes, it generally provide guidance or feedback about the individual does not need to re-contact individuals. An ex- projects that are underway. Some states have be- ception under the Privacy Act allows agencies, in gun to develop procedures for public transparency some cases, “to engage in an intra-agency disclo- about evidence building that could be a model (see sure of data for statistical purposes….”12 box “Case Study: Mississippi’s LifeTracks System”). Federal agencies today operate with limited transparency about individual projects seeking to Privacy Auditing link and analyze confidential data for exclusively statistical purposes. Even agencies at the fore- Transparency in Federal evidence building requires front of Federal evidence building, like CARRA verifying that the Federal government implements and CES, do not currently have easily accessible clear, strict rules about data access and follows information about approved projects, qualified re- through on its promises to the American people to searchers, and datasets linked for evidence build- keep data for evidence building safe and secure. ing. Further, because CARRA and CES are located One way to promote public trust through ac- within the Census Bureau, they do not have their countability is to require privacy audits, which own independent advisory boards. A recent Gal- assess available protections within an organiza- lup survey reported that nearly two-thirds of re- tion against legal requirements and privacy best spondents were more likely to support combining practices. Privacy audits document the status of an organization’s risk associated with information 12. U.S. Office of Management and Budget (OMB), Guidance for misuse and make recommendations to limit this Providing and Using Administrative Data for Statistical Purposes” risk. A privacy audit also reviews an organization’s (memorandum M–14–06, Washington, D.C.: OMB, Executive Of- fice of the President, February 14, 2014);https://obamawhitehouse. archives.gov/sites/default/files/omb/memoranda/2014/m-14-06.pdf 13. CEP staff correspondence with Census Bureau staff in April (accessed August 10, 2017). 2017.

Case Study: Mississippi’s LifeTracks System

Mississippi LifeTracks is a state longitudinal devoted a portion of their website to pub- data system that, in addition to programmatic lic accountability. The website lists approved uses, allows for the analysis of administrative projects and completed projects and cites data from multiple state agencies to assess ed- state-level statistics based on their results. This ucation and workforce outcomes in the state.1 demonstration of the value of the program and LifeTracks is funded through a combination of the useful information it provides has been U.S. National Center for Education Statistics suggested as the key to the system’s sustain- grants and annual state appropriations. One ability. LifeTracks aims to hold state govern- challenge Mississippi faced in creating the ment accountable for implementing effective LifeTracks system was conveying the value of programs and policies. At the same time, the the system. To accomplish this, Mississippi public can hold government accountable for the use of data in the LifeTracks System be- 1. Domenico Parisi, Mississippi State University, Commission cause of the program’s transparency efforts. Meeting, Washington, D.C., January 13, 2017. 78 The Promise of Evidence-Based Policymaking

compliance with its own privacy-related polices.14 building. Simply providing a way for researchers In the private sector, privacy audits provide “a to see what data are potentially available for evi- means of benchmarking corporate privacy practic- dence building, along with appropriate technical es against what the law requires and what indus- documentation on each dataset to assess relevance try best practices demand.”15 Conducting a privacy and quality, can allow researchers to identify how audit and correcting any identified weaknesses existing data can be used to answer research and demonstrates to the public that an organization is evaluation questions about government programs serious about both transparency and privacy. and policies. Today, there is no coordinated auditing func- The Commission’s study of the existing inven- tion for Federal evidence building. This lack of ac- tory of government data relied on recent work by countability means that there is no consistency in OMB. In 2016, OMB summarized the results of an whether and how Federal departments review and attempt to document existing datasets across gov- audit access to and use of confidential data, mon- ernment through a white paper for the Commis- itor agency compliance with transparency and sion.16 Federal departments already are required privacy rules and procedures, monitor research- to create data inventories to the extent practica- er compliance with the terms of their data access ble, with the ultimate goal of including all agency agreements, recommend appropriate penalties for datasets.17 Today, however, departments’ inven- violations of policies and procedures, or report to tories are of uneven quality and completeness, the Congress and the American people. especially for datasets that agencies determined cannot be publicly released. In addition, the exist- Challenge #4: Datasets that could be used ing inventory does not include datasets that may be relevant for generating evidence about Federal for evidence building do not all have ade- programs and policies but are collected by state quate technical documentation. and local governments or other jurisdictions. A more comprehensive inventory of data assets that The Commission routinely heard about challeng- are available for evidence building—or not avail- es associated with researchers accessing data due able but should be—would be extremely valuable to a lack of information about what datasets are for Federal staff and external researchers. Com- available, what data elements are included in a mission witnesses suggested that any catalog of dataset, the quality of those data, and how the data useful for evidence building should indicate dataset can be accessed. Notwithstanding recent whether the data are restricted so researchers can attempts to develop a data inventory and calls for determine if they are requesting data that may re- improved technical information about datasets, quire additional hurdles to access. government lags behind the private sector in its standards for managing and documenting data Metadata that could be used for evidence building. Techni- cal documentation standards are a necessary pre- Once an inventory is created, potential data users cursor to routine secure evidence generation. still need to understand the contents, characteristics, and quality of individual datasets. One par- Data Inventory ticular kind of technical documentation consists of data about the data, often called “metadata.” During the Commission’s fact-finding phase, com- Especially when administrative data are initially menters stated that a data inventory with basic technical information is fundamental to evidence 16. U.S. Office of Management and Budget, “Comprehensive Data Inventory,” (white paper for the Commission on Evidence-Based 14. Muzamil Riffat, “Privacy Audit—Methodology and Related Policymaking, Washington, D.C., 2016); https://obamawhitehouse. Concerns,” ISACA Journal 1 (2014); https://www.isaca.org/Journal/ archives.gov/sites/default/files/omb/mgmt-gpra/comprehensive_data_ archives/2014/Volume-1/Pages/default.aspx (accessed August 10, inventory.pdf (accessed August 10, 2017). 2017). 17. U.S. Office of Management and Budget, “Open Data Policy— 15. Michael L. Whitener, “Conducting a Privacy Audit,” The Cor- Managing Information as an Asset” (memorandum M–13–13, porate Counselor 27, no. 3 (July 2012); https://iapp.org/media/pdf/ Washington, D.C.: OMB, Executive Office of the President, 2013); knowledge_center/Conducting_a_Privacy_Audit_-_The_Corporate_ https://obamawhitehouse.archives.gov/sites/default/files/omb/ Counselor_-_July_2012.pdf (accessed August 10, 2017). memoranda/2013/m-13-13.pdf (accessed August 10, 2017). The Promise of Evidence-Based Policymaking 79

provided and used for statistical activities, infor- cludes variable definitions, units of measurement, mation about their meaning and intent can be and response ranges. Also, responses to the CEP lost. Making appropriate use of administrative survey suggested that overall, offices that used a data for evidence building requires the develop- higher percentage of their budget for statistical, ment of data documentation, which can be time evaluation, research, and analysis purposes more intensive and require a deep knowledge of the frequently created robust metadata. But among administrative data collected.18 OMB’s report on a Federal departments administering programs and comprehensive government inventory described collecting data that can be useful for evidence the minimal amount of metadata required under building, the completeness and accuracy of meta- the Open Data initiative; however, not all agen- data vary widely. cies have met this minimum requirement.19 Com- mission witnesses requested much more robust Harmonization of Definitions metadata for evidence building. Complete and accurate metadata can help re- Different Federal agencies have different defini- searchers determine whether access to certain tions of common terms and key concepts in their data even makes sense for their specific project data systems. Sometimes, the same agency has and allow them to assess data quality prior to multiple different definitions of the same con- seeking access to microdata. In fact, incomplete cept. These differences evolved because of how or inaccurate metadata may result in researchers the terms are defined in law or because of specific seeking access to confidential data that will not program needs. answer their research question. Having complete This variation in definitions for common and accurate metadata is vital to improving se- terms and key concepts in government presents cure data linkage and access. In fact, standardized challenges in analyzing data across agencies and metadata is an essential element of employing sometimes across jurisdictions. At times, these privacy-preserving technologies such as Secure variations are acknowledged and planned for or Multiparty Computation (see Chapter 3). addressed by programs. In 2008, the Special Sup- In addition to emphasizing the importance of plemental Nutrition Program for Women, Infants, the Federal government maintaining a list of gov- and Children at the U.S. Department of Agricul- ernment-held datasets and providing basic infor- ture updated guidance to states about what data mation about mechanisms for accessing the data- elements should be included in data systems, and sets, many stakeholders called for a list of variables clarified definitions needed for managing case- in the datasets as the most basic form of metadata. loads.21 For some concepts with government-wide Stakeholders described metadata as a critical com- applicability, OMB issues standards to create component of assessing data quality and fitness for use; mon definitions, such as the Standard Occupa- the Commission heard numerous recommenda- tional Classification or a recently revisited stan- tions to create robust administrative dataset docu- dard on race and ethnicity.22 mentation by standardizing metadata.20 Multiple witnesses in the Commission’s public The CEP Survey of Federal Offices found that hearings and other commenters highlighted the all 12 PSAs that answered the survey item on need to harmonize definitions and the meaning documentation already create metadata that in-

21. U.S. Department of Agriculture (USDA), Food and Nutrition Service, Functional Requirements Document for a Model WIC Infor- 18. Aileen Rothbard, “Quality Issues in the Use of Administrative mation System (Washington, D.C.: USDA, 2008); https://www.fns. Data Records,” in Actionable Intelligence: Using Integrated Data Sys- usda.gov/sites/default/files/apd/FReD%20v2.0%20Final.pdf (ac- tems to Achieve a More Effective, Efficient, and Ethical Government, cessed August 10, 2017). edited by John Fantuzzo and Dennis P. Culhane (New York, NY: Palgrave MacMillan, 2016): 72–103. 22. U.S. Office of Management and Budget, “Standards for Main- taining, Collecting, and Presenting Federal Data on Race and 19. U.S. Office of Management and Budget, “Comprehensive Data Ethnicity,” Federal Register 81 (September 30, 2016); https:// Inventory.” www.federalregister.gov/documents/2016/09/30/2016-23672/ 20. Data quality pertains to data completeness, data accuracy, and standards-for-maintaining-collecting-and-presenting-federal-data- consistent definitions of data elements. The fitness for use is de- on-race-and-ethnicity; OMB, “2010 Standard Occupational Classifi- termined based on the intended use, and may be limited by data cation (SOC)—OMB’s Final Decisions,” Federal Register 74 (January quality or other challenges associated with overly aggregated data 21, 2009); https://www.bls.gov/soc/soc2010final.pdf (both accessed forms, use restrictions, or ethical considerations. August 10, 2017). 80 The Promise of Evidence-Based Policymaking

Interagency Employer Data Matching Workgroup

An interagency workgroup developed strate- ent-child relationships within firms, and the gies for matching and reusing data on U.S. em- dynamic nature of such relationships. The ployers across Federal datasets for statistical workgroup found two primary challenges in purposes. In 2016, the group published a white matching employer data collected by the Fed- paper that summarized key issues for match- eral government: (1) the lack of a common ing employer data and identified best practices universal identifier and (2) poor quality of the that could be implemented across agencies.1 underlying identifying data. Matching employer data can be complex The interagency group identified the most because of the different levels of data (estab- important data elements for matching em- lishment versus firm or enterprise), the par- ployer data across Federal agencies. The most common elements that are most essential to matching include Employer Identification 1. Employer Data Matching Workgroup. “Employer Data Number, Legal Entity Identifier, legal business Matching Workgroup White Paper.” (white paper for the Com- mission on Evidence-Based Policymaking, Washington, D.C. name, trade name, physical and mailing ad- 2017). dress, and country code.

of outcome metrics across Federal laws. Several imizing the possible risk of harm to individuals or witnesses expressed a desire for the Federal gov- organizations. The Commission concluded that ernment to establish common definitions for per- the Congress and the President must empower formance metrics across the Higher Education government to develop state-of-the-art capacity to se- Opportunity Act, the Workforce Innovation and curely combine existing data and provide secure data Opportunity Act, and the Perkins Career and Tech- access for exclusively statistical purposes in a priva- nical Education Act to reduce burden and improve cy-protective and transparent way. opportunities for integrating data across programs In the Commission’s vision, linkages of confi- with similar goals and target populations. Another dential data and access to those data for solely sta- witness identified inconsistencies in the applica- tistical purposes will occur in a highly protective tion of geographic definitions across the Federal privacy, legal, and technological environment, de- government and suggested they be further har- scribed in Chapters 2 and 3. Project-by-project re- monized.23 As described in the box “Interagency views will help the evidence-building community Employer Data Matching Workgroup,” some Fed- work toward the ideal of permitting access to data eral agency personnel recognize the importance of for projects with great promise for improving the consistent definitions for key matching variables public good, while limiting the risk of re-identifi- to combine economic data on firms and business- cation and other privacy harms. At the same time, es for statistical analysis. a more transparent process for statistical analysis of confidential government data by researchers Recommendations and government agencies will reduce unintended inefficiencies that can inadvertently hamper the The Commission’s strategy seeks to extract the generation of evidence. maximum value of data for public good while min- The Commission offers five recommendations regarding the implementation of the NSDS as a service to support privacy-protective, transparent, 23. Timothy Slaper, Indiana Business Research Center, Indiana University, Commission Public Hearing, Chicago, January 5, 2017. and accountable evidence generation. The Promise of Evidence-Based Policymaking 81

REC. 4-1: The National Secure Data mission also recommends that NSDS should have (1) an institutional placement that gives it neces- Service (NSDS) should be established sary independence, (2) designation as a PSA with as a separate entity in the Department of authority to protect data under CIPSEA, (3) sep- Commerce that builds upon and enhances arate funding, and (4) an improved governance existing expertise and infrastructure in the structure offering enhanced transparency. Federal government, especially at the Cen- Using the expertise that already exists, NSDS sus Bureau, to ensure sufficient capacity in will be well placed to quickly begin implementing activities, such as developing agreements with secure record linkage and data access for data providers, developing model approaches for evidence building. securely sharing data across agencies, and facilitating secure access to data for researchers external to The Census Bureau already has established, in pi- government with approved projects that serve the lot form, some of the capacities that are central public good (see Figure 10). The Commission envi- to the Commission’s vision for the NSDS. CARRA sions that when the relevant parts of CARRA and and the CES at the Census Bureau each work to CES become part of the NSDS they would continue combine data that can be made available for re- to manage the FSRDC network and that research- searchers, primarily through the FSRDCs. The ers external to government will continue to access Commission concluded that the Census Bureau’s confidential data through the FSRDC network. current operations, and other capacity through- The Commission’s vision requires that the out the Federal government, form a starting point Federal government stay on the cutting edge of for implementing Recommendations 2-1 and 2-2. technological approaches to protecting data pri- Building on the Census Bureau’s history and cul- vacy and security. The technical approaches for ture of data stewardship and public trust, NSDS combining data require a high degree of expertise can be a cornerstone of future Federal evidence and continuous training to keep abreast of rapidly building with a focus on secure access to Federal evolving methods. data, public transparency, and a consistent set of Concentrating this expertise in a Federal center legal and technological privacy protections. for excellence means that the government can de- The existing Census Bureau infrastructure in velop best practices for data protection and secure CARRA and CES supports, on a small scale, inter- access. Substantially restricting the individuals nal research and operational uses of administra- who have access to identifiable records on indi- tive data and, in partnership with the FSRDC net- viduals or businesses will provide a higher degree work, some externally motivated research projects of safety and security for confidential data. using administrative data. The current infrastruc- Bringing expertise together within govern- ture should be expanded to support increased ment will help identify and implement feasible evidence building. Rather than build an entirely new techniques and technologies. Specifically, the new and potentially duplicative set of functions, NSDS should be a center for excellence in develop- a cost-effective approach to implementing the ing, using, and sharing best practices for transpar- NSDS is to adapt and scale the existing elements ently and securely using confidential data for evi- of CARRA and CES and build upon additional ex- dence building. New privacy-protective techniques pertise throughout the Federal government and such as Secure Multiparty Computation (described its partners in the evidence-building community. in Chapter 3) may allow individuals in the Feder- To maximize efficiency, the President and the al evidence-building community to combine data Congress should reallocate designated staff and and conduct analyses without directly accessing or resources from the Census Bureau to form the core storing information. The NSDS should be a lead- infrastructure for NSDS, supplementing those re- er within government for advancing that goal. sources as needed with expertise from other large The NSDS could also implement provable privacy data collecting and disseminating components of methods such as differential privacy (described in the government. To achieve an “optimal arrange- Chapter 3) to create datasets with minimal and ment” for expanding evidence building, the Com- quantifiable risk of re-identification. Furthermore, 82 The Promise of Evidence-Based Policymaking

Figure 10. The NSDS in Action

Agency A Agency B

Direct Identiﬁers Direct Identiﬁers Admin. Admin. Data Data

National Secure Data Service (NSDS)

Combines Data and Formats into Linkable Datasets Direct Identifiers Direct Identifiers Removes and Statistical Statistical Deletes Direct Data Data 15 Identifiers Assigns Expiration Date to Data Housed in NSDS Final Checks Data Minimization Quality Control Documentation Transparency Portal

Data Cleared for Statistical Use 30 Assigns Expiration Project Dataset Date to All Project Data

Secure Use of Datasets

Disclosure Review and Clearance of Results

Public Release of Statistical Results The Promise of Evidence-Based Policymaking 83

best practices should be shared throughout the develops and expands at NSDS. government and lead to more secure data access Building on the existing capacity in the Federal and better privacy protections across the Federal government, and particularly at the Census Bu- evidence-building community. reau, and supplementing it with resources drawn Because of the existing pilot activities at Cen- from across government will ensure that the ex- sus, the implementation of NSDS can begin im- pertise for evidence building can be available not mediately once the Congress and the President just for projects to advance the Census Bureau’s authorize its establishment. Work may then con- mission, but government wide. tinue in phases, with the core activities that support NSDS fully devolved from the Census Bureau REC. 4-2: The NSDS should establish a within two to three years. Steering Committee that includes A phased approach to establishing and implementing NSDS will allow it to conduct its own representatives of the public, Federal de- pilot projects, develop data documentation and partments, state agencies, and academia. integration procedures, and develop and enhance relationships with data providers that allow op- The public must have a voice in how the Federal portunities for public input (see the box “Potential government uses data for evidence building. Pub- Phases for NSDS Implementation”). lic participation ensures the NSDS’s policies and The NSDS should also have the capacity to pro- practices remain focused on exclusively statisti- vide technical services such as secure data link- cal purposes for the public good. The Commission age and analytical services on a fee-for-service concluded that a service like NSDS requires input basis for states, local governments, and other ju- about how data are used from representatives of risdictions. While technical and analytic services the public, privacy and technology experts, Fed- should not be an initial priority for the NSDS, a eral departments, state agencies, research institu- basic level of such services may be both necessary tions, and other stakeholders. and desirable in the long term to help improve the A representative steering committee for NSDS capacity for evidence building throughout the na- will allow individuals to advise the service on tion and to encourage collaboration across levels strategic policies and provides one avenue for of government. Researchers may also wish to avail organizational accountability. The Commission themselves of technical and analytic capacity as it specifically recommends that such an advisory

Potential Phases for NSDS Implementation

Phase I – Legal Authority. New legal authority Phase III – Pilot Projects and Data Doc- is enacted to establish the National Secure Data umentation. NSDS works with researchers Service (NSDS) in its new location, with a stated and other stakeholders to build out new pri- transition period of 2-3 years. vacy-protecting capacities and conduct a series of pilot projects for scaling implementation. Phase II – Relocation and Staffing. The Sec- The NSDS launches a transparency portal. retary of Commerce hires an administrator. NSDS appoints the steering committee. The rel- Phase IV – New Development and Imple- evant assets, personnel, and resources from the mentation. NSDS develops new relationships Census Bureau are transferred to NSDS and the with data providers to meet user needs and administrator brings in staff from other gov- continues to scale activities as necessary based ernment agencies and outside government as on demand. NSDS expands efforts to imple- needed. ment state-of-the-art technologies and techniques for secure data linkage and access and assesses the quality of the approaches. 84 The Promise of Evidence-Based Policymaking

committee be charged with the following respon- propriate use or failure to follow procedures. Fur- sibilities: ther, the administrator must ensure compliance • Consult with the Secretary of Commerce about with applicable Federal laws and policies, espe- budget and resource allocation decisions relat- cially regarding data confidentiality and security. ed to NSDS operations and privacy protections. The administrator should provide leadership across the Federal evidence-building community • Conduct strategic planning for NSDS. to use state-of-the-art data protection and access methods and technologies. However, the adminis- • Develop and ensure appeals processes are trator should not be expected to also fill the role of implemented for applications that are denied, privacy officer for the NSDS. A separate, designat- for privacy concerns raised after a project is ed NSDS official should be appointed to enforce approved, and for administrative sanctions for the full breadth of privacy-protective approaches non-compliance with applicable policies and necessary and expected for the level of data use procedures. in the NSDS anticipated by the Commission, the Congress, and the President. The designated pri- • Oversee and approve an annual report to the vacy official should be given a role in scrutinizing Congress and the President providing appropri- all data linkage activities, data access activities, ate performance indicators, evaluating progress and other data uses for evidence building. toward fulfilling NSDS’s purpose, and describ- A designated privacy official at the NSDS could: ing implementation challenges including those (1) support OMB in privacy standard setting ac- that may require new statutory authority. tivities; (2) establish and promulgate NSDS privacy policies, including for researcher qualification, • Receive and review audit reports on priva- project approval, and data stewardship training; cy-protective protocols and practices; identify (3) ensure NSDS compliance with Federal privacy any ameliorating steps to be taken to reduce laws, regulations, and policies; (4) develop man- privacy risk within the mission of NSDS. datory privacy and data stewardship training for NSDS staff and researchers; (5) review risk assess- While a representative steering committee ments when confidential data are combined across can provide high-level direction and support for departments; and (6) communicate and coordi- NSDS, it would not be feasible or desirable for nate with privacy officers in statistical agencies such a committee to manage the day-to-day op- and departments. erations and provide leadership for the NSDS. The Commission believes there should be a senior ad- REC. 4-3: To ensure exemplary trans- ministrator empowered to administer, oversee, parency and accountability for the and coordinate the activities of the NSDS, such as Federal government’s use of data for evi- implementing regulations and developing proce- dence building, the NSDS should maintain dures to acquire access to data from Federal departments and state agencies. a searchable inventory of approved projects As part of the public participation process, the using confidential data and undergo regular administrator should ensure that individual proj- auditing of compliance with rules govern- ects are approved in consultation with data providing privacy, confidentiality, and access. ers. Information about those projects will then be made available as described in Recommendation The Commission believes that advancing beyond 4-3, and will include who the eligible researcher the status quo and achieving unparalleled trans- is and what data they are using for what purpose. parency means (1) telling the public about how Because the American public expects the Fed- government data are used for evidence building eral government to manage the risks of using data and (2) regularly auditing adherence to priva- for evidence building, the administrator must also cy-protecting policies to assure the public that the prescribe data access control methods consistent Federal evidence-building community only uses with OMB policies and standards, including estab- data for approved exclusively statistical purposes. lishing data use agreements, training for staff and The President, through collaboration between researchers, and determining penalties for inap- the NSDS and OMB, in its government-wide The Promise of Evidence-Based Policymaking 85

standard setting role, should use existing statutory mitigate those cause(s), and a process for revising authority to establish a clear example of transpar- procedures to reduce the risk of similar future in- ency in Federal evidence building. The President cidents. should authorize a single internet-based portal Finally, NSDS should report annually to the that gives details about each project using integrat- Congress, the President, and the public about ed confidential data for evidence building or each uses of confidential data for evidence building external researcher accessing confidential data for and identify and discuss any problems that arise. evidence building. The NSDS is the logical place to Other accountability mechanisms are established develop, host, and maintain the transparency por- through other Commission recommendations; for tal because of its role in combining data across the example, Recommendation 2-8 calls for a single, topical domains of Federal departments. streamlined process for external researcher access Approved projects facilitated by NSDS will be an that is transparent about data sensitivity and sets important input to the portal. In addition, other data access restrictions appropriately, and Rec- PSAs should provide NSDS details on projects they ommendation 3-1 provides for the public posting undertake with other agencies to link confiden- and use of risk assessments for public release of tial data as well as approved projects for external de-identified confidential data. researcher access to confidential data. The information in the portal should minimally include for REC. 4-4: The NSDS should have spe- each approved project: the project name (which cific administrative and implementa- some Federal agencies already publish), a project abstract, the datasets involved, the products of the tion flexibilities including the ability to analyses, and the name(s) and contact information leverage public-private partnerships and to of the approved researcher(s). The transparency collect and retain user fees. portal should also feature the transparency strategies discussed as part of Recommendation 3-1 and In order to implement the NSDS in a manner that could include those discussed under Recommen- allows it to serve the government as a whole and dation 2-8. These strategies include a public inven- continually adapt to emerging needs and chang- tory of data available for evidence, including an ing technology, some administrative flexibility analysis of the sensitivity of the data and publicly beyond the usual government practices will be available risk assessments for the public release of required. Specifically, success for the NSDS will de-identified confidential data under the Privacy require legal authority for grant-making, cooper- Act and CIPSEA. The portal also should include a ative agreements, workforce development, and mechanism for public feedback. other activities that facilitate engagement with Once established, the NSDS transparency por- partner organizations to accelerate the develop- tal will provide a valuable resource for the public ment of new methods and technologies. Such to understand how data are being used and doc- authority could include the ability to sponsor umentation to establish an audit trail. The Con- a Federally Funded Research and Development gress and the President should choose to assign Center to support research and development re- responsibility for auditing to the Government lated to innovative privacy-protective approaches. Accountability Office or to a designated inspector By encouraging research and development in pri- general. Auditors should ensure compliance with vacy-enhancing and secure access technologies, privacy protocols and ensure that enforcement the NSDS can both capitalize on and encourage mechanisms are appropriately applied. Further, innovation in the private sector with joint ben- auditors should be responsible for reviewing the efits to government. Development of new tech- data use and handling procedures and processes nologies related to providing data access, such as in place for confidential data. The NSDS should es- virtual data access technologies, will help address tablish and auditors should monitor a “near miss” increased demand. reporting system for procedural or process viola- The NSDS should have the authority to collect tions to create a feedback and learning cycle about and spend user fees, with sufficient flexibility to risks. Such a system could include descriptions of adjust rates based on changes in demand or other any violations and re-identifications that may oc- factors. Explicit authority should be provided in cur, determinations of cause(s), actions taken to appropriations bills for other Federal departments 86 The Promise of Evidence-Based Policymaking

to transfer funds to the NSDS to support its cen- secure environment. The technical documentation tralized activities, as noted in Chapter 5. requirements of new technologies for securely ac- The NSDS can provide a clear public benefit cessing data such as Secure Multiparty Computa- by facilitating generation of evidence about gov- tion further guide the Commission’s conclusion ernment programs and policies. It will likely not that improvements to the government’s data in- be feasible, however, for the NSDS to cover its ventory, metadata, and definitions are necessary expenses entirely with user fees. Therefore, the precursors to increased evidence generation. Commission believes that while user fees for OMB’s ongoing effort beginning in 2013 to de- some costs associated with the NSDS can support velop a data inventory is a productive start, but “self-funding,” other funding mechanisms will be much more needs to be done to make this infor- necessary to achieve the Commission’s vision. mation complete and useful for evidence building. Such mechanisms could include a direct appropri- The establishment of a comprehensive searchable ation or reimbursable funding from other depart- inventory, through which the public can learn ments. The NSDS will need some level of flexible about the data that government collects, would resources to ensure it can implement cutting-edge improve transparency. With robust metadata, re- technologies that enhance privacy. searchers inside and outside government will be better able to identify which data are needed and REC. 4-5: The Office of Management useful for answering policy questions, conducting program evaluations, and reducing inefficient and Budget should increase efforts to and unnecessary data requests. These metadata make information available on existing Fed- should be sufficient to allow users to assess quali- eral datasets including data inventories, ty and implement privacy-enhancing data match- metadata, and data documentation in a ing technologies, such as Secure Multiparty Com- searchable format. putation. The metadata must also allow the NSDS and OMB to assess the sensitivity of data in spe- Administrative datasets often contain extensive cific contexts and to determine access restrictions records about individuals and businesses receiv- and privacy protection protocols as described in ing government benefits and services, sometimes Recommendation 2-8. including multiple entries on transactions. Given The Commission recognizes that, despite their their size, frequent updates and revisions, and potential for adding value, developing a data in- data structures, use of administrative data for ev- ventory, technical documentation, and consis- idence building requires knowing which data are tent definitions for key concepts across datasets available and what they mean. is a burden and challenge. In the short term, the The Commission intends that while data will be Commission recommends prioritizing metadata accessible through the NSDS, most of those data and other technical documentation related to the will remain in their original locations in the fu- datasets researchers consider important for evi- ture. The NSDS will only maintain a minimal core dence building, including those described in Ap- set of data for linking confidential data within its pendix D. ■ The Promise of Evidence-Based Policymaking 87

5 Strengthening the Evidence-Building Capacity within the Federal Government

Build and maintain a strong Federal infrastructure for the sustained production and use of evidence.

Recommendations

5-1: The President should direct Federal ments, including through any reorganization or departments to increase capacity for evi- consolidation within OMB that may be neces- dence building through the identification or essary and by bolstering the visibility and role of tablishment of a Chief Evaluation Officer, in ad- interagency councils. dition to needed authorities to build a high performing evidence-building workforce. 5-4: The Congress and the President should align administrative processes to 5-2: The Congress and the President support evidence building, in particular by should direct Federal departments to de- streamlining the approval processes for new velop multi-year learning agendas that support data collections and using existing flexibilities the generation and use of evidence. in procurement policy.

5-3: The Congress and the President should 5-5: The Congress and the President direct the Office of Management and Bud- should ensure sufficient resources to sup- get (OMB) to coordinate the Federal govern- port evidence-building activities about Federal ment’s evidence-building activities across depart- government programs and policies. 88 The Promise of Evidence-Based Policymaking

mplementing the features of the Commission’s maximize evidence building, Federal departments I vision to improve the use of data for evidence must have the capacity to support the full range building requires more than just improving access of analytic approaches required for evidence to data and privacy protections. Government must building, including the development of statistics, have the capacity to analyze data, and then apply evaluation, and policy research. These functions insights to inform policymaking. This chapter ad- must be operational, appropriately resourced, and dresses the alignment and empowerment of the well-coordinated both within and across depart- Federal evidence-building community—including ments. Strong leadership within government that those who generate, manage, and analyze data, prioritizes evidence building and creates the de- those who transform information into evidence, mand for evidence is vital for institutionalizing and those who support those functions through these functions. Without a strong institutional the routine processes of government. foundation, other recommendations related to improving data access, establishing the National “Data are necessary, but not Secure Data Service (NSDS), and implementing sufficient, to create evidence. enhanced privacy protections will not have the ” comprehensive impact that is needed. – Naomi Goldstein, Deputy Assistant Secretary for Planning, Research, and Evaluation in the Administration for Children Findings and Families, Commission Meeting, The Commission received input from numerous November 4, 2016 stakeholders, both internal and external to the Federal government, identifying challenges to in- Today, evidence building takes place unevenly creasing the capacity of the Federal evidence-build- across the government, both within the internal ing community and numerous ways in which the Federal evidence-building community and across practices of the Federal government could be partnerships with external members of the evi- streamlined to increase the volume, quality, and dence building community. Within departments timeliness of evidence production. The collective inside the Federal government, key actors include input stressed the importance of addressing these Principal Statistical Agencies (PSAs) and other sta- challenges directly. Based on this feedback, the tistical programs, evaluation and policy research Commission identified five primary challenges offices, program administrators, performance that currently restrict the capacity of the Federal management offices, policy analysis offices, and government’s evidence-building community. privacy offices. Often the work of these offices occurs in silos, leading to duplication of effort or Challenge #1: The capacity of Federal de- missed opportunities for collaboration, and efforts partments to support the full range of ev- to coordinate evidence building across the Federal government have been challenging. In addition, idence-building functions is uneven, and administrative functions that support Federal where capacity for evidence building does government operations across departments are exist, it is often poorly coordinated within poorly aligned to support and prioritize evidence departments. building. Thus, major gaps remain in the Federal evidence-building community’s capacity, which Currently, some departments operate all, and hamper the ability to build on the community’s other departments operate only some, of the ev- many strengths to increase evidence production. idence-building functions necessary to support Federal departments must enhance their ca- evidence-based policymaking. Where these ac- pacity for evidence building to support the growth tivities are taking place in different units across a of evidence-based policymaking. Those generat- department, these efforts must also be coordinating, supporting, and using evidence within and ed to maximize the department’s capacity to ful- across departments throughout the Federal gov- ly address a specific research or policy question. ernment must be empowered and organized to In pursuit of more and better evidence, depart- work together and accomplish shared goals. To ments may choose to pursue different avenues for The Promise of Evidence-Based Policymaking 89

increasing their capacity for evidence building. In “Organizational culture is an some cases, departments will opt to strengthen their human capital by hiring staff or establish- ongoing dialogue between leaders ing units with particular expertise. In other cases, and others in their organization departments may decide that leveraging exper- about ‘how we do things around tise outside of government is a more efficient approach to expanding capacity. here.’ Data and evidence are Regardless of how a department may pur- necessary but not sufficient. sue expanded capacity, the coordination of ev- Continuing dialogue based on the idence-building functions is vital for achieving maximum efficiency. Currently, some departments data and evidence is critical.” prioritize coordination across units while other de- – Seth Harris, Former U.S. Department of partments lack a coordinating infrastructure. The Labor Acting Secretary, Commission Meeting, following section (1) describes each of the evi- March 13, 2017 dence-building functions needed within Federal departments, (2) offers existing approaches for ex- The evaluation and policy research functions panding the evidence-building capacity of depart- are newer additions to the evidence-building ments through human capital strategies and exter- community in many departments. The American nal partnership arrangements, and (3) addresses Evaluation Association defines evaluation as “as- the importance of coordinating evidence-building sessing the strengths and weaknesses of individ- efforts within a Federal department. ual programs, policies, and organizations to improve their effectiveness, efficiency, and worth… Evidence-Building Functions within It uses systematic data collection and analysis to Federal Departments address questions about how well government programs and policies are working, whether they Key actors within the Federal evidence-building are achieving their objectives, and, no less impor- community include: the PSAs and other statistical tantly, why they are or are not effective.” 1 Policy programs, evaluation and policy research offices, research is related to evaluation and often pre- program administrators, performance manage- cedes formal evaluation, but plays a fundamen- ment offices, and policy analysis offices. These are tally different role in evidence building.2 Because central entities that support an organizational cul- these functions are less institutionalized in many ture that enables evidence building. Federal departments, they require different solu- Thirteen PSAs form the centerpiece of the larg- tions to bolster their development compared with er Federal Statistical System, which also includes longer standing functions such as the develop- dozens of statistical programs spread across Fed- ment of statistics. While some excellent examples eral departments, embedded within programmat- of evaluation and policy research offices exist in ic agencies. For many decades, the PSAs have col- the Federal government today, many departments lected most of their data by conducting surveys of will likely need additional capacity to meet the individuals, businesses, and other organizations. Commission’s vision of cultivating these two es- PSAs now increasingly supplement these surveys pecially important types of evidence building. with administrative data, creating new opportunities, and also new relationships with program agencies. Because of their existing legal confi- 1. American Evaluation Association, “An Evaluation Roadmap for dentiality protections, culture of data protection, a More Effective Government” (Washington, D.C.: AEA, October 2013); (accessed August 10, 2017). and substantive and technical expertise, PSAs can http://www.eval.org/d/do/472 serve as building blocks for the modern system of 2. Research includes basic and applied activities conducted in the pursuit of new knowledge, the former directed at general appli- evidence building in the United States. The grow- cations or theory and the latter focused on practical objectives. ing capabilities of many PSAs to transform and an- “Policy research” is used synonymously with “policy-relevant re- alyze administrative data should be expanded to search” both of which are considered gaining knowledge about programs and policies. See OMB, Section 84: Character Classifica- meet the needs for government evidence building tion (Schedule C) in “Preparation, Submission, and Execution of and the public good. the Budget” (Circular A–11, Washington, D.C.: OMB, July 2016). 90 The Promise of Evidence-Based Policymaking

Program administrators play an increasingly tions, geographies, or contexts.5 important role in evidence-building activities, as To the extent that capacity is uneven in gov- more administrative data collected for program ernment, important evidence-building activities administration purposes are being transformed may be crowded out by other priorities within for use in evidence building. In addition, program departments. Departments face numerous pres- administrators are one audience for the evidence sures to respond to immediate and important being generated, as they have the capacity to ad- requests that limit the ability to deploy staff or ministratively implement program and policy ad- funding for evidence building. For example, pri- justments based on the results of research and oritizing funding for new programs and initia- evaluation. tives may mean longstanding programs receive Performance management offices are responsi- less attention for measuring or assessing program ble for advising and assisting departmental lead- outcomes. Program administrators whose perfor- ership to ensure that the mission and goals are mance is assessed based on their meeting annual achieved cost effectively through strategic and targets may be reluctant to devote program re- performance planning, measurement, analysis, sources to longer-term evidence-building activ- regular assessment of progress, and the use of ities. In most departments, the development of high-quality performance information and other annual performance measures aimed at provid- evidence to improve results.3 ing timely information for decisions in the bud- Centralized policy analysis offices support get process itself consumes an array of resources. many departments today and typically perform While performance measurement (or monitoring) an important translation function. Policy analy- is an important component of the broader field of sis translates knowledge gained from evaluation evaluation,6 a focus on short-term activities may and policy research both inside and outside gov- jeopardize the ability to prioritize long-term proj- ernment, making the information accessible to ects that do not align with the time horizons of program administrators and operators, policy- policymakers and program administrators. The makers, including the Congress, and the public. capacity must exist for both short- and long-term The dissemination of information helps inform activities. and spur action. These “knowledge brokers” serve as intermediaries between evidence users and Mechanisms for Expanding Capacity producers, and play an important role within for Evidence Building in Federal the evidence-building community.4 Increasingly, as a greater volume of evidence is produced in Departments the future, knowledge brokering efforts also will Strengthening the program evaluation function need to expand. Policy analysis offices sometimes within the Federal government is an important sponsor or rely on formal dissemination tools for first step in expanding evidence building. Sever- evidence, such as the What Works Clearinghouse al strong evaluation units currently exist across operated by the U.S. Department of Education, government and work with contractors, other and the Clearinghouse for Labor Evaluation and government staff, and academic institutions to Research, operated by the U.S. Department of implement government’s evaluation function. Labor, which offer compilations of research and Such offices include the Office of Planning, Re- evaluation to provide grantees with a valuable search, and Evaluation (OPRE) within the Admin- starting point for determining which interven- istration for Children and Families (ACF) of the tions may be most appropriate to their popula- U.S. Department of Health and Human Services (HHS) and the Chief Evaluation Office within the

3. OMB, “Overview of Federal Evidence-Building Efforts” (white paper for the Commission on Evidence-Based Policymaking, 5. What Works Clearinghouse; https://ies.ed.gov/ncee/wwc; Clear- Washington, D.C.: OMB, 2016); https://obamawhitehouse.archives. inghouse for Labor Evaluation and Research; https://clear.dol.gov gov/sites/default/files/omb/mgmt-gpra/overview_of_federal_evidence_ (both accessed August 10, 2017). building_efforts.pdf (accessed August 10, 2017). 6. Kathryn Newcomer and Clinton T. Brass, “Forging a Strategic 4. Karol Olejniczak, Estelle Raimondo, and Tomasz Kupiec, “Eval- and Comprehensive Approach to Evaluation Within Public and uation Units as Knowledge Brokers: Testing and Calibrating an In- Nonprofit Organizations,”American Journal of Evaluation 37, no. 1 novative Framework,” Evaluation 22, no. 2 (April 2016): 168–189. (March 2015): 80–99. The Promise of Evidence-Based Policymaking 91

Department of Labor (DOL). (See the box “U.S. To both support and benefit from routine ev- Department of Labor—Chief Evaluation Officer”). idence building, Federal departments require a OMB supports evaluation activities across govern- range of expertise including in-house expertise ment and encouraged the administrative creation to design and conduct evaluations or surveys, ad- of a voluntary coordinating unit called the Inter- vanced technical knowledge required for combin- agency Council on Evaluation Policy. ing or analyzing data, and expertise in writing and As noted in the Analytical Perspectives sec- managing highly technical evidence-building con- tion of the President’s Fiscal Year 2018 Budget, tracts. In seeking to increase capacity for evidence “Centralized or chief evaluation offices play an building, some Federal departments may choose important role in an evidence infrastructure that to establish new units or expand the functions of can develop and sustain agency capacity to build an existing unit. The CEP Survey of Federal Offices and use evidence.” 7 In this vein, the Commission showed that over half (54 percent) of responding received testimony suggesting that a Chief Eval- offices reported difficulty hiring staff with the ap- uation Officer could assume responsibility for (1) propriate skill set to match the work requirements establishing department-wide evaluation and re- for evidence-building activities.8 Functions that search policies that encourage rigor, credibility, are newer in government, such as program eval- independence, and transparency; (2) coordinat- uation, have particular challenges in attracting ing and supporting technical expertise for evalua- and retaining qualified staff and, therefore, may tion and research within the department; and (3) require some adjustments in existing Federal hu- identifying and setting priorities for departmental man resource policies. Today, while the U.S. Office program evaluation and policy research, with ap- of Personnel Management (OPM) recognizes stat- propriate attention to the mission and context of isticians, economists, information specialists, and each department. Particularly in very large or de- policy analysts as official and distinct occupations centralized organizations, departments may have within government, no occupational series exists multiple centers for evaluation, which could be specifically for the field of program evaluation. coordinated by a Chief Evaluation Officer. External partnerships also can be an effective way to expand government capacity for production

7. OMB, “Building and Using Evidence To Improve Government Effectiveness,” inAnalytical Perspectives, Budget of the United States Fiscal Year 2018 (Washington, D.C.: OMB, May 2017); https://www. 8. Based on 79 offices that reported that they collect or use data whitehouse.gov/sites/whitehouse.gov/files/omb/budget/fy2018/ap_6_ for statistics, evaluation, research, or policy analysis or spend a evidence.pdf (accessed August 10, 2017). portion of their budget for such purposes.

U.S. Department of Labor—Chief Evaluation Officer

The U.S. Department of Labor (DOL) is cur- ects designed to build evidence about DOL rently the only Cabinet-level department to programs and policies. have a Chief Evaluation Officer. Established in 2. Complements, not duplicates, agency eval- 2010, the Chief Evaluation Office, led by the uation functions. Chief Evaluation Officer, coordinates, manages, and implements DOL’s evaluation program. 3. Raises the quality of evaluations and aware- The Chief Evaluation Office is an independent ness and knowledge of evaluation method- evaluation office, located organizationally in ology and standards. the Office of the Assistant Secretary for Policy. 4. Improves use and dissemination of evalua- At the department level, the Chief Evaluation tion results. Officer: 5. Improves access to, quality of, and use of 1. Designs, funds, and oversees the imple- data. mentation of a portfolio of evaluation proj- 92 The Promise of Evidence-Based Policymaking

of evidence by leveraging the expertise of the expand staff capacity and build staff expertise. non-Federal evidence-building community. The Federal departments have mechanisms for hir- Robert Wood Johnson Foundation encouraged ing staff from colleges and universities and oth- the Commission to consider how “partnerships er institutions on a temporary basis, such as the between researchers, service providers, and large Intergovernment Personnel Act (IPA) Mobility institutions—both governmental and corporate— Program. Assignments under the IPA program are can improve the capacity to develop evidence.” 9 intended to “facilitate cooperation between the Engaging successfully with external partners re- Federal government and the non-Federal entity” lies on having staff and structures in place to es- and to serve a “sound public purpose.” 10 tablish and support collaboration with academics, non-profits, foundations, and businesses. While Coordination of Evidence-Building tools like grant-making or cooperative agreements Functions within Federal Departments have been used to good effect at some agencies to build external partnerships, not all agencies have Currently, the coordination of evidence-building the necessary authority to use these tools for part- activities within departments varies widely. In nerships. some departments, very little coordination exists, Partnerships can be especially useful for small- whereas other departments place a high priority er agencies with limited capacity. Agencies must on coordination. For example, HHS supports an continually develop their workforce and increas- entire unit, the Office of the Assistant Secretary ingly need staff with specialized expertise in pri- for Planning and Evaluation (ASPE), which is re- vacy protections, data security, computer science, sponsible for major activities in policy coordina- and statistical methods. Partnering with non- tion, legislation development, strategic planning, governmental entities can be an efficient way to

10. Office of Personnel Management, “Hiring Information: In- tergovernment Personnel Act;” https://www.opm.gov/policy-data- 9. Robert Wood Johnson Foundation, submission to the Commis- oversight/hiring-information/intergovernment-personnel-act (ac- sion’s Request for Comments. cessed August 10, 2017).

Case Study: HHS Data Council

The Data Council within the U.S. Department of data standards and statistical policies, and pri- Health and Human Services (HHS) is an exam- vacy protection policies. An important activity ple of successful coordination of data resources of the Data Council includes leveraging HHS within a Federal department.1 The HHS Data statistical and administrative data collections Council is the principal internal advisory body to support evidence building. The HHS Data to the Secretary for health and human services Council develops recommendations regarding data and statistical policy, and facilitates collab- the collection, analysis, and dissemination of oration and coordination among various offices data to guide future decisions and enhance the involved in evidence building, meeting month- health and well-being of Americans. ly to discuss department-wide issues related to The Council is co-chaired by the Assistant data and statistical policy. The HHS Data Coun- Secretary for Planning and Evaluation (ASPE) cil plays an essential role leading the develop- or their designee, the head of an operating divi- ment and implementation of an integrated data sion or their designee (currently the Director of collection strategy, coordination of analysis ac- the Agency for Healthcare Research and Quali- tivities, development and implementation of ty), and the Director of the National Center for Health Statistics, who also serves as the senior advisor to the Secretary on health statistics. 1. U.S. Department of Health and Human Services, “HHS Data Council: An Introduction;” https://aspe.hhs.gov/hhs-data- The office of the ASPE serves as Executive Di- council-introduction (accessed August 10, 2017). rector for the Council. The Promise of Evidence-Based Policymaking 93

policy research, evaluation, and economic anal- cision-making.” 11 Such learning agendas typically ysis across the entire department. (See the box identify the most pressing research and policy “Case Study: HHS Data Council”). questions facing a department at a particular time, Harnessing administrative data as a more inte- and provide departmental leaders with a mecha- gral part of evidence building heightens the need nism to prioritize research questions within bud- for meaningful collaboration within departments. get and policy timeframes. Coordination within a department is vital for the The Office of Policy Development and Research efficient production of evidence. within the U.S. Department of Housing and Ur- ban Development (HUD) developed a learning agenda titled the “Research Roadmap.” 12 The Re- Challenge #2: Federal departments fre- search Roadmap outlines the key research ques- quently do not have an integrated approach tions relevant to HUD’s mission, and enables HUD or a long-range plan for evidence building. leadership to be responsive to OMB’s direction to strengthen the use of evidence and technology to To make evidence-based policymaking routine in improve government performance. HUD refreshes the Federal government, a broad base of evidence the Research Roadmap every five years, engaging a and information must be available in order to in- broad stakeholder group both inside and outside of form decision-making. Departments can benefit the department, to ensure that the priorities cap- from an organized approach by which they regular- tured reflect the needs of the department and the ly identify short- and long-term priority research broader field. Learning agendas also can be used to and policy questions relevant to the department’s communicate research priorities to external part- mission and legal responsibilities. The result of ners to help catalyze targeted evidence-building ac- this exercise is a learning agenda, which can be tivities outside the Federal government. used by leadership to prioritize the set of research and policy questions to be pursued by the depart- A Portfolio of Evidence ment over a given period. The evidence-building community within a department can in turn use Evidence building and use must become a regu- the learning agenda as a coordination tool. lar feature of program oversight and operations to The effective implementation of learning agen- promote continual learning, program refinement, das requires sustained leadership and support. and accountability. Today, nearly every Feder- The various units within the evidence-building al government program collects information to community in a department must work togeth- support program operations and monitor perfor- er to determine the best approach to answering mance. The data gathered through these activi- a priority research or policy question, and to al- ties provide valuable basic information about the locate the work appropriately across the different scope and reach of programs, such as identifying evidence-building functions. A well-coordinated the number of participants in a program. Depart- evidence-building community within a Federal ments generally need more in-depth information department will be able to leverage the various to understand how a program is implemented, strengths of each function, resulting in a more ro- the costs of administering the program, and the bust project. extent to which policies yield anticipated results. Policy decisions should be reviewed using a broad Learning Agendas set of methodologies, including descriptive statistics, process studies, implementation evaluations, The development of a learning agenda is a con-

crete step that some Federal departments have 11. U.S. Agency for International Development, “Implementing taken towards prioritizing evidence building. a Learning Agenda Approach;” https://usaidlearninglab.org/sites/ Learning agendas are defined as “a set of broad default/files/resource/files/defining_a_learning_agenda.pdf (accessed questions directly related to the work an agency August 10, 2017). conducts that, when answered, enable the agency 12. U.S. Department of Housing and Urban Development (HUD), HUD Research Roadmap: 2017 Update (Washington, D.C., January to work more effectively and efficiently, particu- 2017); https://www.huduser.gov/portal/pdf/ResearchRoadmap- larly pertaining to evaluation, evidence, and de- 2017Update.pdf (accessed August 10, 2017). 94 The Promise of Evidence-Based Policymaking

impact evaluations, using randomized control tri- The Commission identified numerous exam- als where appropriate, and meta-analysis. The re- ples of Federal programs developing increasingly sults of all relevant evidence-building activities is rigorous portfolios of evidence, but these pro- referred to as a portfolio of evidence. grams are still the exception, rather than the rule. The American Evaluation Association suggests One example is the Teen Pregnancy Prevention that in selecting a methodological approach, the Program administered by HHS, which was de- first step is to “identify the important evaluation signed to address high teen pregnancy rates in questions [needing to be] answered to effective- the United States by replicating evidence-based ly direct the future of Federal programs” allowing models and testing innovative strategies. Evi- evaluators to then “identify which scientific meth- dence building was woven into the program from ods are best suited to answer those questions.” 13 the start, including a full range of studies from The Commission finds that the need for evidence implementation assessments to impact evalua- about programs is context specific and should be tions, using random assignment when appropri- tailored to produce the most relevant, valid, and ate. The program is administered by the Office of reliable information possible for evaluating indi- Adolescent Health, which used a multipronged vidual programs and policies. approach to evaluation, including the establish- Two of the Commission’s guiding principles for ment of uniform performance measures, a series evidence-based policymaking outlined in Chapter 1 of grantee‐led evaluations, and multiple cross-site are particularly relevant when considering both the evaluations. The purpose of this framework was generation of evidence and the application of evi- to address the question of whether the replicated dence by policymakers. The principle of rigor states evidence‐based teen pregnancy prevention pro- “evidence should be developed using well-designed grams and the new, innovative strategies for pre- and well-implemented methods tailored to the venting teen pregnancy were effective. question being asked.” For example, when seeking Evelyn Kappeler, former Director of the Office to determine program impacts, random assignment of Adolescent Health at HHS, described the pro- generally is preferable for developing causal state- gram as follows: ments. The Institute of Education Sciences and the Directorate for Education and Human Resources Our experience with the first cohort of the National Science Foundation established of [Teen Pregnancy Prevention grant- guidelines for evaluation that conclude “Efficacy, ees] brings to light several important effectiveness, and scale-up research should use issues regarding the capacity of the study designs that will yield impact estimates with organizations to conduct rigorous strong causal validity . . . generally and when feasi- evaluations. Many of these lessons ble, they should use designs in which the treatment have applications for government and comparison groups are randomly assigned.” 14 entities, which are conducting evalua- In another principle adopted by the Commis- tions in community settings. The chal- sion, policymakers are urged to approach evidence lenges and lessons learned include the with humility. The Commission’s humility princi- importance of providing for a plan- ple says “Care should be taken not to over-gener- ning and piloting period early in the alize from findings that may be specific to a partic- grant cycle, measuring and monitor- ular study or context.” Even when evaluated using ing fidelity and adaptations, ensuring methods appropriate to stated questions, contexts high quality program implementation, may differ, circumstances can change, and fidelity incorporating evaluation effectively to program design may diminish over time while once program implementation has conducting an evaluation. already begun, ensuring strong contrast between treatment and control, providing intensive evaluation tech- 13. George Grob, American Evaluation Association, Commission nical assistance, and disseminating all Public Hearing, Washington, D.C., October 21, 2016. evaluation results transparently.15 14. Institute of Education Sciences, U.S. Department of Education, and the National Science Foundation, Common Guidelines for Edu- cation Research and Development (August 2013); https://ies.ed.gov/ 15. Evelyn Kappeler, Office of Adolescent Health, Commission pdf/CommonGuidelines.pdf (accessed August 10, 2017). Meeting, Washington D.C., November 4, 2016. The Promise of Evidence-Based Policymaking 95

The challenges cited with regard to the Teen evaluation and policy research within program Pregnancy Prevention program demonstrate the design. One challenge is that programs, policies, complexity of program evaluation. These challenges and regulations are generally designed without in- can make it difficult to keep the attention of leader- volving the evidence-building community. Engag- ship on evidence building over time. Even with an ing the evidence-building community as partners appropriate organizational structure and sufficient during the program design process will help en- staff capacity, Federal departments still may not sure that the program can be rigorously evaluated prioritize evidence building among the competing and that research questions will meet the needs demands on their resources without the right set of of program managers and policymakers. Barriers incentives and expectations. As the quote from Ms. to early collaboration between evaluators and pro- Kappeler suggests, developing and sustaining Fed- gram designers include limited evaluation capac- eral leadership in support of evidence building is a ity and expertise, program offices’ concerns about formidable undertaking. Senior department leaders how evaluation results will be used, and lack of play an important role in making the generation and access to quality data. Several public comments use of evidence routine in Federal policymaking. submitted to the Commission highlighted core el- Today, Federal departments lack individual ements that must be incorporated into program incentives for senior leaders to produce and use design, including the following:17 evaluation and policy-relevant research. While the Commission does not offer recommendations • Clearly stated goals, objectives, logic, outputs, to address individual motivation, it learned that and desired outcomes. departmental approaches could include incorporating measures into the performance reviews of • Recognition of roles for different types of career senior leaders that reflect their effective- analyses based on program, policy, or regula- ness in producing and using evidence. Similarly, tion stage and development (e.g., prospective, the Senate could use the confirmation process to pilots, implementation, retrospective, system- seek affirmative responses regarding a political atic reviews/meta-analysis). appointee’s support for producing and using evidence to inform decision-making. • Incentives for evidence production. The political process can itself introduce uncer- tainties about the value of evidence and its use in • Sufficient legislative authority or flexibility to policymaking. A report from the Bipartisan Policy enable evaluation, including mechanisms to Center observes that “the fragmented nature of encourage innovation and to conduct impact the legislative process, the clash of partisan inter- evaluations, using random assignment when ests, and the labyrinth of legislative procedures appropriate. and practices make it more difficult to establish a coordinated structure for using research and eval- • Incorporation of data collection needs and uations in a more systematic fashion in the leg- requirements; a focus on data quality to islative process.”16 Establishing clear expectations ensure meaningful and useful information is through reauthorizations of programs, develop- gathered in order to assess program costs and ment of regulations, the budgeting process, and outcomes. other routine interactions between the President, the Congress, and Federal departments will rein- Nearly all programs collect some form of ad- force the demand for more and better evidence. ministrative data about program beneficiaries or the services offered. How programs use that Program Design information, however, varies greatly. As the evidence-building community increasingly relies The statute that established the Commission’s on those data, programs and evidence building charge includes multiple references to embedding

17. Based on comments submitted to the Commission by J-PAL 16. Bipartisan Policy Center, Congress and Evidence-Based Policy- North America, Laura and John Arnold Foundation, and American making: Creating a 21st Century Legislature (Washington, D.C.: Bi- Evaluation Association; https://www.cep.gov/library/testimony.html partisan Policy Center, 2017): 13. (accessed August 10, 2017). 96 The Promise of Evidence-Based Policymaking

become increasingly interdependent. The infor- policy through required transactional reviews of mation collected by programs must be analyzed or agency information collection requests, systems made available to those who can interpret the data of record notices, and other related reporting in a manner that does not compete with, but rath- and by chairing interagency councils, such as the er enhances, programs’ ability to implement day- Privacy Council and the Interagency Council on to-day operations and achieve their intended goals Statistical Policy. The law also requires OIRA to en- in the most cost effective manner. Leadership that sure the sufficiency of budget proposals for statis- recognizes the value of evaluation and other evi- tical activities. These responsibilities coexist with dence building, and the role of administrative data OIRA’s regulatory review and coordination func- in those activities, can ensure that this approach tion, which accounts for the majority of OIRA’s becomes the norm. The Commission finds that ex- resources. In addition to OIRA, OMB also houses pectations for evaluation established by the Con- several other statutorily created offices that play a gress and the President can promote continuous role in evidence building. These other offices in- production of valid and reliable evidence about clude the CIO, who oversees information technol- programs and policies and their costs. ogy policies that have direct bearing on information privacy, confidentiality, metadata and Open Challenge #3: The current organizational Data; the Chief Performance Officer, charged in law with overseeing performance management structure of OMB does not optimize the activities within the Federal government; and agency’s ability to coordinate evidence the Office of Federal Procurement Policy, which building across the Federal government. helps set Federal contracting policies. OMB also houses a small team focused primarily on Federal Federal departments often struggle to coordinate program evaluation. These offices work together evidence-building activities within their own de- with OMB’s Resource Management Offices to pri- partments, and coordinating across departments oritize policy and resource allocations across the can be even more challenging. Building evidence Executive Branch. Each office reports to a different within topical silos like labor, transportation, and political appointee, and each can issue guidance education alone misses opportunities to address or directives to agencies that enable or discourage cross-cutting research and policy questions. For activities related to evidence building. example, questions about the impact of housing The capacity of OMB to effectively coordinate assistance on health outcomes or the impact of the Federal evidence-building community has food security on child development would nec- been complicated by the ways in which the roles essarily rely on the integration of data resources of these offices have evolved over time, resulting managed by two separate Federal departments. A in confusion or inconsistent guidance for agen- lack of coordination across government for evi- cies. For example, the PRA expressly directs that dence building leads to unnecessary burden and OIRA facilitate government-wide “information cost from duplicative data collection, missed op- management” policies.18 The E-government Act portunities for programmatic collaboration, and of 2002 fractured implementation responsibili- a less robust response to a crosscutting policy or ties within OMB by establishing a separate office programmatic question. charged with responsibility for all information technology policy, overlapping with the infor- The Critical Role of OMB mation policy function in OIRA. Perhaps more problematic, many agency organizational struc- OMB serves as the central coordinator for many tures developed to mirror OMB’s organizational government-wide processes, but currently, re- arrangement. As the demand for evidence to sup- sponsibilities related to evidence building are dis- port the policymaking process continues to grow, persed across the organization without consistent, the operational silos within OMB will likely only sustained internal coordination. By law, the Office become more constraining for the timely produc- of Information and Regulatory Affairs (OIRA) is tion of evidence across government. the unit within OMB responsible for developing government-wide information, privacy, confidentiality, and statistical policy. It implements such 18. Paperwork Reduction Act, 44 USC § 3503 (1995). The Promise of Evidence-Based Policymaking 97

Due to OMB’s current organizational structure, Coordinating Evidence Building Across departments typically need to coordinate with Federal Departments multiple offices within OMB before launching new evidence-building activities. This structure In addition to improving the organizational struc- places the burden of coordination on individual ture of OMB, the Commission finds that interagen- departments, rather than OMB, and further chal- cy coordination is vital for the efficient operation lenges the ability to coordinate throughout the of the Federal evidence-building community (see evidence-building community. Because OMB is the box “Why Is Coordinating Evidence Building the hub of evidence and information policy in the across Government Important?”). OMB already Federal government, the Commission finds that convenes and participates in several interagency fragmentation of its evidence-building functions coordination groups related to evidence building. hampers its ability to sufficiently prioritize and One key group, the Interagency Council on Statis- coordinate evidence building. The Commission tical Policy,19 has a statutory role in advising OMB finds further that ensuring OMB is structured to on the needs and policies of the statistical system. effectively coordinate the evidence-building community is crucial for realizing the full benefits of the Commission’s recommendations. 19. Paperwork Reduction Act, 44 USC § 3504(e)(8) (1995).

Why is Coordinating Evidence Building Across Government Important?

Coordination of evidence-building activities dents attending them. VA tracks colleges using across government is particularly important VA “Facility Codes,” which have no relation to when a research or policy question spans mul- either of the two different methods of track- tiple topical domains. For example, to under- ing colleges used by ED. ED officials have spent stand the impact of federal student aid and years trying to build a crosswalk between the the GI Bill on veteran educational outcomes, two separate systems within the department one approach would be to combine data from for tracking colleges, and currently, both ap- both the U.S. Department of Education (ED) pear in ED’s Integrated Postsecondary Educa- and the U.S. Department of Veterans Affairs tion Data System database. Crosswalking the (VA). However, both agencies track veterans data collected by ED and VA is even more com- who are students differently. Specifically, ED plicated, and is especially difficult in the case tracks student outcomes, but cannot identify of for-profit education companies with numer- which students in its database are veterans, in ous campus and online locations. Harmonizing part because of a skip pattern introduced on the VA and ED data collections to permit the ED’s Free Application for Federal Student Aid tracking of colleges and the students who at- around the same time that the Post 9/11 GI Bill tend them is important to understanding the was enacted, causing most students not to be success of hundreds of billions of dollars of asked if they are veterans. The exact opposite taxpayer investment in Federal student aid and is true at the VA. The VA tracks which veterans the GI Bill.1 are using the GI Bill, but does not track student outcomes, at least not robustly. The problem becomes more prominent 1. Carrie Wofford, Veterans Education Success; Mark Schnei- when one considers the differing methods by der, American Institutes for Research, Commission Public which ED and VA track colleges and the stu- Hearing, Washington D.C., October 21, 2016. 98 The Promise of Evidence-Based Policymaking

Additional coordinating committees, including the eral government awarded just under a half trillion Privacy Council, Performance Improvement Coun- dollars in contracts to entities to carry out work cil, and Interagency Council on Evaluation Policy, on behalf of the Federal government.23 Contracts emerged more recently.20 These groups provide are an important mechanism used by Federal de- valuable communities of practice within the Fed- partments to carry out research and evaluation eral government to support knowledge generation, activities. Across government, numerous offices problem solving, best practice dissemination, and often identify contract procurement as a barrier shared services. The Commission encourages the to the efficient production of evidence. In the CEP continuation and prioritization of the leadership Survey of Federal Offices, 40 percent of responses of these Councils in order to bring maximum value identified lack of ability to execute and manage to the Federal evidence-building community. contracts as a barrier to their ability to use data for evidence building, including 17 percent who identified this as a moderate or major barrier.24 Challenge #4: Administrative processes One challenge with contracts relates to how are not tailored or aligned to support evi- to best structure a procurement at the outset of dence-building activities. a project to support evidence building. This challenge can be addressed through better applica- During the Commission’s fact-finding phase, both tion of existing flexible guidance related to pro- governmental and non-governmental witnesses curement applied to research and development identified numerous administrative barriers within contracts.25 The Commission finds that existing the Federal government that hamper the efficient regulations acknowledge that certain types of production of evidence. In a follow-up to the Com- services, specifically those identified as research mission public meeting held in November 2016, and development, may be more appropriately the Interagency Council on Evaluation Policy sub- procured with a more flexible, yet riskier, con- mitted a list of the top barriers to increasing Feder- tract. The Federal Acquisition Regulation states al evidence-building capacity, highlighting bureau- “because of the importance of technical consider- cratic barriers that “discourage evaluation, create ations in research and development, the choice of inefficiencies and impose additional costs when contract type should be made after obtaining the conducting a Federal evaluation, particularly issues recommendations of technical personnel.”26 Ap- related to PRA, interagency agreements, and pro- plying this standard consistently throughout the curement.” 21 Evidence building requires support evidence-building community holds the potential from administrative functions, but these functions to improve the government’s ability to issue con- are not always aligned or tailored to fulfill the par- tracts that are properly structured to support evi- ticular needs of the evidence-building community. dence-building activities. A second challenge relates to staff familiarity Procurement with and ability to manage contracts that support Procurement is the process by which government evidence-building activities. Acquisition staff who acquires supplies and services using funding ap- may be unfamiliar with contracts that are not propriated by the Congress, including through structured as the more typical fixed-price variety grants and contracts.22 In fiscal year 2016, the Fed- and with the flexibilities available under research and development contracts may seek to limit an agency’s preference to use a riskier contract type in 20. The Performance Improvement Council was established support of evidence-building activities. Similarly, by Executive Order 13450 in 2007 and was later codified in the GPRA Modernization Act of 2010. The Interagency Council on Evaluation Policy was established administratively in 2015. The 23. For the total value of awarded contracts (as well as grants, Privacy Council was established by executive order in 2016; loans, and other assistance) made by the Federal government by https://obamawhitehouse.archives.gov/the-press-office/2016/02/09/ fiscal year, go towww.usaspending.gov . executive-order-establishment-federal-privacy-council (accessed Au- gust 10, 2017). 24. Includes offices that reported that they collect or use data for statistics, evaluation, research, or policy analysis or spend a por- 21. Interagency Council on Evaluation Policy, Comments to the tion of their budget for such purposes. Commission, “‘Top-Five List’ of Issues and Solutions Related to Federal Evaluation Activity,” November 22, 2016. 25. Federal Acquisition Regulation, 48 C.F.R. § 35.000 (2005). 22. Federal Acquisition Regulation, 48 C.F.R. § 2.101 (2005). 26. Federal Acquisition Regulation, 48 C.F.R. § 35.006 (2005). The Promise of Evidence-Based Policymaking 99

technical staff responsible for overseeing the day- be administered under the collection, also must to-day management of the contract may be unfa- be made available for two public comment periods miliar with proper techniques for managing the of 60 and 30 days during this sequential process. performance of a riskier contract type. Differing Completing these steps requires four months at a goals between acquisition professionals and tech- minimum, but six to nine months is considered a nical staff may lead to difficulties in identifying more realistic time frame for approval, including the right contract type and raise concerns about incorporation of public comments as well as nec- the capacity of the government to properly man- essary departmental and OMB clearances.29 age the contract. During the Commission’s fact-finding phase, members of the evidence-building community Information Collection Reviews from both inside and outside the Federal government mentioned that PRA requirements are The PRA directs OMB to coordinate requests for burdensome and time-consuming. The Laura information collections initiated by Federal de- and John Arnold Foundation noted that the PRA partments.27 The law requires that any collection processes cause “long delays and bottlenecks for for ten or more individuals from a survey, ques- agencies seeking approval of evaluations that have tionnaire, or form undergo a review to ensure stymied efforts by agencies to increase the num- collected information achieves public benefit and ber of high-quality studies of important research maximum utility. The vast majority of data collec- questions.” 30 Six of the 10 evaluation offices that tions undertaken by the evidence-building com- responded to the CEP Survey of Federal Offices munity fall within the scope of the PRA, as do noted that information collection requirements administrative data collections undertaken, spon- acted as a moderate or major barrier to their evi- sored, or required by Federal departments. The dence-building activities. purposes embodied by the PRA are important for There are two primary areas of concern for ensuring that Federal information collections are those who identify the Information Collection Re- necessary, useful, and of high quality. However, quest review and approval process as a barrier to the Commission finds that the current structure engaging in evidence building with maximum effi- of the information collection review and approv- ciency. The first issue relates to the broad net cast al process can be inefficient. These inefficiencies by the PRA regarding the size of information col- lead to problematic delays in data collection and lections requiring OMB review and approval. The challenges to OMB’s ability to maximize the coor- low threshold number of respondents that trig- dination and transparency of Federal information gers the Information Collection Request clearance collections. process (10 or more) means most collections the By law, there are three primary actors involved Federal government wishes to undertake require in processing an information collection for ap- OMB review and approval; few collections for ev- proval. The agency wishing to sponsor the collec- idence building can be implemented with fewer tion prepares an Information Collection Request, than 10 respondents. The broad definition of what which must meet the standards outlined in the constitutes an “information collection” means PRA, including elements such as the need, the that almost every effort to collect standardized in- plan, and the public burden estimated for the col- formation from the public, including information lection. The departmental CIO is responsible for from grantees spending Federal funds, triggers certifying that the collection meets these stan- the Information Collection Request review and dards prior to submitting the package to OMB. OMB approval process. is responsible for review and approval of the pack- The second issue is the length of time required age.28 The Information Collection Request, along to gain approval for an information collection. with the proposed data collection instruments to Delays prior to starting data collection may be

27. Paperwork Reduction Act, 44 USC § 3501–3520 (1995). 29. Stuart Shapiro, “The Paperwork Reduction Act: Benefits, Costs, and Directions for Reform,” Government Information Quarterly 30 28. The 1995 amendments to the PRA used the phrase “senior of- (2013): 204–210. ficial,” which was later changed to Chief Information Officer in the Clinger-Cohen Act (Public Law 104–106, February 10, 1996; 30. Letter submitted to the Commission by the Laura and John Public Law 104–208, September 30, 1996). Arnold Foundation. 100 The Promise of Evidence-Based Policymaking

compounded by Institutional Review Board reproduced. The Commission believes that a respon- views also required for certain types of research. sible investment of resources in more and better These delays often mean that the window of op- evidence holds the potential to yield substantial portunity passes for collecting critical baseline savings in the longer term as programs that are data from participants in a new program and also improved become more cost-effective, and as pro- can inhibit the ability of agencies to gather re- grams that are not effective are discontinued. al-time information about program operations. As Katherine O’Regan, former Assistant Secretary “Using evidence to improve at HUD, noted in her testimony to the Commis- government is what taxpayers sion, if the collection of such information cannot take place in a timely manner, agencies are “left to expect—smart and careful use of make policy decisions and program changes with limited resources to best address very little information; often reacting to anecdote national priorities. rather than a more complete picture.”31 ” Within OMB, OIRA is tasked with the review – Analytical Perspectives, Budget of the and approval of new information collections. Be- United States Government, Fiscal Year 2018 cause of the high priority of the regulatory function within OIRA, the swift review of new collec- The Commission recognizes that resource pri- tions often is not a priority. The statutory language oritization is essential to ensuring the goals of of the PRA intentionally casts a broad net to en- the Commission are achieved, but it is not rec- sure that the majority of information collections ommending an infusion of large sums of funding undertaken by the Federal government are includ- to create new agencies or to launch massive new ed in the required review and approval process. evidence-building endeavors. Instead, throughout However, flexibilities currently exist within the this report, the recommendations of the Commis- PRA statute that could be appropriately applied sion balance the need to prioritize evidence build- for evidence building activities. For example, the ing while recognizing fiscal constraints. In some PRA provides OMB with the ability to delegate departments, sufficient resources already exist to some authority to approve proposed collections of enable evidence building, though such resources information.32 Exploring opportunities to delegate may have use restrictions that inhibit the most authorities could create an opportunity to focus cost-effective approach for evidence building. In additional attention within OMB on other purpos- several instances shared with the Commission, es and benefits of the PRA, such as coordinating restrictive funding procedures appear to have information collections across government and hampered or terminated otherwise positive evi- improving quality. dence-building practices. Specifically, with regard to the procurement challenges discussed previ- Challenge #5: The Federal evidence-build- ously, the lack of funds that are available across multiple years is one considerable barrier to carrying community has insufficient resources ing out long-term projects. Evidence-building ac- and limited flexibilities that restrict the tivities are generally today considered “non-sev- ability to expand evidence-building activ- erable services” under Federal appropriations law, ities. which means that they must be fully funded up- front or structured as a multi-year contract. De- Through the course of the Commission’s research partments seldom have sufficient funding to fully and deliberations, the topic of resources repeat- fund the entire cost of large or complex studies edly emerged as a major perceived need for the up front; rather, departments must break those evidence-building community and a challenge contracts into phases. Doing so, however, requires for improving the volume and quality of evidence considerable extra staff time and additional costs to the government and its contractors. Forcing departments to issue multiple contracts to conduct 31. Katherine O’Regan, HUD, Commission Meeting, Washington, a single evaluation or research project distracts D.C., November 4, 2016. from efficiently implementing projects aimed at 32. Paperwork Reduction Act, 44 USC § 3507(i)(1) (1995). generating more and better evidence. The Promise of Evidence-Based Policymaking 101

Some Federal departments struggle to identi- than hinder, evidence-building and sufficient re- fy resources that can be made available for evi- sources and flexibilities must be provided to sup- dence-building purposes. Flexibilities in funding port the entire enterprise. To build capacity across have allowed other departments to pursue evi- the Federal evidence-building community, the dence-building activities within existing resourc- Commission identified five recommendations: es. Several departments have the legal authority to set-aside, or allocate, a set amount of funding for REC. 5-1: The President should direct their evidence-building budgets that can be trans- Federal departments to increase ca- ferred across budget accounts. The explicit transfer authority in appropriations is one approach to pacity for evidence building through the reducing the need for burdensome efforts to justi- identification or establishment of a Chief fy budget transfer under general authorities, such Evaluation Officer, in addition to needed as the Economy Act. DOL, for example, receives authorities to build a high performing evi- funding available for multiple years for evalua- dence-building workforce. tion through the set-aside and transfer authority in its appropriation, with a requirement to notify appropriations committees about evaluations Ensuring the routine and continuous production of to be conducted.33 The Department of Justice and evidence to support policy decisions and program HHS also have programs that include small fund- administration will require the capacity to leverage ing set-asides for evidence building. Agencies each evidence-building function—statistics, evalu- like the Social Security Administration create a ation, and policy research—within Federal depart- reliable funding stream for information technol- ments. Today, some combination of these functions ogy improvements through set-asides. The same exists in all major departments, including through strategy could be used to support the full suite of collaborations with non-governmental partners, evidence-building activities, including data collec- but these functions are not necessarily operating tion and curation, policy-relevant research, and at the level needed. This recommendation directs evaluation. Federal departments to establish the capacity to undertake the full range of evidence-building ac- Recommendations tivities through internal human resource strategies and by leveraging partnerships with external The Commission identified a series of opportu- partners. Federal departments should also con- nities to build and maintain a strong Federal in- duct a regular inventory of units that perform evi- frastructure for the sustained production and use of dence-building functions across the department as evidence. To maximize the generation of evidence, part of their strategic planning process. the Federal evidence-building community must To advance the goal of strengthening the gov- support a full range of analytic approaches, in- ernment’s more nascent program evaluation func- cluding statistics, evaluation, and policy research. tion, the Commission recommends that Federal Such expertise may be established within Federal departments identify or establish a Chief Evalu- departments, and may be strengthened through ation Officer. A Chief Evaluation Officer can help partnerships with the non-Federal evidence-build- lead efforts to coordinate the department’s evi- ing community. To achieve the greatest gains, ev- dence-building activities. Further, OPM should idence building must be well-coordinated both take immediate steps to support the growth and within and across departments. Strong leadership development of the program evaluation field, in- that prioritizes evidence building and creates the cluding the establishment of occupational struc- demand for evidence is vital for institutionaliz- tures to address critical needs regarding technical ing these functions within departments and en- expertise and contract management. One approach suring coordination across the evidence-building to address these staffing challenges would be for community. In addition, administrative functions OPM to collaborate with agencies that already must be aligned and tailored to support, rather have high-functioning program evaluation offices to identify human capital strategies to support 33. Division H, Section 107, Consolidated Appropriations Act, the development of the evaluation field. For newer 2017 (Public Law 115–31). evaluation offices that may be struggling to grow 102 The Promise of Evidence-Based Policymaking

their capacity, OPM could identify hiring strategies learning agendas through routine strategic plan- focused on recruiting and retaining professionals ning reduces the burden for identifying knowledge with the specialized skills needed to directly exe- gaps and helps align evidence building with depart- cute evaluations on behalf of the Federal governmental strategic priorities. The resulting document ment or manage evaluation contracts. Such strate- also helps to clearly communicate the high-priority gies might include the establishment of a targeted research and policy questions of a department both occupational series that would allow departments to the public and to the many different actors with- flexibility to hire based on specific criteria most in the evidence-building community. relevant for the position and the function. The Congress and the President should pro- Tools and mechanisms to leverage external vide sufficient and appropriate authority for de- partnerships and build agency capacity should also partments to design programs and policies that be made available to Federal agencies in the evi- enable a portfolio of evidence to support contin- dence-building community, including grant-mak- uous learning and information needed to ensure ing authority and cooperative agreement authority, accountability. The Commission acknowledges to encourage the use of partnerships with founda- the value of research and evaluation that takes tions, universities, and others. The President should place at various points throughout the lifecycle of also encourage the increased use of programs, such a program, as well as the broad variety of research as the IPA program, that allow agencies to bring on methods that may be employed to address differ- temporary personnel from outside the Federal gov- ent types of research questions. When appropriate ernment to help expand capacity and advance the and feasible, randomized controlled trials should department’s learning agenda. be conducted to understand program impacts. Federal departments should be encouraged to REC. 5-2: The Congress and the Presi- routinely evaluate programs and policies, including their cost effectiveness. Continuous improvement dent should direct Federal depart- can be facilitated by supporting a cycle of first pilot ments to develop multi-year learning agen- testing a new program, policy, or regulation, then das that support the generation and use of conducting research to learn from the pilot test, and evidence. finally, adapting the program, policy, or regulation based on what was learned through the research. In The supply of evidence to support policymaking is establishing new programs in law or reauthorizing more likely to increase when there are consistent existing programs, the Congress and the President signals from policymakers that the production of should strongly encourage that pilot and demon- evidence is a priority. The Congress and the Pres- stration projects are evaluated, and provide flexi- ident can take steps to encourage departments bility to design policies that allow experimentation, to develop more and better evidence. As a start- including phased implementation options. ing point, the Congress and the President should encourage Federal departments to inventory the REC. 5-3: The Congress and the Presi- units responsible for various evidence-building dent should direct the Office of Man- activities to ensure that each unit is recognized and operating in concert with other units engag- agement and Budget (OMB) to coordinate ing in evidence-building activities. Such an inven- the Federal government’s evidence-build- tory could be developed through the regular qua- ing activities across departments, includ- drennial strategic planning process. ing through any reorganization or consoli- The President should encourage Federal depart- dation within OMB that may be necessary ments to develop multi-year learning agendas in consultation with program and evidence-building and by bolstering the visibility and role of units. The learning agenda should be updated on interagency councils. a regular basis and contain both short- and long- term evidence-building priorities for each depart- The Commission recognizes the President’s and ment. Through the development of learning agen- OMB’s prerogative in organizing and optimizing das, departments should proactively identify where OMB’s structure and resources. Implementing a the need for more evidence is greatest. Developing government-wide vision for evidence building The Promise of Evidence-Based Policymaking 103

consistent with the Commission’s vision, though, To generate a greater volume of evidence in a may require substantial changes in how govern- more efficient manner, foundational adminis- ment operates. The Commission believes that trative processes must be aligned and tailored having this effort championed at OMB is critical. to better support evidence building. The Com- Many of the Commission’s recommendations rely mission identified a specific set of actions relat- implicitly on the integration of evidence building ed to procurement and streamlining the review functions within departments. Different compo- and approval processes for new data collections nents of the evidence-building community, in- that would require little cost, but offer substan- cluding statistical agencies, evaluation offices, pri- tial benefits and savings while making it easier to vacy offices, and performance management units produce evidence. Specifically, OMB should clarify must increasingly work together to realize the the applicability of the research and development Commission’s vision. Because many department procurement policies to contracts that support ev- structures for implementing evidence-building ac- idence-building activities, and should support the tivities mirror the structures at OMB, careful con- establishment of training for technical staff who sideration should be given to whether a consolida- are responsible for managing research and devel- tion of activities necessary for evidence building opment contracts that support evidence building. should occur in the near term at OMB. A focus on In partnership with this approach, training should evidence building can get crowded out by other be developed for technical staff responsible for priorities. The President and OMB should careful- managing evidence-building contracts, including ly consider how a greater commitment to founda- effective management techniques for oversight tional, critical information policy setting and co- and accountability of riskier service contracts. An ordination responsibilities and resources can be additional strategy would be to establish a Gov- achieved, in light of the significant, high stakes ernment-wide Acquisition Contract (GWAC) for and fast-moving regulatory responsibilities with- use in issuing contracts for evidence-building ac- in OIRA, responsibilities that currently comprise tivities and services. the overwhelming majority of OIRA’s work. This The Congress and the President also should includes any reorganizations or required statutory take steps to streamline the PRA’s review and changes that could improve the impact of OMB’s approval process for Information Collection Re- investment in evidence-based policymaking. quests, while ensuring that information coordi- Efficiently implementing evidence-building nation, transparency, and data quality remain activities across government requires a strong co- central features of PRA. OMB should pilot ap- ordination function to address cross-cutting re- proaches for better meeting the needs of the ev- search and policy questions, minimize duplicative idence-building community in conducting ICR efforts, and reduce the burden on the public. For reviews, including expanding the use of the del- the Federal government to maximize resources egated authority already authorized under the available for evidence building across government, PRA. The process of review and approval internal OMB must provide leadership that spans the Ex- to Federal departments might also be strength- ecutive Branch. In addition, OMB’s role in leading ened by removing the departmental CIO from the many interagency councils must be strengthened ICR review and approval process and instead as- to ensure these efforts are appropriately visible signing this responsibility to the senior data poli- and influential in improving how government cy official proposed in Recommendation 3-3. At a agencies collaborate. minimum, OMB should issue guidance to Feder- al agencies describing the flexibilities under the REC. 5-4: The Congress and the Pres- PRA, including the process for acquiring delegat- ident should align administrative ed approval authority, and strengthening depart- processes to support evidence building, in mental capabilities to design rigorous studies to advance evidence building. particular by streamlining the approval The Congress should consider additional statu- processes for new data collections and us- tory changes to the PRA to streamline the review ing existing flexibilities in procurement process. These include assessing the rationale for policy. the current standard under the PRA that requires 104 The Promise of Evidence-Based Policymaking

the review and approval of all new data collections Appropriators can further support evidence build- of 10 or more respondents, and considering short- ing that often spans the domain of multiple poli- ening the initial public comment period for new cies and jurisdictional silos by enabling the transfer collections from 60 days to 30 days. of funding across the evidence-building community. The Congress and the President should enable REC. 5-5: The Congress and the Presi- and encourage transfers across budget accounts that support multi-departmental evidence-build- dent should ensure sufficient resourc- ing needs. Explicit transfer authority in appropri- es to support evidence-building activities ation bills will enable departments to spend less about Federal government programs and time justifying transfers under general authorities policies. (e.g., the Economy Act), and enable them to focus on the production of evidence. Insufficient resources for evidence building can be In establishing new programs in law or reau- perceived as an insurmountable limitation to the thorizing existing programs, the Congress and the pursuit of more and better evidence about gov- President should enable the use of new set-aside ernment programs and policies. There are several authorities of up to 1 percent of program admin- actions that should be taken by the Congress and istration resources to support the full suite of evi- the President to signal the importance of evidence dence-building activities, including data collection building within Federal departments, and set the and curation, policy-relevant research, and evalu- expectation for the production and application of ation. In addition, the Congress and the President evidence through the provision of flexibilities or a should establish and grant Federal departments commitment of resources. access to Evidence Incentive Funds to supplement The Congress, through the appropriations pro- the production of future research, evaluation, and cess, and the President should provide depart- related activities identified in departmental learn- ments in active pursuit of a learning agenda access ing agendas. Evidence Incentive Funds in each de- to multi-year funding to pursue articulated evi- partment are conceptualized by the Commission dence-building priorities. The availability of multi- to operate similarly to Working Capital Funds or year funds for departments that establish learning Salary and Expense accounts. The funds could be agendas both creates an incentive for the genera- created by taking up to 10 percent of unobligated tion of learning agendas and enables departments balances at the end of a fiscal year to be allocated to more efficiently contract for multiyear studies.34 for future evidence-generating activities. ■

34. Interagency Council on Evaluation Policy, Comments to the Commission, “‘Top-Five List’ of Issues and Solutions Related to Federal Evaluation Activity,” November 22, 2016.

The Promise of Evidence-Based Policymaking 105

Conclusion: Possibilities with More and Better Evidence

he Commission’s recommendations pres- in all evidence-building activities. The risks of T ent a comprehensive strategy for addressing re-identifying individuals in data used for evi- the greatest problems facing evidence building dence building will be continuously reviewed and today: unintentional limits on data access, inad- mitigated. The evidence-building community will equate privacy practices, and insufficient capacity have avenues for secure access to the information to generate the amount of quality evidence need- they need to answer questions from policymak- ed to support policy decisions. The Congress, the ers, program administrators, and the public. The President, and the American people are ill-served institutions of government will further enhance by this state of affairs. The Commission believes their systems for stewarding data and enabling ev- that fully implementing the Commission’s recom- idence building. Principal Statistical Agencies will mendations will lead to substantial progress in ad- take on increased leadership within the Federal dressing these challenges, enabling more and bet- government in facilitating the secure statistical ter evidence for our society, generated in a more use of administrative data. Program offices with- secure fashion. in the Federal government, state officials, and the Over the past century, numerous commissions public will demand and use statistical analysis to and panels offered strategies to improve the U.S. support routine decision-making. Perhaps most evidence-building system. The changes made in importantly, there will be greater accountability response to their recommendations have not kept and transparency about evidence-building ap- pace with the need. Twenty years ago, many of the proaches and uses of data for the Congress, the recommendations included in this Commission’s President, and the American public. strategy could not have been feasibly implement- In developing recommendations, the Commis- ed. In this report, the Commission proposes a sion relied on five guiding principles related to the modernization of the country’s evidence-produc- concepts of privacy, rigor, transparency, humility, ing capacity that uses available approaches and and capacity (described in Chapter 1). Each of that incorporates new technologies and methods these principles is reflected in the recommenda- as they come on line. This strategy will enrich the tions developed by the Commission for building capabilities for producing evidence in the U.S. and and using evidence to improve our society. The ensure a formidable ability to use evidence to im- principle for respecting individual privacy and prove government policies. confidentiality is captured in virtually every rec- The Commission is confident that, with full ommendation in this report, particularly those in and ongoing implementation of the Commis- Chapters 2, 3, and 4 that seek to improve priva- sion’s recommendations, privacy-protective ap- cy protections. The principle of rigor encourages proaches will continuously improve to ensure well-designed and well-implemented methods; the confidentiality of individuals’ information improved access to administrative data to answer 106 The Promise of Evidence-Based Policymaking

important research questions, as recommended in ment spending be treated differently? Taxpayers Chapters 2 and 3, advances this principle. The rec- and policymakers should receive credible infor- ommendations in Chapter 5 concerning learning mation to know and understand how well the agendas and portfolios of evidence as tools for ad- programs and policies they fund achieve their in- dressing policymakers’ questions are grounded in tended goals. the principle of humility in the development and Generating and using evidence to inform pol- use of evidence. In all of the recommendations, icymaking and program administration is not a the Commission encourages increased transpar- partisan issue. The strategy described in this re- ency about evidence-building efforts, particularly port offers a responsible approach to improving with the new transparency portal recommended how government officials, private researchers, in Chapter 4. In addition to the creation of the foundations, non-profits, the business communi- National Secure Data Service which will enable ty, and the public interact to make sure govern- greater capacity for the entire evidence-building ment delivers on its promises. community, Chapter 5 includes numerous recom- All of these recommendations will depend on mendations that seek to improve government’s the leadership of the President and the Congress capacity for evidence building while encouraging in calling for credible evidence to support policy the use of information. decisions throughout government. Whether mak- The Commission’s recommendations recognize ing decisions on funding allocations, assessing the complexity of improving evidence about gov- new regulations, or understanding how to im- ernment programs and policies. The Commission’s prove processes for efficiently providing services, recommendations seek to create an environment evidence is needed in every decision made by gov- that enables routine production of evidence to ernment officials—career civil servant, political meet the country’s informational needs. And appointee, or elected official. Without the use of the Commission’s recommendations embody the evidence in our democracy, we are only guessing spirit of bipartisan cooperation. at whether government programs and policies are People need credible information to inform achieving their intended goals. their actions. We review ratings on websites be- This is a milestone moment. The Congress and fore making online purchases. We inspect homes the President should seize it by working togeth- before moving in. We seek second opinions before er to enact the laws and develop the regulations major surgeries. Businesses rely on data and anal- necessary to implement the Commission’s recom- ysis to make their decisions. Why should govern- mendations. ■ The Promise of Evidence-Based Policymaking 107

Acronyms

ACF: Administration for Children and Families (U.S. HUD: U.S. Department of Housing and Urban Department of Health and Human Services) Development

ASPE: Office of the Assistant Secretary for Planning ICR: Information Collection Request and Evaluation (U.S. Department of Health and Human Services) IES: Institute of Education Sciences (U.S. Department of Education) CARRA: Center for Administrative Records Research and Applications (Census Bureau, U.S. Department IPA: Intergovernment Personnel Act of Commerce) LEHD: Longitudinal Employer-Household Dynamics CASD: Centre d’Accès Sécurisé aux Données Program (“Secure Access Data Center” – France) NCES: National Center for Education Statistics (U.S. CEP: U.S. Commission on Evidence-Based Department of Education) Policymaking NDNH: National Directory of New Hires CES: Center for Economic Studies (Census Bureau, NSDS: National Secure Data Service (proposed by U.S. Department of Commerce) recommendations) CIO: Chief Information Officer NSF: National Science Foundation CIPSEA: Confidential Information Protection and OIRA: Office of Information and Regulatory Affairs Statistical Efficiency Act of 2002 (Office of Management and Budget, Executive CMS: Centers for Medicare and Medicaid Services Office of the President) (U.S. Department of Health and Human Services) OMB: U.S. Office of Management and Budget DARE: Drug Abuse Resistance Education (Executive Office of the President)

DOL: U.S. Department of Labor OPM: U.S. Office of Personnel Management

ED: U.S. Department of Education OPRE: Office of Planning, Research, and Evaluation (Administration for Children and Families, U.S. ED-DRB: disclosure review board at the U.S. Department of Health and Human Services) Department of Education PII: Personally identifiable information eMOU: Enterprise Memorandum of Understanding PRA: Paperwork Reduction Act of 1995 FIPPs: Fair Information Practice Principles PSA: Principal statistical agency FITARA: Federal Information Technology Acquisition Reform Act SMC: Secure Multiparty Computation

FNS: Food and Nutrition Service (U.S. Department of SNAP: Supplemental Nutrition Assistance Program Agriculture) TANF: Temporary Assistance for Needy Families FSRDC: Federal Statistical Research Data Center UI: Unemployment Insurance FTI: Federal tax information USDA: U.S. Department of Agriculture GPRA: Government Performance and Results Act of VA: U.S. Department of Veterans Affairs 1993 WIC: Special Supplemental Nutrition Program for GWAC: Government-wide Acquisition Contract Women, Infants, and Children HHS: U.S. Department of Health and Human WIOA: Workforce Innovation and Opportunity Act Services of 2014

The Promise of Evidence-Based Policymaking 109

Appendix A: Evidence-Based Policymaking Commission Act of 2016

PUBLIC LAW 114–140—MAR. 30, 2016 130 STAT. 317

Public Law 114–140 114th Congress An Act To establish the Commission on Evidence-Based Policymaking, and for other pur- Mar. 30, 2016 poses. [H.R. 1831] Be it enacted by the Senate and House of Representatives of the United States of America in Congress assembled, Evidence-Based Policymaking SECTION 1. SHORT TITLE. Commission Act This Act may be cited as the ‘‘Evidence-Based Policymaking of 2016. Commission Act of 2016’’. SEC. 2. ESTABLISHMENT. There is established in the executive branch a commission to be known as the ‘‘Commission on Evidence-Based Policymaking’’ (in this Act referred to as the ‘‘Commission’’). SEC. 3. MEMBERS OF THE COMMISSION. (a) NUMBER AND APPOINTMENT.—The Commission shall be com- prised of 15 members as follows: (1) Three shall be appointed by the President, of whom— President. (A) one shall be an academic researcher, data expert, or have experience in administering programs; (B) one shall be an expert in protecting personally- identifiable information and data minimization; and (C) one shall be the Director of the Office of Manage- ment and Budget (or the Director’s designee). (2) Three shall be appointed by the Speaker of the House of Representatives, of whom— (A) two shall be academic researchers, data experts, or have experience in administering programs; and (B) one shall be an expert in protecting personally- identifiable information and data minimization. (3) Three shall be appointed by the Minority Leader of the House of Representatives, of whom— (A) two shall be academic researchers, data experts, or have experience in administering programs; and (B) one shall be an expert in protecting personally- identifiable information and data minimization. (4) Three shall be appointed by the Majority Leader of the Senate, of whom— (A) two shall be academic researchers, data experts, or have experience in administering programs; and (B) one shall be an expert in protecting personally- identifiable information and data minimization. (5) Three shall be appointed by the Minority Leader of the Senate, of whom—

110 The Promise of Evidence-Based Policymaking

130 STAT. 318 PUBLIC LAW 114–140—MAR. 30, 2016

(A) two shall be academic researchers, data experts, or have experience in administering programs; and (B) one shall be an expert in protecting personally- identifiable information and data minimization. (b) EXPERTISE.—In making appointments under this section, consideration should be given to individuals with expertise in economics, statistics, program evaluation, data security, confidentiality, or database management. President. (c) CHAIRPERSON AND CO-CHAIRPERSON.—The President shall select the chairperson of the Commission and the Speaker of the House of Representatives shall select the co-chairperson. Deadline. (d) TIMING OF APPOINTMENTS.—Appointments to the Commis- sion shall be made not later than 45 days after the date of enactment of this Act. (e) TERMS; VACANCIES.—Each member shall be appointed for the duration of the Commission. Any vacancy in the Commission shall not affect its powers, and shall be filled in the manner in which the original appointment was made. (f) COMPENSATION.—Members of the Commission shall serve without pay. (g) TRAVEL EXPENSES.—Each member of the Commission shall be allowed travel expenses, including per diem in lieu of subsistence, at rates authorized for employees of agencies under subchapter I of chapter 57 of title 5, United States Code, while away from their homes or regular places of business in the performance of services for the Commission. SEC. 4. DUTIES OF THE COMMISSION. Recommenda- (a) STUDY OF DATA.—The Commission shall conduct a com- tions. prehensive study of the data inventory, data infrastructure, database security, and statistical protocols related to Federal policymaking and the agencies responsible for maintaining that data to— (1) determine the optimal arrangement for which administrative data on Federal programs and tax expenditures, survey data, and related statistical data series may be integrated and made available to facilitate program evaluation, continuous improvement, policy-relevant research, and cost-benefit analyses by qualified researchers and institutions while weighing how integration might lead to the intentional or unintentional access, breach, or release of personally-identifiable information or records; (2) make recommendations on how data infrastructure, database security, and statistical protocols should be modified to best fulfill the objectives identified in paragraph (1); and (3) make recommendations on how best to incorporate outcomes measurement, institutionalize randomized controlled trials, and rigorous impact analysis into program design. (b) CLEARINGHOUSE.—In undertaking the study required by subsection (a), the Commission shall— (1) consider whether a clearinghouse for program and survey data should be established and how to create such a clearinghouse; and Evaluation. (2) evaluate— (A) what administrative data and survey data are relevant for program evaluation and Federal policy-making and should be included in a potential clearinghouse;

The Promise of Evidence-Based Policymaking 111

PUBLIC LAW 114–140—MAR. 30, 2016 130 STAT. 319

(B) which survey data the administrative data identified in subparagraph (A) may be linked to, in addition to linkages across administrative data series, including the effect such linkages may have on the security of those data; (C) what are the legal and administrative barriers to including or linking these data series; (D) what data-sharing infrastructure should be used to facilitate data merging and access for research purposes; (E) how a clearinghouse could be self-funded; (F) which types of researchers, officials, and institutions should have access to data and what the qualifications of the researchers, officials, and institutions should be; (G) what limitations should be placed on the use of data provided; (H) how to protect information and ensure individual privacy and confidentiality; (I) how data and results of research can be used to inform program administrators and policymakers to improve program design; (J) what incentives may facilitate interagency sharing of information to improve programmatic effectiveness and enhance data accuracy and comprehensiveness; and (K) how individuals whose data are used should be notified of its usages. (c) REPORT.—Upon the affirmative vote of at least three-quar- Recommenda- ters of the members of the Commission, the Commission shall tions. submit to the President and Congress a detailed statement of its findings and conclusions as a result of the activities required by subsections (a) and (b), together with its recommendations for such legislation or administrative actions as the Commission considers appropriate in light of the results of the study. (d) DEADLINE.—The report under subsection (c) shall be submitted not later than the date that is 15 months after the date a majority of the members of the Commission are appointed pursuant to section 3. (e) DEFINITION.—In this section, the term ‘‘administrative data’’ means data— (1) held by an agency or a contractor or grantee of an agency (including a State or unit of local government); and (2) collected for other than statistical purposes. SEC. 5. OPERATION AND POWERS OF THE COMMISSION. (a) EXECUTIVE BRANCH ASSISTANCE.—The heads of the following Consultation. agencies shall advise and consult with the Commission on matters within their respective areas of responsibility: (1) The Bureau of the Census. (2) The Internal Revenue Service. (3) The Department of Health and Human Services. (4) The Department of Agriculture. (5) The Department of Housing and Urban Development. (6) The Social Security Administration. (7) The Department of Education. (8) The Department of Justice. (9) The Office of Management and Budget. (10) The Bureau of Economic Analysis. (11) The Bureau of Labor Statistics.

112 The Promise of Evidence-Based Policymaking

130 STAT. 320 PUBLIC LAW 114–140—MAR. 30, 2016

(12) Any other agency, as determined by the Commission. Deadline. (b) MEETINGS.—The Commission shall meet not later than 30 days after the date upon which a majority of its members have been appointed and at such times thereafter as the chairperson or co-chairperson shall determine. (c) RULES OF PROCEDURE.—The chairperson and co-chairperson shall, with the approval of a majority of the members of the Commission, establish written rules of procedure for the Commis- sion, which shall include a quorum requirement to conduct the business of the Commission. (d) HEARINGS.—The Commission may, for the purpose of carrying out this Act, hold hearings, sit and act at times and places, take testimony, and receive evidence as the Commission considers appropriate. (e) CONTRACTS.—The Commission may contract with and com- pensate government and private agencies or persons for any purpose necessary to enable it to carry out this Act. (f) MAILS.—The Commission may use the United States mails in the same manner and under the same conditions as other agencies of the Federal Government. (g) GIFTS.—The Commission may accept, use, and dispose of gifts or donations of services or property. SEC. 6. FUNDING. (a) IN GENERAL.—Subject to subsection (b) and the availability of appropriations— (1) at the request of the Director of the Census, the agencies identified as ‘‘Principal Statistical Agencies’’ in the report, published by the Office of Management and Budget, entitled ‘‘Statistical Programs of the United States Government, Fiscal Year 2015’’ shall transfer funds, as specified in advance in appropriations Acts and in a total amount not to exceed $3,000,000, to the Bureau of the Census for purposes of carrying out the activities of the Commission as provided in this Act; and (2) the Bureau of the Census shall provide administrative support to the Commission, which may include providing physical space at, and access to, the headquarters of the Bureau of the Census, located in Suitland, Maryland. (b) PROHIBITION ON NEW FUNDING.—No additional funds are authorized to be appropriated to carry out this Act. This Act shall be carried out using amounts otherwise available for the Bureau of the Census or the agencies described in subsection (a)(1). SEC. 7. PERSONNEL. Appointment. (a) DIRECTOR.—The Commission shall have a Director who shall be appointed by the chairperson with the concurrence of the co-chairperson. The Director shall be paid at a rate of pay established by the chairperson and co-chairperson, not to exceed the annual rate of basic pay payable for level V of the Executive Schedule (section 5316 of title 5, United States Code). (b) STAFF.—The Director may appoint and fix the pay of additional staff as the Director considers appropriate. (c) EXPERTS AND CONSULTANTS.—The Commission may procure temporary and intermittent services under section 3109(b) of title 5, United States Code, at rates for individuals which do not exceed the daily equivalent of the annual rate of basic pay for a comparable position paid under the General Schedule.

The Promise of Evidence-Based Policymaking 113

PUBLIC LAW 114–140—MAR. 30, 2016 130 STAT. 321

SEC. 8. TERMINATION. The Commission shall terminate not later than 18 months after the date of enactment of this Act.

Approved March 30, 2016.

LEGISLATIVE HISTORY—H.R. 1831 (S. 991): HOUSE REPORTS: No. 114–211 (Comm. on Oversight and Government Reform). SENATE REPORTS: No. 114–151 (Comm. on Homeland Security and Governmental Affairs) accompanying S. 991. CONGRESSIONAL RECORD: Vol. 161 (2015): July 27, considered and passed House. Vol. 162 (2016): Mar. 16, considered and passed Senate, amended. Mar. 17, House concurred in Senate amendment. Æ

The Promise of Evidence-Based Policymaking 115

Appendix B: Commissioner Biographies

Katharine G. Abraham Sherry Glied University of Maryland New York University Commissioner and Chair Commissioner Katharine G. Abraham is Professor of Econom- In August 2013, Sherry Glied became Dean of New ics and Survey Methodology at the University of York University’s Robert F. Wagner Graduate School Maryland. She has written extensively on the ef- of Public Service. From 1989-2013, she was Profes- fects of labor market policies and institutions on sor of Health Policy and Management at Columbia firm and worker behavior, labor market adjust- University’s Mailman School of Public Health. She ment over the business cycle, and the measure- was Chair of the department from 1998-2009. On ment of economic activity. Abraham first joined June 22, 2010, Glied was confirmed by the U.S. the University of Maryland faculty in 1987, after Senate as Assistant Secretary for Planning and holding prior positions at the Brookings Institu- Evaluation at the U.S. Department of Health and tion from 1985 to 1987 and at the Massachusetts Human Services, and served in that capacity from Institute of Technology’s Sloan School of Man- July 2010 through August 2012. She has previous- agement from 1980 to 1985. From 2011 to 2013, ly served as Senior Economist for health care and Abraham served as a Member of the Council of labor market policy on the Council of Economic Economic Advisers and from 1993 to 2001, she Advisers in 1992-1993, under Presidents Bush and served as Commissioner of the Bureau of Labor Clinton. Glied’s principal areas of research are in Statistics. Abraham is a Fellow of the Society of health policy reform and mental health care policy. Labor Economists and of the American Statistical Association and has served as a Vice President of the American Economic Association. Abraham re- Robert M. Groves ceived a Bachelor’s degree from Iowa State Uni- Georgetown University versity and a Ph.D. from Harvard University. Commissioner Robert M. Groves is the Provost of Georgetown University and the Gerard J. Campbell, S.J. Profes- Ron Haskins sor in the Math and Statistics Department as well Brookings Institution as the Sociology Department. He served as director Commissioner and Co-Chair of the U.S. Census Bureau during the conduct of the Ron Haskins is a Senior Fellow and holds the 2010 Census of Population. Groves studies how to Cabot Family Chair in Economic Studies at the improve large-scale sample surveys and censuses. Brookings Institution, where he co-directs the His research has focused on the impact of mode of Center on Children and Families. Haskins is also data collection on responses in sample surveys, the a senior consultant at the Annie E. Casey Founda- influences on survey participation, the use of adap- tion and is Past-President of the Association for tive research designs to improve the cost and error Public Policy Analysis and Management. Prior to properties of statistics, and public concerns about joining Brookings and Casey, he spent 14 years on privacy affecting attitudes toward statistical agen- the staff of the House Ways and Means Human cies. Groves is an elected member of the National Resources Subcommittee, serving as the subcom- Academy of Sciences and the National Academy of mittee’s Staff Director after Republicans became Medicine. He is also the chair of the Committee the majority party in the House after the 1994 on National Statistics at the National Academies elections. He holds a Bachelor’s degree in Histo- of Sciences, Engineering, and Medicine. He is an ry, a Master of Arts in Teaching, and a Ph.D. in appointed member of the National Science Board, Developmental Psychology from the University of overseeing the National Science Foundation, as North Carolina at Chapel Hill. In his Washington well as the Board of the Pew Research Center. career, he has focused on evidence-based policy, early childhood education, marriage and family formation, poverty, equal opportunity, abused and neglected children, and budget issues. 116 The Promise of Evidence-Based Policymaking

Robert Hahn Poverty by Half in 10 Years. Previously, she was University of Oxford a member of the Advisory Committee for Social, Commissioner Behavioral and Economic Sciences for the Nation- Robert Hahn is Professor and Director of Economics al Science Foundation and the National Advisory at the Smith School of Enterprise and the Environ- Committee of the Robert Wood Johnson Founda- ment at the University of Oxford; a senior fellow at tion Scholars in Health Policy Research Program. the Institute for New Economic Thinking at Oxford; a senior fellow at the Georgetown University Cen- ter for Business and Public Policy, and a non-resi- Jeffrey Liebman dent senior fellow at the Brookings Institution. Harvard University Hahn worked at American Enterprise Institute for Commissioner two decades, where he co-founded and directed Jeffrey Liebman is the Malcolm Wiener Professor the AEI-Brookings Joint Center. Previously, Hahn of Public Policy at the Harvard Kennedy School worked for the Council of Economic Advisers and where he directs the Taubman Center for State was the chief economist on the White House draft- and Local Government and the Rappaport Insti- ing team for the 1990 Clean Air Act Amendments. tute for Greater Boston. Liebman teaches courses He also has served on the faculties of Harvard Uni- in social policy, public sector economics, Ameri- versity and Carnegie Mellon University. Hahn is can economic policy, and public sector manage- currently conducting several behavioral economics ment. In his research, he studies tax and budget experiments aimed at encouraging the conservation policy, social insurance, and public sector inno- of energy and water resources, improving labor pro- vation. During the first two years of the Obama ductivity, improving health outcomes, and under- Administration, Liebman served at the U.S. Office standing the welfare benefits of new technologies. of Management and Budget, first as Executive As- In addition, he co-founded the Community Prepa- sociate Director and Chief Economist and then as ratory School, an inner-city middle school in Provi- Acting Deputy Director. From 1998 to 1999, Lieb- dence, Rhode Island, that provides opportunities for man served as Special Assistant to the President disadvantaged youth to achieve their full potential. for economic policy and coordinated the Clinton Administration’s Social Security reform technical working group. For the past five years, his Harvard Hilary Hoynes Kennedy School Government Performance Lab University of California, Berkeley has been providing pro bono technical assistance Commissioner to state and local governments interested in im- Hilary Hoynes is a Professor of Economics and proving the results they achieve for their citizens. Public Policy and holds the Haas Distinguished Chair in Economic Disparities at the University of California, Berkeley. From 2011 to 2016, she was a Bruce D. Meyer co-editor of the leading journal in economics, the University of Chicago American Economic Review. Hoynes specializes Commissioner in the study of poverty, inequality, food and nutri- Bruce D. Meyer is the McCormick Foundation Pro- tion programs, and the impacts of government tax fessor at the Harris School of Public Policy at the and transfer programs on low-income families. University of Chicago where he has been since 2004. Current projects include evaluating the effects of From 1987 to 2004, Meyer was a professor in the access to the social safety net in early life on later Economics Department at Northwestern University. life health and human capital outcomes, examin- He is also a Research Associate of the National Bu- ing the effects of the Great Recession on poverty, reau of Economic Research and a Visiting Scholar at and the role of the safety net in mitigating in- the American Enterprise Institute. He studies pover- come losses. Professor Hoynes is a member of the ty and inequality, tax policy, government safety net American Economic Association’s Executive Com- programs such as unemployment insurance, work- mittee and a panel member of the Committee on ers’ compensation, food stamps, and Medicaid, and National Statistics at the National Academies of the accuracy of household surveys. His most recent Sciences, Engineering, and Medicine on Building work includes research on trends in poverty and in- an Agenda to Reduce the Number of Children in equality, the consequences of disability, the effects The Promise of Evidence-Based Policymaking 117

of Medicaid, and errors in household surveys. He uty Director and Chief Operating Officer of the currently serves on the Technical Advisory Commit- U.S. Census Bureau. She has also served as Deputy tee to the Bureau of Labor Statistics, the American Under Secretary for Economic Affairs at the U.S. Economic Association Committee on Government Department of Commerce; Principal Associate Di- Relations, and as an officer of the Business and Eco- rector and CFO at the U.S. Census Bureau; Senior nomic Statistics Section of the American Statistical Vice President for Economic, Labor, and Population Association. Meyer received his Bachelor’s degree Studies at NORC at the University of Chicago; and and Master’s degree in economics from Northwest- Chief Operating Officer at McManis & Monsalve ern University and his Ph.D. in economics from the Associates, a business analytics consulting firm. Massachusetts Institute of Technology. She is an adjunct professor at the Trachtenberg School of Public Policy and Public Administration at The George Washington University. Potok is an Paul Ohm elected Fellow of the National Academy of Public Georgetown University Administration. She currently serves on the Board Commissioner of Directors for the Institute of Pure and Applied Paul Ohm is a Professor of Law at the Georgetown Mathematics at the University of California Los University Law Center. He specializes in infor- Angeles; the Board of Trustees for the Arthur S. mation privacy, computer crime law, intellectual Flemming Award, which recognizes outstanding property, and criminal procedure. achievement by federal employees with fewer than 15 years of service; and The George Washington University Trachtenberg School Advisory Board. Allison B. Orris Potok received her Ph.D. in public policy and public U.S. Office of Management and Budget administration at The George Washington Univer- Commissioner sity, with an emphasis on program evaluation. (Departed the Commission on January 20, 2017) Allison B. Orris served as the Associate Adminis- trator of the Office of Information and Regulato- Kathleen Rice Mosier ry Affairs at the U.S. Office of Management and Faegre Baker Daniels, LLP Budget until January 2017. Orris served in several Commissioner roles at the U.S. Department of Health and Hu- Kathleen Rice Mosier is a Counsel at Faegre Baker man Services Centers for Medicare & Medicaid Daniels, LLP, where she provides advice and assis- Services from 2009 to 2014, including Senior Poli- tance to organizations on issues relating to privacy, cy Advisor, Acting Director of the Division of State data security, risk management, and compliance Demonstrations and Waivers, and Director of the with applicable laws and regulations. Before join- Low Income Programs Analysis Group in the Of- ing FaegreBD, she spent nearly 20 years in law fice of Legislation. Before joining CMS, Orris was enforcement and national security. She served as a Senior Legislative Associate at the Center on a Counsel on the Senate Select Committee on In- Budget and Policy Priorities from 2006 to 2009. telligence, as an Assistant General Counsel for the From 2002 to 2006, she practiced law at Powell Federal Bureau of Investigation, and as an Assistant Goldstein LLP in Washington, D.C. Orris received U.S. Attorney for the Southern District of Florida. a Bachelor’s degree in History from Columbia She is a frequent author and speaker on privacy University and a J.D. from Yale University. and data security matters. She is also a Certified Information Privacy Professional, International As- sociation of Privacy Professionals, and an Adjunct Nancy Potok Professor of Law at Notre Dame Law School. U.S. Office of Management and Budget Commissioner (Appointed to the Commission on March 10, 2017) Robert Shea Nancy Potok is Chief Statistician of the United Grant Thornton, LLP States and Chief of the Statistical and Science Pol- Commissioner icy Branch in the U.S. Office of Management and Robert Shea is a principal and a member of the Budget. Prior to January 2017, she served as Dep- Public Sector practice at Grant Thornton, LLP. He 118 The Promise of Evidence-Based Policymaking

leads Strategy and Communications for Grant his Ph.D. in economics in 1992 from the Univer- Thornton Public Sector and provides performance sity of Chicago. improvement services to international, federal, and state and local government agencies. Before joining Grant Thornton, Shea served as Associ- Kim R. Wallin ate Director for Management at the U.S. Office of D.K. Wallin, Ltd. Management and Budget (OMB). Before joining Commissioner OMB, Shea served as counsel to the Senate Com- Kim R. Wallin is currently in private practice at her mittee on Governmental Affairs, legislative direc- CPA firm, D.K. Wallin Ltd., which she founded in tor for Congressman Pete Sessions (TX), and pro- 1984. She took eight years off to be the Controller fessional staff member for the House Committee for the State of Nevada. Wallin was the first CPA to on Government Reform and Oversight. be elected to the office of Nevada State Controller in 50 years and the first Certified Management Accountant to ever hold this office. As Control- Latanya Sweeney ler, she received the Association of Government Harvard University Accountants Presidents Award and Excellence in Commissioner Government Award and the National Association Latanya Sweeney is Professor of Government and of Comptrollers President’s Award for her work to Technology in Residence at Harvard University. improve transparency in government with the use Her mission is to create and use technology to as- of technology. She also received the National As- sess and solve societal, political and governance sociation of State Auditors, Comptrollers and Trea- problems, and to teach others how to do the same. surers President’s Award for her work on various Her primary area of focus is the scientific study of U.S. Presidential work groups and for her continued technology’s impact on humankind. She serves as efforts to improve efficiency and accountability in the Editor-in-Chief of the Journal of Technology Government. Wallin is a graduate of the University Science, Director of the Data Privacy Lab at Har- of Nevada, Las Vegas with a degree in Business vard, and Faculty Dean at Currier House. Sweeney Administration with a major in accounting. formerly served as the Chief Technology Officer, also known as the Chief Technologist, at the U.S. Federal Trade Commission. Commission Staff

Rochelle (Shelly) Wilkie Martinez, Kenneth R. Troske Executive Director University of Kentucky H. Lucas Hitt, Deputy Executive Director Commissioner Sharon A. Boivin, Senior Policy and Kenneth R. Troske is the Associate Dean for Grad- Research Analyst uate Programs and Outreach and Richard W. and Anne Fletcher, Senior Policy and Janis H. Furst Endowed Chair in Economics at Research Analyst the Gatton College of Business and Economics at the University of Kentucky, as well as a Research Nicholas Hart, Policy and Research Director Fellow with the Institute for the Study of Labor Michael Hawes, Privacy Consultant (IZA) in Bonn, Germany. Troske served as a mem- Kristy L. Howell, Senior Policy and ber of the Congressional Oversight Panel whose Research Analyst task was to assess the existing condition of Amer- ica’s financial markets and the regulatory system Mary D. McKoy, Chief, External Affairs as well as to monitor the actions of the Treasury Kathryn McNamara, Librarian and Department and financial institutions to deter- Records Coordinator mine if their actions are in the best interest of Sara Stefanik, Policy and Research Analyst the American economy. His primary research ar- Robin L. Wyvill, Meetings and Events eas are labor and human resource economics. He Coordinator ■ received his undergraduate degree in economics from the University of Washington in 1984 and The Promise of Evidence-Based Policymaking 119

Appendix C: Commission’s Fact-Finding and Deliberative Processes

Fact-Finding Process o Justin Erlich—California Attorney General’s Office Following the initial appointments of commissioners, the Commission convened its first public o Katherine Wallman—Office of Management meeting in July 2016. Over the course of the follow- and Budget, Executive Office of the ing eight months, the Commission studied issues President (former) about the current state of evidence production and o Marc Groman—Office of Management and use in the Federal government as well as the Fed- Budget, Executive Office of the President eral government’s policies and practices to protect (former) data confidentiality. The Commission’s fact-find- o Marc Rotenberg—Electronic Privacy ing process included public meetings, public hear- Information Center ings, and meetings with organizations to which Commissioners or staff were invited or that staff o Michael Basil—Illinois Department of initiated, a survey of Federal offices, a Request for Innovation & Technology Comments in the Federal Register, and other public • November 4, 2016: Considerations for the input received through email correspondence. Commission Related to Evaluation Public Meetings o Adam Gamoran—William T. Grant The Commission’s seven public meetings included Foundation a total of 49 invited witnesses. o Demetra Nightingale—U.S. Department of • July 22, 2016: Introductory Meeting Labor (former) o Devin O’Connor—Office of Management o Evelyn Kappeler—Office of Adolescent and Budget, Executive Office of the Presi- Health, U.S. Department of Health and dent (former) Human Services o Jeri Mulrow—Bureau of Justice Statistics, o Jim Sullivan—University of Notre Dame U.S. Department of Justice o Katherine O’Regan—U.S. Department of o John Righter—Committee on Health, Edu- Housing and Urban Development (former) cation, Labor & Pensions, U.S. Senate o Kelly Fitzsimmons—The Edna McConnell o Mary Bohman—Economic Research Ser- Clark Foundation vice, U.S. Department of Agriculture o Matthew Klein—New York City Center for o Nancy Potok—Census Bureau, U.S. Depart- Economic Opportunity ment of Commerce (former) o Naomi Goldstein—Administration for o Raj Chetty—Stanford University Children and Families, U.S. Department of Health and Human Services o Ted McCann—Office of the Speaker, U.S. House of Representatives o Tanya Beer—Center for Evaluation Innovation • September 9, 2016: Key Considerations in Privacy Relevant to the Commission’s Charge • December 12, 2016: Considerations for the Commission Related to Federal Models o Aimee Guidera—Data Quality Campaign (former) o Barry Johnson—Statistics of Income Division, Internal Revenue Service, U.S. o Cynthia Dwork—Microsoft Research Department of Treasury (former) 120 The Promise of Evidence-Based Policymaking

o David Grusky—Stanford University o Andrew Reamer—The George Washington o Erica Groshen—Bureau of Labor Statistics, University U.S. Department of Labor (former) o Brian Moyer—Bureau of Economic Analysis, o Marilyn Seastrom—National Center for U.S. Department of Commerce Education Statistics, U.S. Department of o Erin Ulrich—Colorado Department of Public Education Health and Environment o Niall Brennan—Centers for Medicare and o Maria Cancian—University of Wisconsin Medicaid Services, U.S. Department of o Seth Harris—U.S. Department of Labor Health and Human Services (former) (former) o Ron Jarmin—Census Bureau, U.S. Department of Commerce Public Hearings • January 13, 2017: International and State The Commission convened three open public Models for Managing Data hearings—in Washington, D.C., Chicago, and San Francisco—during which any member of the public o Charles Rothwell—National Center for who requested to testify before the Commission Health Statistics, U.S. Department of was allowed to present. A total of 39 members of Health and Human Services the public presented information during the three o David Mancuso—Washington State hearings. Department of Social and Health Services o Domenico Parisi—Mississippi State • October 21, 2016: Washington, D.C. University o Amanda Janice Roberson—Institute for o Ivan Thaulow—Statistics Denmark Higher Education Policy o Kenneth Dodge—Duke University o Carrie Wofford—Veterans Education Success o Robert Goerge—University of Chicago o Christine Keller—Association of Public & o Roxane Silberman—Secure Data Access Land-grant Universities Centre, France o Clyde Tucker—American Statistical o Shawna Webster—National Association for Association Public Health Statistics and Information Systems o Daniel Crowley—National Prevention Science Coalition to Improve Lives o Stefan Bender—Deutsche Bundesbank o David Medina—Results for America o Tanvi Desai—Administrative Data Research Network, United Kingdom o Emmett McGroarty—American Principles Project • February 24, 2017: Role of Legal Standards o Erin Knowles—United States Parents and Technology in Maximizing Data Security Involved in Education and Privacy o George Grob—American Evaluation o Alexandra Wood—Harvard University Association o Bradley Malin—Vanderbilt University o Kelleen Kaye—The National Campaign to o Daniel Goroff—Alfred P. Sloan Foundation Prevent Teen and Unplanned Pregnancy o Jerome Reiter—Duke University o Mark Schneider—American Institutes for o Lars Vilhuber—Cornell University Research o Quentin Wilson—Public Performance March 13, 2017: Federal Statistical and Part- • Improvement Researcher nership Infrastructure for Evidence Building — Opportunities and Limitations o Rachel Fishman—New America The Promise of Evidence-Based Policymaking 121

o Rachel Zinn—Workforce Data Quality Additional Meetings Campaign In addition, Commissioners and staff met with o RK Paleru—Booz Allen Hamilton experts, and participated in conference panel pre- o Sara Dube—Pew-MacArthur Results First sentations to solicit additional input from more Initiative than 40 organizations. These meetings supple- mented information gathered during the Com- o Tiffany Jones—The Education Trust mission’s fact-finding phase. o Tom Allison—Young Invincibles o Actionable Intelligence for Social Policy Public—Academic Research Colloquium • January 5, 2017: Chicago, IL o Cassie Creswell—Raise Your Hand Action/ o American Evaluation Association Parent Coalition for Student Privacy o American Public Human Services o Donna K. Ginther—University of Kansas Association National Summit o Lint Barrage—Rhode Island Innovative o Amy O’Hara, Census Bureau , U.S. Policy Lab Department of Commerce (former) o Margaret Levenstein—ICPSR o Anna McDowell, Statistics New Zealand o Matthew Stagner—Mathematica Policy o Army Analytics Group Research o Association for Public Policy Analysis & o Quentin Palfrey—J-PAL North America Management o Timothy Slaper—Indiana Business Research o Association of Public Data Users Webinar Center o Bipartisan Policy Center o Tom Schenk, Jr.—City of Chicago o Bureau of Economic Analysis Advisory o V. Joseph Hotz—Duke University Committee o Virginia Knox—MDRC o Census Scientific Advisory Committee o Center for Regional Economic • February 9, 2017: San Francisco, CA Competitiveness o Andrew Wiegand—Social Policy Research o Center for the Study of Social Policy Associates o Committee on Population Statistics o Cindy Guy—Annie E. Casey Foundation o Council of Professional Associations on o David Johnson—University of Michigan Federal Statistics o Jim Hill—Proofpoint Systems, Inc. o Cross Agency Learning Community on o Joy Bonaguro—City and County of San Research and Evaluation Francisco o Defense Manpower Data Center, U.S. o Karen Levesque—RTI International Department of Defense o Karen R. Effrem—Education Liberty Watch o Defense Personnel and Security Research o Mary Ann Bates—J-PAL North America Center, U.S. Department of Defense o Maryann Feldman—University of North o Federal Committee on Statistical Carolina—Chapel Hill Methodology Policy Conference o Sandra Torosian—United Parents Involved o Federal Economic Statistics Advisory in Education Committee o Susan Dreyfus—Alliance for Strong Families o Federal Evaluators and Communities o Federation of Associations in Behavioral & Brain Sciences 122 The Promise of Evidence-Based Policymaking

o Frauke Kreuter, University of Maryland o Transportation Research Board Special o Health and Human Services Administrative Task Force on Data for Decisions and Data Technical Expert Panel Performance Measures o Interagency Council on Evaluation Policy o U.S. Census Bureau and Economic Research Service Administrative Data Conference o Jenny Hunter Childs, Census Bureau, U.S. Department of Commerce o U.S. Department of Health and Human Services Data Council o John Abowd, Census Bureau, U.S. Department of Commerce o Workforce Data Quality Campaign: Fly—In Conference o John Mitchell, Stanford University o Julia Lane, New York University Other Stakeholder Input o Kathy Stack, Laura and John Arnold The Commission issued a Request for Comments Foundation in the Federal Register and accepted comments o Laura and John Arnold Foundation and Bill by email, which generated over 350 responses and Melinda Gates Foundation Workshop from the public (see online Appendix G). Finally, on the use of data to understand the current or potential capacity of Federal agencies to engage in aspects of evi- o Liz McPherson, Statistics New Zealand dence-based policymaking, the Commission ad- o Mark Mazur, U.S. Department of the ministered a survey to 209 offices of the Federal Treasury (former) government that the Commission identified as o MDRC likely to be generating or using evidence (see online Appendix E). o Michael Duff, Stanford University o Michigan Retirement Research Center, Financial Research Symposium Deliberative Process Following the formal stakeholder input processes, o Minister Hiro Matsumoto, Embassy of the Commission compiled and reviewed the body Japan of information collected. A series of 12 memoranda o National Association of State Workforce that compiled existing literature, stakeholder in- Agencies Labor Market Information put, and options for recommendations were devel- Committee oped. Each memorandum was initially discussed o National Conference of State Legislatures in small workgroups then formally discussed by Commissioners during closed session meetings as o National Governors Association the Commission deliberated. Multiple memoranda o National Institute of Standards and involved discussion during more than one meeting; Technology, Privacy Risk Assessment the list below reflects the initial discussion points: Meeting April 3, 2017 o Naval Postgraduate School • o Vision Statement and Evidence Principles o Open Data Center Korea o National Secure Data Service Implementation o Pew Charitable Trusts o Data Access and Use—Legal and Statutory o Pew Results First & State Data Project Team Issues o Results for America o Data Access and Use—Federal Policy and o Ross Young, United Kingdom Statistics Standards Authority o Increasing Access to State—Collected o Society for Benefit Cost Analysis Annual Administrative Data Conference o Tiered Access with a National Secure Data Service The Promise of Evidence-Based Policymaking 123

• May 15, 2017 • June 27, 2017 o Enhancing Collaboration in the Federal o Collaborating for Evidence Building Evidence Ecosystem o Incorporating Evaluation and Policy o The Role of the Federal Statistical System Research in Program Design within the Evidence Ecosystem o Administration o Protecting Privacy and Increasing Confidentiality Once the Commission developed tentative recommendations, the final two closed session meetings o Protecting Privacy by Increasing of the Commission were dedicated to discussing Transparency About Evidence Generation report text. and Benefits July 11, 2017 o Establishing a National Secure Data Service • • July 20, 2017 June 7, 2017 • Commissioners collaborated on draft report text o Role of the National Secure Data Service during this time. Commissioners cast digital votes and the PSAs in the Federal Evidence of approval from July 28 to August 2. All 15 Com- Ecosystem missioners cast votes in the affirmative approving o Institutionalizing and Incentivizing the the findings and recommendations in the report. ■ Evaluation and Policy Research Functions Across Government

The Promise of Evidence-Based Policymaking 125

Appendix D: Examples of Data Productive for Evidence Building

During the course of the Commission’s fact-find- and individual information. Statutory Citation: ing phase, numerous experts and members of 26 USC § 6103. the public suggested specific sources of data that could be useful for evidence-building activities. • The National Directory of New Hires*, main- Statistical uses of administrative data collected tained by the Administration for Children through program operations—such as enrollment and Families (ACF) Office of Child Support and utilization information—as well as statistical Enforcement in the Department of Health and data collected directly through topical surveys in- Human Services (HHS), includes quarterly tended to support evidence building offer oppor- wage information, Unemployment Insurance tunities to enhance evidence-based policymaking benefit information, and information on new- in the United States. Secure access to data, includ- ly hired employees. Statutory Citation: 42 USC § ing data in the domains listed below, is addressed 653(j)(5). throughout the Commission’s report. The data described below are not intended • Unemployment Insurance (UI)* quarterly to reflect a comprehensive inventory of govern- wage records are collected by states as part of ment data, but rather reflect suggestions provid- their administration of state UI programs oper- ed to the Commission about some data that have ated through the Federal-state UI partnership. the potential to be useful for evidence building. Numerous other administrative and survey data • The Wage Record Interchange System sources are likely to be no less valuable for evi- (WRIS)*, maintained by the Department of dence building in their respective policy domains. Labor, facilitates the exchange of wage data In some cases, the purposes for which data may between states for performance accountability be used are defined narrowly, limiting their use purposes, enabling improved reporting on the for evidence building. The Commission includes outcomes experienced by participants in Fed- recommendations to reconsider these limitations. erally funded state employment and training Recommendation 2-4 proposes a review of such programs. It allows researchers access to wage statutes to ensure that limitations that preclude data on a state-by-state approval basis for proj- the use of administrative data for evidence build- ects with a direct benefit to these programs. ing are applied only when the Congress and the The WRIS is being revised to reflect changes President deem the limitations still to be nec- introduced by the Workforce Innovation and essary. In some cases, existing laws specifically Opportunity Act, passed in 2014, and will be prohibit the collection or analysis of information called the State Wage Interchange System. to support evidence building. Recommendation Statutory Citation: 42 USC § 1111. 2-5 calls for a reconsideration of such bans and restraint in the enactment of future bans. Data Health relevant for implementing Recommendations 2-4 data are and 2-5 include those indicated with an asterisk. • Health Services Research Program collected by the Substance Abuse and Mental Health Services Administration (SAMHSA) Administrative Data in HHS. Data include research related to substance abuse treatment, access to mental Income, Wages, and Earnings health care, healthcare costs, and patient outcomes. Statutory Citation: 42 USC § 290aa. • Federal Tax Information* is collected by the Internal Revenue Service (IRS) in the Depart- • Medicaid, Medicare, and Children’s Health ment of the Treasury and includes business Insurance Program (CHIP) data are collected 126 The Promise of Evidence-Based Policymaking

by the Centers for Medicare and Medicaid Ser- ture’s Family and Nutrition Services. SNAP data vices (CMS) in HHS. Data include information include information about program eligibility, on Medicare claims, beneficiaries and provid- length of services, and amounts of benefits. ers, Medicaid eligibility, and claims. Statutory Statutory Citation: 7 USC § 2011 et seq. Citations: 42 USC § 1396 (Medicaid and CHIP), 42 USC § 1395 (Medicare). • Supplemental Security Income (SSI) Program data collected by states and SSA includes • Vital Records administrative data are collected income assistance information for those partic- by the National Center for Health Statistics ipants who are needy, aged, blind, or disabled. (NCHS) in HHS from vital registrars in all states Statutory Citation: 42 USC § 1383. and U.S. territories. Vital Records files include information about births and deaths. Statutory • Temporary Assistance for Needy Families Citation: 42 USC § 242k(h). (TANF) is administered by ACF’s Office of Family Assistance in HHS. TANF program data Human Services includes caseloads, financial information, workforce participation rates and other pro- • Child Welfare Program data include statistics gram participant characteristics. Statutory Cita- collected through the Adoption and Foster tion: 42 USC § 611-611a. Care Analysis and Reporting System (AFCARS) and through the National Child Abuse and Ne- • Unemployment Insurance (UI) payments glect Data System (NCANDS). AFCARS collects (benefits) provide temporary financial assis- case-level information on all foster children and tance to unemployed workers who meet the those who have been adopted in the United requirements of State law. Information about States. NCANDS includes data on child abuse payments includes the benefits paid and dura- and neglect cases in the United States. Both data tion. Statutory Citation: 29 USC § 49b(b). collection systems are administered by the Chil- dren’s Bureau of ACF. Statutory Citations: 42 USC Other § 679 (AFCARS) and 42 USC § 5119 (NCANDS). • Business Program Participation data, such as • Head Start Program* data are collected by local the small business loan program at the Small jurisdictions through a program funded by the Business Administration includes information ACF in HHS about program participants. Statu- on lenders, borrowers, and loan amounts. tory Citation: 42 USC § 9836a and 9844. Data collected by the Economic Development Administration reflects information about • Social Security Disability Insurance (SSDI) grantees use of funds to support workforce data are collected about program length of ser- development and other regional economic vice, benefits, eligibility, appeals, and denials opportunities. Statutory Citations: 15 USC § 638 through the Social Security Administration (Small Business Administration) and 42 USC § (SSA) and Disability Determination Services in 3192 (Economic Development Administration). each state. Statutory Citation: 42 USC § 401. • Criminal justice data include crime incident • Social Security Old Age and Survivors Insur- reports and criminal history record informa- ance (OASI) Program data includes information, or “rap sheets.” Crime incident data are tion on retired and survivor beneficiaries and collected through the Federal Bureau of Inves- benefit amounts received.Statutory Citation: 42 tigation (FBI) National Incident-Based Report- USC § 401. ing System (NIBRS). Data include the nature and types of specific offenses in the incident, • Supplemental Nutrition Assistance Program characteristics of the victim(s) and offender(s), (SNAP) data are collected by all states through a types and value of property stolen and recov- program funded by the Department of Agricul- ered, and characteristics of persons arrested in The Promise of Evidence-Based Policymaking 127

connection with a crime incident. Rap sheets to serve a statistical purpose. In some cases, the sta- are created and maintained by law enforce- tistical purposes for which a statistical agency may ment agencies and accessible via the FBI Inter- grant access to confidential data under applicable state Identification Index for law enforcement law are defined very narrowly; the Census Bureau’s and other purposes. They contain information law, for example, permits the use of sworn agents about an individual’s arrests and subsequent such as researchers only to help the agency carry dispositions, such as jail sentences. Statutory out its work, and projects must specifically advance Citations: 42 USC § 3732 and 3735 (NIBRS), 28 the mission of the Census Bureau irrespective of USC § 534 (criminal records). the broader value that access to data might have for evidence building. Examples of statistical data from • Federal Student Aid* data are collected by the Principal Statistical Agencies are listed below. the Department of Education. Data include information about the Federal assistance given 1 to students and families under Title IV of the Statistical Agency Data Higher Education Act of 1965, including stu- • Bureau of Economic Analysis, in the Depart- dent aid applications, student loan applications ment of Commerce, collects data on multina- and receipt, and characteristics of applicants. tional enterprises and international services Statutory Citation: 20 USC § 1092b(d)(2). transactions. Statutory Citations: 22 USC § 3101- 3108. • Housing Assistance Data are collected by public housing agencies funded by the Department • Bureau of Justice Statistics, in the Department of Housing and Urban Development (HUD) and of Justice, collects data on crime, offenders, maintained in shared systems, including the crime victims, and the operations of the crim- Public and Indian Housing Information Center inal justice system at all levels of government. and the Tenant Rental Assistance Certification Statutory Citations: 42 USC § 3732, 42 USC § 3735 System. Data collected include characteristics and 3789g. of properties and units as well as information about the households receiving subsidized • Bureau of Labor Statistics, in the Department housing assistance. Data regarding partici- of Labor, collects data on employment and un- pation in homeless assistance programs are employment, projections of the labor force and collected at the local level through Homeless employment by industry and occupation, prices Management Information Systems. Statutory and inflation, consumer expenditures, wages Citations: 42 USC § 3535 and 3543. and employee benefits, occupational injuries and illnesses, and productivity and technolog- • Military and Veteran Program data are col- ical change. Statutory Citations: 29 USC § 2, 2b, 5, lected by a variety of sources, including the 6, 8, and 9; 19 USC § 2393; 29 USC § 673. Department of Defense and the Department of Veteran Affairs, among other departments. • Bureau of Transportation Statistics, in the These data include demographic, health, and Department of Transportation, collects data other information about members of the active on transportation and transportation systems, duty military and their families as well as ser- including freight and travel statistics. Statutory vices provided to veterans. Citations: 49 USC § 6302(c); 49 USC § 6306; 49 USC § 6307; 49 USC § 6313. Although Federal statistical agencies produce many statistical data products that are available to • Census Bureau, in the Department of Com- the public without restriction, access to data files merce, conducts the Decennial Census and produced by these agencies that contain confidential information about individual people, businesses, or organizations is, quite appropriately, tightly 1. U.S. Congressional Research Service, Selected Information Re- garding Statistical and Evaluation Entities in the Executive Branch controlled. This is true even when access is sought [Memo] (February 28, 2017), by Clinton T. Brass and Jennifer D. Williams; https://www.cep.gov/library/testimony.html . 128 The Promise of Evidence-Based Policymaking

administers surveys that collect information on related to education in the United States and the demographic and economic health of the in other nations. Statutory Citations: 20 USC § United States, such as the American Communi- 9541-9544, 20 USC § 9546), 20 USC § 9573. ty Survey and the Survey of Income and Program Participation. Statutory Citation: 13 USC § 9. • National Center for Health Statistics, in HHS, collects data on the extent and nature of illness • Economic Research Service, in the Depart- and disability of the population of the United ment of Agriculture, collects data on the eco- States and several other specified metrics.Stat - nomic and social science aspects of agriculture, utory Citation: 42 USC § 242k. rural development, food, commodity markets, and the environment. Statutory Citation: 7 USC • National Center for Science and Engineering § 2276. Statistics, in the National Science Foundation, conducts periodic surveys and data collections • Energy Information Administration, in the on science, engineering, technology, and re- Department of Energy, collects data and infor- search and development. Statutory Citation: 42 mation which is relevant to energy resource USC § 1862p. reserves, energy production, demand, and technology, and related economic and statis- • Office of Research, Evaluation, and Statistics, tical information. Statutory Citation: 42 USC § in SSA, is responsible for conducting policy 7135(d). research and evaluation, providing statistical data on SSA programs, sponsoring special-pur- • National Agricultural Statistics Service, in the pose survey data collections and studies, and Department of Agriculture, conducts the Cen- other activities related to SSA’s mission. sus of Agriculture and collects data on agricultural production and the economic and envi- • Statistics of Income Division, in the IRS of the ronmental status of the farm sector. Statutory Department of the Treasury, collects and dis- Citations: 7 USC § 2276, 7 USC § 2204g. seminates income, financial, and tax information and also provides periodic reports on items • National Center for Education Statistics, in from other returns and schedules. Statutory Ci- the Department of Education, collects data tation: 26 USC § 6108(c). ■ The Promise of Evidence-Based Policymaking 129

The Commission’s collection of public input, survey data, and related materials are available via the National Archives and Records Adminis- tration. The materials available from the Archives include: Appendix E. Report on the CEP Survey of Federal Offices Appendix F. CEP Public Meeting Materials and Presentations Appendix G. CEP Public Input–Hearing Testimony and Other Public Comments Appendix H. Prior Commissions Related to Evidence Building

The Promise of Evidence-Based Policymaking Report of the Commission on Evidence-Based Policymaking

September 2017