This is the accessible text file for GAO report number GAO-11-193 entitled '2010 Census: Data Collection Operations Were Generally Completed as Planned, but Long-standing Challenges Suggest Need for Fundamental Reforms' which was released on December 14, 2010. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Requesters: December 2010: 2010 Census: Data Collection Operations Were Generally Completed as Planned, but Long-standing Challenges Suggest Need for Fundamental Reforms: 2010 Census: GAO-11-193: GAO Highlights: Highlights of GAO-11-193, a report to congressional requesters. Why GAO Did This Study: Although the U.S. Census Bureau (Bureau) generally completed the field data collection phase of the 2010 Census consistent with its operational plans, at $13 billion, 2010 was the costliest census in the nation’s history. Moving forward, it will be important to both refine existing operations as well as to reexamine the fundamental approach to the census to better address long-standing issues such as securing participation and escalating costs. As requested, this report reviews (1) the conduct of nonresponse follow-up (NRFU), where enumerators collect data from households that did not return their census forms, (2) the implementation of other field operations critical to a complete count, and (3) potential reexamination areas that could help produce a more cost-effective 2020 Census. The report is based on GAO’s analysis of Bureau data and documents, surveys of local census office managers, and field observations. What GAO Found: Nationally, the Bureau was well positioned to implement NRFU and subsequent field operations. The Bureau achieved a mail response rate of 63 percent, which was within its expectations, and recruited nearly 3.8 million total applicants for census jobs, which was 104 percent of its staffing goal. Moreover, the Bureau completed NRFU under budget, reportedly spending $1.59 billion on the operation, about $660 million (29 percent) less than the Bureau initially estimated. Most of the Bureau’s local census offices (LCO) also completed NRFU ahead of the 10-week allotted time frame. Despite these operational successes, the Bureau encountered some notable challenges. For example, the pace of NRFU may have fostered a culture that tended to emphasize speed over quality, as those LCOs with higher percentages of less-complete questionnaires were more likely to have completed NRFU in 53 days or less (the average time LCOs took to complete NRFU). The Bureau also had to overcome issues with critical information technology (IT) systems. For example, performance problems with the IT system used to manage NRFU led to processing backlogs. Although the Bureau developed workarounds for the issue, it hindered the Bureau’s ability to fully implement quality-assurance procedures as planned. The Bureau generally completed other follow-up operations designed to improve the accuracy of the data consistent with its plans. One of these activities was the vacant/delete check (VDC), where enumerators verified housing units thought to be vacant or nonexistent. The Bureau completed VDC two days ahead of schedule, but encountered duplicate addresses on the address list used for the operation, which could indicate a more systemic problem with the quality of the Bureau’s address list. While it will be important to refine existing census-taking activities— many of which have been in place since 1970—results of prior censuses point to the fact that simply improving current methods will not bring about the reforms needed to control costs and maintain accuracy. The cost of conducting the census has, on average, doubled each decade since 1970. At the same time, because of demographic and attitudinal trends, securing a complete count has become an increasing challenge. As a result, a fundamental reexamination of the nation’s approach to the census will be needed for a more cost-effective enumeration in 2020. Potential focus areas include new data collection methods; the tenure of the Census Director; and ensuring the Bureau’s approaches to human-capital management, knowledge sharing, and other internal functions are aligned toward delivering more cost-effective outcomes. The Bureau recognizes that fundamental changes are needed and has already taken some important first steps, including developing a strategic plan. To help ensure the Bureau’s efforts stay on track and to avoid problems it had in planning for prior censuses, it will be important for the Bureau to issue a comprehensive operational plan that includes performance goals, milestones, cost estimates, and other critical information that could be updated regularly. What GAO Recommends: GAO recommends that the Census Director refine NRFU and other field follow-up efforts by, among other things, emphasizing quality as much as speed during NRFU and by incorporating best practices in its IT acquisition-management policy. To help ensure reform efforts stay on track, the Bureau should develop an operational plan that integrates performance, budget, and other information. The Department of Commerce generally agreed with GAO’s findings and recommendations. View [hyperlink, http://www.gao.gov/products/GAO-11-193] or key components. For more information, contact Robert Goldenkoff at (202) 512-2757 or goldenkoffr@gao.gov. [End of section] Contents: Letter: Background: NRFU Was Generally Successful; Refinements Could Improve Procedures for 2020: Key Follow-up Operations Were Generally Completed as Planned: Fundamental Reforms Will Be Needed for a More Cost-Effective Census in 2020: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: Comments from the Department of Commerce: Appendix III: GAO Contact and Staff Acknowledgments: Related GAO Products: Figures: Figure 1: The Average Cost of Counting Each Housing Unit (in Constant 2010 Dollars) Has Escalated Each Decade While Mail Response Rates Have Declined: Figure 2: The Bureau Met Its Minimum Mail Response Rate Goal of 59 Percent in All but 11 States, but Rates Generally Declined Compared to 2000: Figure 3: The Expected and Actual Number of Cases Completed during NRFU: Figure 4: Digital Fingerprint Scanner: Abbreviations: AA: assignment area: Bureau: U.S. Census Bureau: CCM: census coverage measurement: FBI: Federal Bureau of Investigation: IT: information technology: LCO: local census office: MaRCS: Matching Review and Coding System: NPC: National Processing Center: NRFU: nonresponse follow-up: PBOCS: Paper-Based Operations Control System: PI: person interviewing: VDC: vacant/delete check: [End of section] United States Government Accountability Office: Washington, DC 20548: December 14, 2010: The Honorable Thomas R. Carper: Chairman: The Honorable John McCain: Ranking Member: Subcommittee on Federal Financial Management, Government Information, Federal Services, and International Security: Committee on Homeland Security and Governmental Affairs: United States Senate: The Honorable Darrell E. Issa: Ranking Member: Committee on Oversight and Government Reform: House of Representatives: The Honorable Wm. Lacy Clay: Chairman: The Honorable Patrick T. McHenry: Ranking Member: Subcommittee on Information Policy, Census and National Archives: Committee on Oversight and Government Reform: House of Representatives: One of the final acts of the decade-long census life cycle is to occur in the remaining days of 2010 when, as required by law, the U.S. Census Bureau (Bureau) is to release to the President the state population counts used to apportion Congress.[Footnote 1] Although some additional work and more data releases lie ahead, and information on the accuracy of the count is not scheduled to be available until early 2012, this much is clear: the Bureau generally completed the enumeration phase of the 2010 Census on schedule and consistent with its operational plans, and largely surmounted a series of risks that jeopardized the success of the headcount. As you know, an operationally successful census was no small accomplishment. Various social and demographic trends such as an increasingly diverse population and a distrust of government made a complete count an extraordinary challenge in 2010. At the same time, the Bureau had to overcome a variety of internal management challenges including shortcomings with critical information technology (IT) systems. We have long reported that the decennial census is a shared national undertaking, where the Bureau, Congress, government agencies at all levels, private organizations, and ultimately the public at large, all play vital roles in securing a complete count. That the Bureau completed key operations on schedule, obtained an acceptable participation rate, and is on track for meeting legally mandated deadlines for reporting population figures is a tremendous credit to the people of this nation for completing their census forms and cooperating with enumerators; the hundreds of thousands of career and temporary Bureau employees who diligently implemented a vast array of census-taking activities, often under difficult circumstances; public, private, tribal, and nonprofit organizations of all sizes for voluntarily partnering with the Bureau and raising awareness of the census; and finally to Congress, which provided the necessary support while holding the Bureau accountable for results. Despite these impressive achievements, the 2010 Census required an unprecedented commitment of resources, including recruiting more than 3.8 million total applicants--roughly equivalent to the entire population of Oregon--for its temporary workforce; and it escalated in cost from an initial estimate of $11.3 billion in 2001 to around $13 billion, the most expensive population count in our nation's history. Further, our oversight of the 1990, 2000, and now 2010 Censuses suggests that the fundamental design of the enumeration--in many ways unchanged since 1970--is no longer capable of delivering a cost- effective headcount given the nation's increasing diversity and other sociodemographic trends. Indeed, beginning in 1990, we reported that rising costs, difficulties in securing public participation, and other long-standing challenges required a revised census methodology, a view that was shared by other stakeholders.[Footnote 2] For 2010, the Bureau eliminated the long- form questionnaire in an effort to boost response rates, and refined other census-taking activities, but the basic approach to the enumeration is essentially the same as it was 40 years ago, and achieving acceptable results using these conventional methods has required an increasingly larger investment of fiscal resources, resources that in the coming years will become increasingly scarce. In short, as the nation turns the corner on the 2010 Census, it will be vitally important to both identify lessons learned from the current decennial census to improve existing census-taking activities, as well as to reexamine and perhaps fundamentally transform the way the Bureau plans, tests, implements, monitors, and evaluates future enumerations in order to address long-standing challenges. As requested, this report assesses the implementation of (1) nonresponse follow-up (NRFU), the largest and most costly census field operation, where the Bureau sends enumerators to collect data from households that did not mail back their census forms, and (2) other key follow-up field operations that were critical for ensuring a complete count; and (3) identifies key questions and focus areas that will be important for the Bureau, Congress, and census stakeholders to consider going forward now that planning for the next enumeration is underway. This report is one of three we are releasing today. Among the other two, one focuses on the Bureau's efforts to reach out to and enumerate hard-to-count populations, while the other examines the implementation of operations aimed at reducing census coverage errors. Both reports identify preliminary lessons learned, as well as potential focus areas for improvement.[Footnote 3] In reviewing NRFU, we examined the pace of production, the fingerprinting of census workers as part of a background check, and the performance of a critical automated system. The follow-up operations we reviewed for this report include the vacant/delete check (VDC), where the Bureau verifies the status of housing units flagged earlier in the census as being unoccupied or nonexistent; and census coverage measurement (CCM), where the Bureau assesses the completeness and accuracy of the census count. For all three objectives, we (1) analyzed Bureau cost and progress data as well as planning and other pertinent documents; (2) conducted periodic surveys of the Bureau's 494 local census office (LCO) managers using a series of online questionnaires that asked about their experience in managing LCO activities; and (3) made field observations at 28 locations across the country selected for various factors such as their geographic and demographic diversity, and including parts of such urban areas as Atlanta, Boston, Chicago, Detroit, New Orleans, New York City, San Francisco, and Tucson, as well as less-populated areas such as Meridian, Mississippi, and New Castle, Delaware. We also interviewed Bureau officials at headquarters and LCO managers and staff, and reviewed our prior work on the planning and implementation of the 1990, 2000, and 2010 Censuses. Moreover, to help inform a reexamination of the nation's approach to the census, in addition to the above, we reviewed our prior work on governmentwide reexamination, as well as leading practices and attributes in the areas of IT management, organizational performance, collaboration, stewardship, and human capital.[Footnote 4] Appendix I includes additional information on our scope and methodology and a list of LCOs we visited. Data presented in this report measuring operational timeliness and data quality were drawn from Bureau management and operational data systems. To assess the reliability of the data, we reviewed Bureau electronic documentation to gain information about the data and their sources, and followed up with agency officials knowledgeable about the data in cases where we had questions about potential errors or inconsistencies. On the basis of our efforts, we determined that the data were sufficiently reliable for the purposes of supporting the findings and recommendations in this report. We conducted this performance audit from December 2009 until December 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audits to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. On December 7, 2010, the Secretary of Commerce provided written comments on a draft of this report (see app. II). The Department of Commerce generally agreed with the overall findings and recommendations of the report. In addition, the Secretary of Commerce provided the Bureau's technical comments and suggestions where additional context might be needed, and we revised the report to reflect these comments where appropriate. Background: The decennial census is a constitutionally mandated enterprise critical to our nation. Census data are used to apportion congressional seats, redraw congressional districts, and help allocate hundreds of billions of dollars in federal aid to state and local governments each year. A complete count of the nation's population is an enormous challenge requiring the successful alignment of thousands of activities, hundreds of thousands of temporary employees, and millions of forms. Indeed, over the past year, in an effort to secure a complete count, the Bureau mailed out questionnaires to about 120 million housing units for occupants to complete and mail back; hand- delivered approximately 12 million questionnaires--mostly in rural locations as well as in areas along the Gulf Coast affected by recent hurricanes--for residents to fill out and return by mail; went door-to- door collecting data from the approximately 46.6 million households that did not mail back their census forms; and conducted operations aimed at counting people in less-conventional dwellings such as migrant-worker housing, boats, tent cities, homeless shelters, nursing homes, dormitories, and prisons. In short, the decennial census is large, logistically complex, and, at a cost now estimated at around $13 billion, expensive. In developing the 2010 Census, the Bureau faced three significant internal challenges: critical IT systems had performance problems during testing, cost-estimates lacked precision, and some key operations were not tested under census-like conditions. These were some of the issues that led us to designate the 2010 Census a GAO high- risk area in 2008.[Footnote 5] Although every census has its decade-specific difficulties, sociodemographic trends such as concerns over personal privacy, more non-English speakers, and more people residing in makeshift and other nontraditional living arrangements make each decennial increasingly challenging and do not bode well for the cost-effectiveness of future counts. As shown in figure 1, the cost of enumerating each housing unit has escalated from around $16 in 1970 to around $98 in 2010, in constant 2010 dollars (an increase of over 500 percent). At the same time, the mail response rate--a key indicator of a successful census-- has declined from 78 percent in 1970 to 63 percent in 2010. The mail response rate is an important figure because it determines the NRFU workload and ultimately, NRFU costs. In many ways, the Bureau has to invest substantially more resources each decade just to match the prior decennial's response rate. Figure 1: The Average Cost of Counting Each Housing Unit (in Constant 2010 Dollars) Has Escalated Each Decade While Mail Response Rates Have Declined: [Refer to PDF for image: combined vertical bar and line graph] Year: 1970; Average cost per housing unit (in constant 2010 dollars): $16; Mail response rate: 78%. Year: 1980; Average cost per housing unit (in constant 2010 dollars): $30; Mail response rate: 75%. Year: 1990; Average cost per housing unit (in constant 2010 dollars): $39; Mail response rate: 66%. Year: 2000; Average cost per housing unit (in constant 2010 dollars): $70; Mail response rate: 66%. Year: 2010; Average cost per housing unit (in constant 2010 dollars): $98 (projected); Mail response rate: 63%. Source: GAO analysis of Census Bureau data. Note: In the 2010 Census the Bureau used only a short-form questionnaire. For this report we use the 1990 and 2000 Census short- form mail response rate when comparing 1990, 2000, and 2010 mail-back response rates. Census short-form mail response rates are unavailable for 1970 and 1980, so we use the overall response rate. [End of figure] In our earlier work on high-performing organizations, we noted that the federal government must confront a range of new challenges to enhance performance, ensure accountability, and position the nation for the future.[Footnote 6] Nothing less than a fundamental transformation in the people, processes, technology, and environment used by federal agencies to address public goals will be necessary to address public needs. Ultimately, however, the federal government needs to change its culture to be more results-oriented. For the Bureau, as with all federal agencies, this means ensuring, among other things, that its culture embraces results rather than outputs; follows matrixes rather than stovepipes; forms partnerships rather than protecting turf; focuses on risk management rather than risk avoidance; and takes proactive approaches rather than behaving reactively. NRFU Was Generally Successful; Refinements Could Improve Procedures for 2020: The Bureau Met Its Response Rate Goal, but Recruited More Enumerators Than Needed and Should Revisit Its Staffing Model: Nationally, in terms of workload (as determined by the mail response rate) and staffing levels, the Bureau was well positioned to implement NRFU. With respect to the response rate, the Bureau expected a level of 59 percent to 65 percent. The actual mail response rate on April 19, when the Bureau initially determined the universe of houses to visit for NRFU, was just over 63 percent, well within the Bureau's range of estimates. This translated into an initial workload of 48.6 million housing units. Achieving this response rate was an important accomplishment as the nation's population is growing steadily larger, more diverse, and according to the Bureau, increasingly difficult to find and reluctant to participate in the census. High response rates are essential because they save taxpayer dollars. According to the Bureau, for every 1 percentage point increase in mail response in 2010, the Bureau saved $85 million that would otherwise have been spent on the follow-up efforts. According to the Bureau, it costs 42 cents to mail back each census form in a postage-paid envelope, compared with an average estimate of around $57 for field activities necessary to enumerate each housing unit in person. Moreover, mail returns tend to have better-quality data, in part because as time goes on after Census Day (April 1), people move, or may have difficulty recalling who was residing with them. As illustrated in figure 2, the Bureau met its expected response rate in all but 11 states. The highest response rate (71.7 percent) was in Minnesota, while the lowest response rate (51 percent) was in Alaska. At the same time, response rates in all but two states--Hawaii and South Carolina--as well as the District of Columbia, declined anywhere from 0.8 to 8.2 percentage points when compared to 2000, thus underscoring the difficulty the Bureau will face in the future in trying to sustain response rates.[Footnote 7] The mail response rate is important because it helps the Bureau determine the housing units that failed to mail back the census questionnaires, and thus are included in the NRFU workload. Figure 2: The Bureau Met Its Minimum Mail Response Rate Goal of 59 Percent in All but 11 States, but Rates Generally Declined Compared to 2000: [Refer to PDF for image: illustrated U.S. map] 2010 mail response rates: Met Bureau minimum mail response rate goal: Arkansas: 59.5% (-5.2); California: 64.6% (-4.3); Colorado: 64.6% (-4.9); Connecticut: 66.1% (-2.8); Delaware: 60.4% (-2.2); District of Columbia: 60.4% (1.5); Florida: 59.1% (-3.6); Georgia: 59.2% (-5); Hawaii: 60.1% (1); Idaho: 74.7% (-2.2); Illinois: 67.3% (-0.8); Indiana: 67% (-1.9); Iowa: 71.3% (-3.4); Kansas: 67.4% (-2.3); Kentucky: 62.6% (-2.6); Maryland: 66.1% (-1.7); Massachusetts: 65% (-3.5); Michigan: 65.4% (-5.8); Minnesota: 71.7% (-2.8); Missouri: 65.5% (-3.2); Montana: 63% (-5.4); Nebraska: 69.1% (-5.1); New Hampshire: 61.8% (-5.0); New Jersey: 64.1% (-3.2); New York: 60.4% (-2.2); North Carolina: 61.5% (-0.9); North Dakota: 67.1% (-4.7); Ohio: 66.2% (-5.0); Oregon: 63.9% (-2.5); Pennsylvania: 67.6% (-2.1); Rhode Island: 62.4% (-3.7); South Carolina: 60.7% (2.7); South Dakota: 65.4% (-8.2); Tennessee: 63.4% (-0.8); Texas: 60% (-3.1); Utah: 65.3% (-1.8); Virginia: 66.2% (-4.1); Washington: 63.4% (-1.7); Wisconsin: 71.4% (-4.3); Wyoming: 61.3% (-3.8). Did not meet Bureau minimum mail response rate goal: Alabama: 58.6% (-2.1); Alaska: 51% (-4.8); Arizona: 58.2% (-4.5); Louisiana: 57.4% (-2.3); Maine: 55.2% (-6.2); Mississippi: 57.5% (-4.1); Nevada: 58.4% (-6.5); New Mexico: 56.7% (-4.0); Oklahoma: 58.8% (-5.2); Vermont: 57.9% (-4.2); West Virginia: 56.9% (-6.7). Source: GAO analysis of preliminary Census Bureau data; Map Resources (map). Note: Number reflects the 2010 response rate as of April 19, 2010. Number in parentheses reflects the percentage change in response rate from 2000 to 2010. 2000 response rate as of April 18, 2000. [End of figure] The mail response rate differs from the participation rate in that it is calculated as a percentage of all housing units in the mail-back universe, including those that are later found to be nonexistent or unoccupied. In contrast, the participation rate is the percentage of forms mailed back by households that received them and is a better measure of cooperation with the census. According to a Bureau press release dated October 21, 2010, the nation achieved a final mail participation rate of 74 percent, matching the final mail participation rate that was achieved for the 2000 Census. Compared to 2000, participation rates for 22 states and the District of Columbia, either met or exceeded their 2000 Census rate. Key factors aimed at improving the mail response rate included the mailing of an advance letter, a reminder postcard, and an aggressive marketing and outreach program. In addition, this is the first decennial census the Bureau sent a second or "replacement" questionnaire to households. Replacement questionnaires were sent to around 25 million households in census tracts that had the lowest response rates in the 2000 Census, and 10 million replacement questionnaires were sent to nonresponding households in other census tracts that had low-to-moderate response rates in 2000. With respect to staffing levels, the Bureau set a recruitment goal of nearly 3.7 million total applicants and achieved 104 percent of this goal by April 25, 2010, recruiting more than 3.8 million total applicants, almost a week prior to the start of NRFU (once the Bureau had an adequate pool of candidates for 2010, it attempted to limit the number of additional applicants, taking such steps as discontinuing the advertising of census jobs in mailed-out census materials). According to the Bureau, based on past experience, it set its recruiting goal at five times the number of persons that needed to be trained to ensure it had an ample pool of candidates in specific areas with specific skills, as well as to ensure it had a sufficient supply of enumerators during the course of its field operations. The Bureau's approach was similar to that used for the 2000 Census despite vast differences in the economy. During the 2000 Census, the Bureau was recruiting in the midst of one of the tightest labor markets in nearly three decades. In contrast, during the 2010 Census, the Bureau was recruiting workers during a period of high unemployment. While having too few enumerators could affect the Bureau's ability to complete NRFU on schedule, overrecruiting has its own costs. For example, there are costs associated with administering and processing the test taken at the time an individual applies for a census job, as well as a $2 charge to have a name background check run on all applicants. Overrecruiting can also be burdensome on applicants as they need to find a test site and take a test before they can be hired for a census job--a job that because the Bureau has overrecruited, may not be available. In looking forward to 2020, it will be important for the Bureau to more precisely refine its recruiting model based on lessons learned from the labor markets in both 2000 and 2010, and use this information to develop more accurate recruiting targets. It will also be important for the Bureau to adhere to recruiting goals so that additional costs are not incurred. The Bureau Completed NRFU $660 Million under Budget: The Bureau budgeted that NRFU would cost around $2.25 billion. However, by the end of the operation, the Bureau reported using approximately $1.59 billion, which was 29 percent lower than budgeted. The Bureau, with congressional approval, also set up a contingency fund of $574 million to cover additional expenses that could have been caused by unfavorable weather and other unforeseen events. However, in the end, contingency money was not needed to complete NRFU. While the Bureau conducted NRFU under budget, the difference between actual and expected NRFU costs also highlights the need for the Bureau to develop an accurate cost model in order to establish more credible cost estimates for 2020. In addition to NRFU, other census operations had substantial variances between their initial cost estimates and their actual costs. In our 2008 report, we noted that the Bureau had insufficient policies and procedures and inadequately trained staff for conducting high-quality cost estimation for the decennial census, and recommended that the Bureau take a variety of steps to improve the credibility and accuracy of its cost estimates, including performing sensitivity and uncertainty analyses.[Footnote 8] The Bureau generally agreed with our recommendation and is taking steps to address them. Most Local Census Offices Finished NRFU ahead of Schedule, but the Bureau's Ambitious Production Schedule May Have Produced Mixed Results: In conducting NRFU, it is important for enumerators to follow Bureau procedures for collecting complete and accurate data while keeping production on schedule so that subsequent activities can begin as planned. Timely completion of NRFU is also important because as time goes on, people move or might have difficulty remembering who was living in a household on Census Day. The Bureau went to great lengths to obtain complete data directly from household members. For example, Bureau procedures generally called for enumerators to make six attempts to reach each household on different days of the week at different times until they obtained needed information on that household. However, in cases where household members could not be contacted or refused to answer all or part of the census questionnaire, enumerators were permitted to obtain data via proxy (a neighbor, building manager, or other nonhousehold member presumed to know about its residents), or if an enumerator after the required six attempts was unable to collect data from either the household or a proxy respondent then the enumerator submitted the incomplete questionnaire to the LCO (this is referred to as a "closeout interview"). Closeout interviews are processed at Bureau headquarters where statistical methods are used to determine household information. For the 2010 Census, NRFU began May 1 and was scheduled to finish July 10, 2010. However, a majority of LCOs generally finished their NRFU workloads ahead of this 10-week time frame. For example, by June 28, 2010, week 8 of the NRFU operation, 342 of the Bureau's 494 LCOs (almost 69 percent), had completed 100 percent of their workload. Figure 3 shows the production levels over the course of NRFU. Figure 3: The Expected and Actual Number of Cases Completed during NRFU: [Refer to PDF for image: multiple line graph] Week of nonresponse follow-up: 1; Percentage of cases completed: 0; Percentage of cases expected to be completed: 0. Week of nonresponse follow-up: 2; Percentage of cases completed: 8; Percentage of cases expected to be completed: 13. Week of nonresponse follow-up: 3; Percentage of cases completed: 21; Percentage of cases expected to be completed: 25. Week of nonresponse follow-up: 4; Percentage of cases completed: 35; Percentage of cases expected to be completed: 40. Week of nonresponse follow-up: 5; Percentage of cases completed: 62; Percentage of cases expected to be completed: 55. Week of nonresponse follow-up: 6; Percentage of cases completed: 82; Percentage of cases expected to be completed: 65. Week of nonresponse follow-up: 7; Percentage of cases completed: 94; Percentage of cases expected to be completed: 79. Week of nonresponse follow-up: 8; Percentage of cases completed: 98; Percentage of cases expected to be completed: 89. Week of nonresponse follow-up: 9; Percentage of cases completed: 99.73; Percentage of cases expected to be completed: 96. Week of nonresponse follow-up: 10; (Scheduled end of nonresponse follow-up 2010); Percentage of cases completed: 99.99; Percentage of cases expected to be completed: 99. Source: GAO analysis of Census Bureau data. [End of figure] A number of factors helped most LCOs complete NRFU ahead of schedule. For example, the Bureau removed almost 2 million late mail returns prior to the start of NRFU, reducing the follow-up workload from 48.6 million to 46.6 million housing units (a 4 percent reduction in NRFU workload). The removal of the late mail returns resulted in a 1.5 percent increase in mail response rate, saving approximately $127.5 million (based on the Bureau's estimate that a 1 percentage point increase in the mail response rate would decrease workload costs by around $85 million). Another factor that was instrumental to the success of NRFU was retaining a sufficiently skilled workforce. Because of high unemployment rates, turnover was far lower than anticipated. Advertising census jobs locally helped to ensure an adequate number of applicants, and according to the Bureau, mileage reimbursement may have been lower, in part because enumerators lived in and had local knowledge about the neighborhoods they were assigned. Further, people may have been more willing to cooperate with enumerators who were from their own community. For example, at a Native American village in New Mexico, local enumerators were aware that according to the community's customs it was considered offensive to launch into business without first engaging in conversation. In addition, local enumerators in hurricane-affected rural areas of Louisiana were able to successfully locate households based on their knowledge of the geography. For example, based on his familiarity with the area, one enumerator we observed was able to locate an assigned household not included on a census map using only a brief description, such as "a white house with green roof away from the road." The Bureau also used weekly production goals that helped LCOs focus on the need to stay on schedule and to track their progress. However, several measures we reviewed underscored the challenge that LCOs faced in hitting these production goals while still maintaining data quality. Significantly, our analysis of Bureau data found that the fast pace of NRFU was associated with the collection of less-complete household data.[Footnote 9] Indeed, after controlling for such variables as response rate and local enumeration challenges, we found that LCOs with higher percentages of proxy interviews and closeout interviews were more likely to have finished NRFU in 53 days or less (the average amount of time LCOs took to complete their NRFU workloads) compared to LCOs with lower percentages of proxy and closeout interviews. As noted above, proxy interviews contain data provided by a nonhousehold member (e.g., a neighbor) and may thus be less reliable than information collected directly from a household member, while a closeout interview is one where no interview is conducted and household information is later determined using statistical methods at Bureau headquarters during data processing. The pace of NRFU and its potential effect on data quality was also reflected in the responses of a number of LCO managers we surveyed. For example, although almost half of the LCO managers responding to our survey said they were satisfied with their ability to meet production goals while maintaining data quality, almost 30 percent of respondents were dissatisfied with their ability to meet production goals while maintaining data quality (around 20 percent responded that they were neither satisfied nor dissatisfied). Some of the LCO managers commented that they felt undue pressure to finish the operation early (sometimes a month earlier than planned) and as a result, finishing early could have come at the expense of accuracy. In one example, an LCO manager noted that it appeared as though the LCOs were in a race to finish NRFU as fast as possible, even if the best data were not received. Another LCO manager said that even though his office was surpassing the daily production goals, he was still being pressured to finish faster, and that accuracy was not mentioned. Also, LCO managers expressed frustration at production goals being changed frequently or unexpectedly moved earlier. Further, during our field visits, some LCO managers we spoke with at the start of NRFU were concerned about meeting production goals as there were not enough assignment area (AA) binders containing maps and address registers for every enumerator due to problems with the Bureau's Paper-Based Operations Control System (PBOCS), a key IT system that we discuss below. To ensure that enumerators had sufficient work, some crew leaders split-up AA binders between two or more enumerators. This is contrary to Bureau procedures which require enumerators to have their own AA binder. When the binders are split, only one enumerator has the required maps. Without maps an enumerator is unable to determine an assignment area's boundaries and ensure that the locations of all housing units are accurately recorded, which can affect data quality. Later in NRFU, managers at two LCOs we visited said they felt pressure to finish NRFU ahead of schedule. At one LCO, managers explained that the regional office wanted to finish NFRU by June 12, or approximately 4 weeks ahead of schedule. However, at that LCO they were only 85 percent complete by week 5, and because NRFU procedures instruct enumerators to make up to six attempts to contact a household, they were not sure how they were going to finish by week 5 without having to accept more refusals and incomplete interviews--leading to potentially more proxy and closeout interviews, thus reducing data quality. At the other LCO, production goals were stretched 15 percentage points above the national goal in order to complete NRFU ahead of schedule. One of the field supervisors at that office told us that he was able to meet the revised production goals by having enumerators share their workload. For example, in the morning, one enumerator would work the AA, and any remaining cases were given to another enumerator in the evening to complete. While this approach might have enhanced efficiency, the sharing of enumerator assignments makes it more difficult for the Bureau's quality-assurance procedures to identify enumerators that are not following procedures and may need to be retrained. Under the Bureau's procedures, AAs are to be assigned to one enumerator at a time. In late-May 2010, while NRFU was still underway, we discussed the pace of the operation with Bureau officials, and whether enumerators were more often accepting less-complete household information. In response, Bureau officials notified the LCOs and reminded them of the importance of following prescribed procedures. Moving forward, as the Bureau conducts its evaluations of its 2010 NRFU operation and begins planning for 2020, it will be important for Bureau officials to closely examine the quality of data collected during NRFU and the pace of the operation, and determine whether it is placing appropriate emphasis on both objectives. The Bureau Improved Its Procedures for Fingerprinting Employees, but More Work Is Needed: To better screen its workforce of hundreds of thousands of temporary census workers, the Bureau fingerprinted its temporary workforce for the first time in the 2010 Census.[Footnote 10] In past censuses, temporary workers were only subject to a name background check that was completed at the time of recruitment. The Bureau, however, encountered problems capturing fingerprints during address canvassing, an operation that the Bureau conducted in the summer of 2009 to verify every address in the country. According to the Bureau, 22 percent of the approximately 162,000 workers hired for address canvassing had unclassifiable prints, or fingerprints that were incomplete or unreadable. The Federal Bureau of Investigation (FBI) determined that this problem was generally the result of errors that occurred when the prints were first taken at the LCOs that affected the readability of the two fingerprint cards that were created for each individual. To address these problems, the Bureau improved its training procedures and purchased additional equipment in order to fingerprint some 580,000 NRFU temporary employees. Specifically, the Bureau refined training manuals used to instruct LCO staff on how to take fingerprints, scheduled fingerprint training closer to when the prints were captured, and increased the length of training. Further, the Bureau used an oil-free lotion during fingerprinting that is believed to raise the ridges on fingertips to improve the legibility of the prints. The Bureau also revised its procedures to digitally capture a third and fourth set of fingerprints when the first two sets of fingerprint cards could not be read. The Bureau purchased around a thousand digital fingerprint scanners (see fig. 4) for this new effort. The Bureau estimated that this additional step could reduce the percentage of temporary workers with unclassifiable prints from 22 percent to approximately 10 to 12 percent, or an estimated 60,000 to 72,000 temporary workers for NRFU. As of May 25, 2010, it reduced the percentage of temporary workers with unclassifiable prints to 8.6 percent of 635,251, or approximately 54,000 temporary workers. Figure 4: Digital Fingerprint Scanner: [Refer to PDF for image: photograph] Source: GAO (May 2010). [End of figure] Fingerprint cards were sent from each LCO to the Bureau's National Processing Center (NPC) in Indiana where they were scanned and sent to the FBI. We visited the NPC during peak processing and observed that NPC was able to adequately handle the workload without any glitches. However, capturing fingerprints at training sites did not go as well. Some LCOs mentioned that collecting fingerprints took more time than expected, thus reducing the time available for enumerator field training. In our observations, at one LCO it took an extra 2 hours to fingerprint enumerators, and at another fingerprinting took so long it carried over to the next day (which put the NRFU instructor behind schedule). Furthermore, almost 50 percent of LCO managers responding to our survey reported dissatisfaction with fingerprinting procedures, compared to about 30 percent of LCO managers who were satisfied. For example, LCO managers commented that they did not have enough time to train staff conducting the fingerprinting or did not have adequate fingerprinting supplies, such as cards and ink pads. Several LCO managers said that the process was time-consuming, yet the additional time spent did not produce higher-quality prints, possibly because staff did not have fingerprinting expertise. Although some LCO managers said they would have preferred more digital fingerprinting, others reported that the digital fingerprint scanners did not work well and were time-consuming to use. In looking forward to 2020, the Bureau should revise or modify training so that field staff are provided with numerous practice opportunities for collecting fingerprints prior to each operation. Workarounds Helped Mitigate PBOCS Issues, but Continuing Problems Hampered the Implementation of Key Quality-Assurance Procedures: Since 2005, we have reported on weaknesses in the Bureau's management and testing of key 2010 Census IT systems. Although the IT systems ultimately functioned well enough for the Bureau to carry out the census, workarounds developed to address performance problems with PBOCS--a workflow-management system crucial for the Bureau's field operations--adversely affected the Bureau's ability to implement key quality-assurance procedures as planned. In June 2005, we noted that the Bureau had not fully implemented key practices important to managing IT, including investment management, system development and management, and enterprise architecture [Footnote 11] management.[Footnote 12] As a result, we concluded that the Bureau's IT investments were at increased risk of mismanagement, and were more likely to experience cost and schedule overruns and performance shortfalls. As development of the IT systems progressed, these problems were realized. For example, the Field Data Collection Automation program, which included the development of handheld computers to collect information for address canvassing and NRFU, experienced substantial schedule delays and cost increases.[Footnote 13] As a result, the Bureau later decided to abandon the planned use of handheld data collection devices for NRFU and reverted to paper questionnaires. The Bureau developed PBOCS to manage the operation. However, as we stated in several testimonies, even with the approach of Census Day, PBOCS had not demonstrated the ability to function reliably under full operational loads required to complete NRFU.[Footnote 14] We noted that the limited amount of time remaining to improve its reliability before it would be needed for key operations created a substantial challenge for the Bureau. Although the Bureau worked aggressively to improve PBOCS performance, the system experienced significant issues at the start of NRFU. For example, despite efforts to upgrade its hardware and software, PBOCS continued to experience system outages, slow performance, and problems generating and maintaining timely progress reports. The Bureau has attributed these issues, in part, to the compressed development and testing schedule, as well as to inadequate performance and interface testing. To mitigate the system's performance issues, the Bureau implemented various workarounds. For example, the Bureau frequently restricted the number of hours that PBOCS was available to users in order to implement software upgrades and perform other system maintenance activities. In addition, the Bureau restricted the number of concurrent PBOCS users at each LCO to help reduce demand on the system. These restrictions often limited the number of concurrent users to 3 to 5 users per LCO, or about 1,500 to 2,500 total users. According to a Bureau official with responsibility for PBOCS, the system was originally intended to provide access for over 7,000 concurrent users. While these workarounds improved the reliability of PBOCS, LCO managers who responded to our survey were consistently dissatisfied with the restrictions on the number of users allowed at one time, and many commented that the restrictions adversely affected their ability to keep up with the workload. Further, the limitations on the number of concurrent users, combined with PBOCS outages and slow performance, delayed the shipping of questionnaires to the data capture centers and resulted in a peak backlog of nearly 12 million questionnaires at the LCOs. The substantial backlog of questionnaires hampered the Bureau's ability to effectively monitor productivity and data quality during NRFU as planned. Nearly 75 percent of LCO manager survey respondents were dissatisfied with the usefulness of PBOCS reports to plan and monitor work during NRFU. A dissatisfied respondent wrote in that the unavailability of reports greatly hampered his LCO's ability to conduct NRFU in an efficient manner. Almost 80 percent of responding LCO managers indicated that their LCO needed to put forth a substantial amount of extra effort to manually prepare reports to track productivity outside of PBOCS. The use of manual processes increased costs at the LCOs and raised the risk of human error. The backlog of questionnaires also hampered the Bureau's ability to conduct NRFU reinterviews, a quality-assurance operation designed to identify enumerators who intentionally or unintentionally produced data errors. PBOCS was to select a sample of cases from each enumerator's completed workload, and these cases would be reinterviewed by another enumerator. Once cases were selected, a quality-assurance enumerator attempted to reinterview the original NRFU respondents in an effort to verify that accurate data was collected during the initial NRFU interview. However, the backlog of questionnaires delayed the selection of reinterview cases and, as a result, some could not be conducted. For example, in areas with large populations of college students, the Bureau conducted NRFU early in order to maximize the probability of enumerating people before they were likely to move out from where they were living on Census Day. In some of those cases, reinterviews could not be conducted since the students had moved out by the time an enumerator was given the case for reinterview. In addition, it also took longer to detect and retrain an enumerator with performance problems. For example, LCO staff reported to us that, because of the delay carrying out reinterviews, it was often too late to retrain enumerators because they had already finished their assignments and were released before the errors were identified. In cases where an enumerator had intentionally falsified work, the enumerator was supposed to be released and all his or her work was to be redone. However, because of the PBOCS delays, falsified cases were sometimes identified after the enumerator was finished with his or her assignment, requiring their entire assignment area to be reenumerated. Identifying errors and falsifications early in the operation would have minimized the number of housing units that needed to be reworked and reduced the burden for respondents. For example, an LCO manager told us that her office was not able to detect an enumerator's falsification until after NRFU, when the enumerator had already moved on to the next operation, requiring the LCO to rework nearly 200 cases. According to our survey, approximately 30 percent of LCO managers who experienced backlogs reported that they had substantial difficulty detecting errors or fraudulent interviewing as a result of the backlog, while more than 20 percent reported moderate difficulty and nearly 50 percent reported slight to no difficulty detecting errors or fraudulent interviewing as a result of the backlog. The implementation of various workarounds helped the Bureau successfully complete NRFU. However, the lack of a fully functioning PBOCS limited the Bureau's ability to effectively monitor productivity or implement quality-assurance procedures as documented in its operational plans. More generally, as the Bureau prepares for 2020, among other actions it will be important for it to continue to improve its ability to manage its IT investments. Leading up to the 2010 Census, we made numerous recommendations to the Bureau to improve its IT management practices by implementing best practices in risk management, requirements development, and testing, as well as establishing an IT acquisition-management policy that incorporates best practices. [Footnote 15] While the Bureau implemented many of our recommendations, it did not implement our broader recommendation to institutionalize these practices at the organizational level. The challenges experienced by the Bureau in acquiring and developing IT systems during the 2010 Census further demonstrate the importance of establishing and enforcing a rigorous IT acquisition management policy Bureau-wide. In addition, it will be important for the Bureau to improve its ability to consistently perform key IT management practices, such as IT investment management, system development and management, and enterprise architecture management. The effective use of these practices can better ensure that future IT investments will be pursued in a way that optimizes mission performance. Key Follow-up Operations Were Generally Completed as Planned: Vacant/Delete Check Operation Finished ahead of Schedule but over Budget: To help ensure that people are counted only once and in the right place, as well as to collect complete and correct information about them, after NRFU the Bureau conducts a number of operations designed to improve the accuracy of the data. One of these operations is the VDC operation, where enumerators verified the Census Day status of vacant and deleted (nonexistent) housing units. VDC also attempts to enumerate late additions to the Bureau's address file, such as newly constructed housing, and units for which the mail-out questionnaire was returned blank or incomplete. The Bureau refers to these additional addresses as supplemental cases. VDC has the potential to boost the accuracy of the census, especially among traditionally undercounted populations. A similar operation in 2000 found that 22 percent of housing units previously identified as vacant, and 25 percent of those previously flagged for deletion, were indeed occupied. Changing the status of these units led to a net gain of 3.1 million people in the 2000 population count. The Bureau completed the VDC operation on August 23, slightly ahead of the original planned completion date of August 25, but also over budget. The Bureau spent about $281 million on VDC, approximately 15 percent over its baseline budget of $244 million. Bureau officials we spoke to attributed the operation's progress to the retention of experienced NRFU staff for VDC. They noted that VDC staff were knowledgeable about procedures and the locations in which they worked, and required less training than they would have if they had been newly hired. With respect to the cost overruns, the Bureau is analyzing why VDC exceeded its budget. According to a Bureau official, additional costs may be related to VDC cases being located farther apart than expected (which would require more staff time and mileage reimbursement) and to enumerators adding more new addresses than expected. The VDC workload of 8.7 million housing units (5.6 million units vacant or flagged for deletion, 2.9 million supplemental addresses, and 0.2 million additions during the operation) was substantially less than the Bureau's previous estimate of 10.4 to 15.4 million units. During our review we found that while the Bureau had updated its total cost estimates for VDC, it had not adjusted day-to-day cost and progress expectations for VDC to account for the reduced workload. Not having the most recent targets for VDC could have impeded the Bureau's ability to effectively monitor the progress of enumerators in the field. We discussed this with Bureau officials, and in mid-July they revised VDC cost and progress estimates to account for the smaller workload, as well as other changes, including an earlier start date and reduced staffing. Further, during our field observations, LCO staff told us that some VDC supplemental addresses had already been enumerated as occupied units during NRFU. These supplemental addresses were slightly different from the NRFU addresses (e.g., 123 Main Street versus 123A Main Street) and appeared to be duplicate addresses. Duplicate addresses are supposed to be checked during field verification (an operation to confirm the existence of certain housing units added to the Bureau's address file) and should not have been in the VDC workload. Because the issue could indicate a nationwide problem, we notified Bureau officials, and in response they instituted a new procedure to identify and process duplicate addresses without making a follow-up visit to the housing unit. Identifying duplicate addresses before they get enumerated a second time is important because unnecessarily visiting a housing unit previously counted can reduce the accuracy of census data and will increase costs. In order to assess the reasons why VDC ran over budget, and as recommended in our June 2008 report, it will be important for the Bureau to document lessons learned for cost elements whose actual costs differ from the estimate.[Footnote 16] Knowing this will allow the Bureau to develop a more accurate cost estimate for VDC in 2020. In addition, to ensure the accuracy of data collected during VDC, it will be important for the Bureau to research how duplicates were inadvertently included in the VDC workload, as this data will help the Bureau compile a better address list for VDC operations in 2020. Census Coverage Measurement Redesigned with Smaller Sample to Reduce Nonsampling Errors: The Bureau attempts to conduct a complete and accurate count of the nation's population; nonetheless, some degree of coverage error is inevitable because of the inherent complexity of counting the nation's large and diverse population and limitations in census-taking methods. These census coverage errors can take a variety of forms, including a person missed (an undercount), a person counted more than once (an overcount), or a person who should not have been counted, such as a child born after Census Day (another type of overcount). And because census data are central to so many critical functions, it is essential to assess census accuracy and improve the process when possible. Statistical measurements of census coverage are obtained by comparing and matching the housing units and people counted by an independent sample or CCM survey to those counted by the census in and around the sample areas. The Bureau has developed separate address lists--one for the entire nation of over 134 million housing units that it is using to conduct the census and one for coverage-measurement sample areas-- and is collecting each set of data through independent operations. The Bureau collected its CCM data from households in sample areas nationwide, as part of an operation that began in the middle of August and was completed in October 2010. In our April 2010 report, we noted that in December 2009 the Bureau made numerous changes to the design of CCM that would reduce nonsampling error--such as human errors made when recording data during interviews--in CCM and its resulting estimates of census accuracy, thus providing census data users with more-reliable estimates.[Footnote 17] These changes include increasing quality- assurance reinterviewing, hiring more CCM supervisors, and adding training for interviewers to improve interview techniques for local or other special situations (such as interviewing people who became homeless or have had to move frequently during the housing crisis). The December decision also reduced the CCM sample size by nearly 45 percent. The Bureau believes that this reduction will generate cost savings to pay for changes to reduce nonsampling error. We believe that, overall, these changes are reasonable efforts to improve survey quality. The Bureau's reduction in sample size will reduce precision of the estimates, yet the proposed changes should reduce nonsampling errors and thus provide users with more-reliable estimates. Another challenge highlighted in our April 2010 report on CCM was determining the optimal time to collect data for some 170,000 housing units during person interviewing (PI), which began August 14 and ended October 16, 2010. The issue is that if the Bureau starts PI too early, it increases the chance that it overlaps with census data collection, possibly compromising the independence of the two different sets of data and introducing what is referred to as a "contamination bias" error into CCM data. However, if the Bureau starts PI too late, it increases the chance that respondents will not accurately remember household information from Census Day, introducing error (known as "recall bias") in the CCM count. In that report we recommended that the Bureau assess the trade-offs between starting early and introducing contamination bias or starting later and risking recall bias. The Bureau responded that it planned to study and measure some recall errors, but that there was no study planned to measure contamination bias in 2010 due to concerns with the possible contamination of census results in the study area. However, since both types of errors--contamination bias and recall bias--could affect the Bureau's conclusions about the accuracy of the census, it will be important for the Bureau to implement our recommendation and assess the trade-offs between the two types of biases in timing decisions. Moreover, this assessment could help the Bureau better inform the optimal timing for future census and coverage-measurement data collection operations. Fundamental Reforms Will Be Needed for a More Cost-Effective Census in 2020: While it will be important to assess and revamp existing census-taking activities, the results of prior enumerations underscore the fact that simply refining current methods--many of which have been in place for decades--will not bring about the reforms needed to control costs while maintaining accuracy given ongoing and newly emerging societal trends. Since 1970, the Bureau has used essentially the same approach to count the vast majority of the population. The Bureau develops an address list of the nation's housing units and mails census forms to each one for occupants to fill out and mail back. Over time, because of demographic and attitudinal trends, securing an acceptable response rate has become an increasing challenge, and the Bureau has spent more money with each census in order to secure a complete count. Indeed, the cost of conducting the census has, on average, doubled each decade since 1970, in constant 2010 dollars. If that rate of cost escalation continues into 2020, the nation could be facing a $30 billion census. Despite the nation's greater investment in each census, the results are often no better than the previous decennial. For example, as noted earlier, while the unit cost of the census jumped from an average of around $70 in 2000 to around $98 in 2010, the mail response rate declined in 48 states. Our concerns about the rising cost and diminishing returns of the census are not new. In the mid-1990s, for example, we and others concluded that the established approach for taking the census in 1990 had exhausted its potential for counting the population cost-effectively and that fundamental design changes were needed to reduce census costs and improve the quality of data collected.[Footnote 18] A fundamental reexamination of the nation's approach to the census will require the Bureau to rethink its approach to planning, testing, implementing, monitoring, and evaluating the census, and addressing such questions as, why was a certain program initiated? What was the intended goal? Have significant changes occurred that affect its purpose? Does it use prevailing leading practices? As one example, a critical factor affecting the cost of the census is the necessity for the Bureau to follow up on nonresponding housing units. The hourly wages of enumerators, their productivity, mileage reimbursement, and the need, in some cases, to return several times to an address to obtain a response can all drive up costs. Administrative records from other government agencies including driver licenses and school records can, if used in lieu of making multiple visits to a housing unit, significantly control costs. However, the Bureau would first need to resolve a number of questions including the quality and the coverage of the information supplied by the records and the policy and legal implications of accessing them. On the basis of our earlier work on high-performing organizations, fundamental reforms will mean ensuring that the Bureau's organizational culture and structure, as well as its approach to strategic planning, human-capital management, internal collaboration, knowledge sharing, capital decision making, risk and change management, and other internal functions are aligned toward delivering more cost-effective outcomes.[Footnote 19] Indeed, some of the operational problems that occurred during the 2010 and prior censuses are symptomatic of deeper organizational issues. For example, the lack of staff skilled in cost-estimation during the 2010 Census points to inadequate human-capital planning, while IT problems stemmed from not fully and consistently performing certain functions including IT investment management. Going forward, it will be important for the Bureau, Congress, and other stakeholders to reach consensus on a number of reexamination areas, including the following, which have particular implications for controlling costs and improving accuracy: * Which data collection approaches, the Internet and administrative records among them, have potential to improve data quality without compromising other Bureau goals and mandates such as confidentiality and timeliness? * To what extent can private-sector and other sources of information such as maps, address lists, and geographic databases be employed to help support the census? * How can the Bureau enhance how it partners with government and nongovernmental organizations, data users, grassroots organizations, and advisory groups to obtain their input and possibly better leverage their knowledge and services? What is the best way of maintaining congressional and stakeholder involvement and dialog throughout the course of the decade? * What opportunities exist for the Bureau to leverage innovations in technology and social media to more fully engage census stakeholders and the general public throughout the decade on census issues, possibly identifying more cost-effective methods? * To what extent can the Bureau use the American Community Survey--an ongoing Bureau survey of population and housing characteristics that is conducted throughout the decade--as a platform to test new census methods and systems? * What are the implications of the Bureau's goal to conduct the 2020 Census at a lower cost than the 2010 Census on a cost per housing unit basis, adjusted for inflation? For example, how would this spending limit affect such considerations as accountability and data quality? * How can the Bureau best balance the acquisition of advanced technology, some of which might not be fully mature until later in the decade, with the need to commit to particular systems sufficiently early in the decade to ensure the systems are fully tested and will work under census-like conditions? * To what extent can the Bureau control costs and improve accuracy by targeting census-taking activities using local response rate and sociodemographic information from the 2010 Census, as well as other data sources and empirical evidence? * What options exist for controlling the costs of particularly labor- intensive operations such as NRFU and building the Bureau's master address list without sacrificing accuracy? * Can stakeholders reach agreement on a set of criteria that could be used to weigh the trade-offs associated with the need for high levels of accuracy on the one hand, and the increasing cost of achieving that accuracy on the other hand? The Bureau, recognizing that it cannot afford to continue operating the way it does unless it fundamentally changes its method of doing business, has already taken some important first steps in addressing these questions as well as other areas. For example, the Bureau is looking to reform certain aspects of its IT systems planning, in part to ensure that the technical infrastructure needed for 2020 will be tested many times before operations begin. The Bureau is also rebuilding its research directorate to lead early planning efforts, and has plans to assess and monitor the skills and competencies needed for the 2020 headcount and evaluate the feasibility of administrative records. Further, the Bureau has already developed a strategic plan for 2020 and other related documents that, among other things, lay out the structure of the Bureau's planning efforts; outline the Bureau's mission and vision for 2020 and the goals the Bureau seeks to meet to accomplish its mission; and describe the Bureau's plans for the research and testing phase of the next enumeration. The Bureau's early planning efforts are noteworthy given the Bureau's long-standing challenges in this area. For example, in 1988, just prior to the 1990 Census, we noted that the Bureau's planning efforts generally started late, experienced delays, were incomplete, and failed to fully explore innovative approaches.[Footnote 20] Planning for the 2000 Census also had its shortcomings. According to the Bureau, staff with little operational experience played key roles in the design process, which resulted in impractical reform ideas that could not be implemented. We also noted that the 2000 Census suffered from a persistent lack of priority-setting, coupled with minimal research, testing, and evaluation documentation to promote informed and timely decision making. And, while the planning process for the 2010 Census was initially more rigorous than for past decennials, in 2004 we reported that the Bureau's efforts lacked a substantial amount of supporting analysis, budgetary transparency, and other information, making it difficult for us, Congress, and other stakeholders to properly assess the feasibility of the Bureau's design and the extent to which it could lead to greater cost-effectiveness compared to alternative approaches.[Footnote 21] As a result, in 2004, we recommended that the Bureau develop an operational plan for 2010 that consolidated budget, methodological, and other relevant information into a single, comprehensive document. Although the Bureau later developed specific performance targets and an integrated project schedule for 2010, the other elements we recommended were only issued piecemeal, if available at all, and were never provided in a single, comprehensive document. Because this information was critical for facilitating a thorough, independent review of the Bureau's plans, as well as for demonstrating to Congress and other stakeholders that the Bureau could effectively design and manage operations and control costs, we believe that had it been available, it could have helped stave off, or at least reduce, the IT and other risks that confronted the Bureau as Census Day drew closer. The Bureau's strategic plan for 2020, first issued in 2009, is a "living" document that will be updated as planning efforts progress. As the approach for 2020 takes shape, it will be important for the Bureau to avoid some of the problems it had in documenting the planning process for the 2010 Census, and pull all the planning elements together into a tactical plan or road map. This will help ensure the Bureau's reform initiatives stay on track, do not lose momentum, and coalesce into a viable path toward a more cost-effective 2020 Census. On the basis of our work on planning for the 2010 Census, a road map for 2020 could include, but not be limited to, the following elements that could be updated on a regular basis: * specific, measurable performance goals, how the Bureau's efforts, procedures, and projects would contribute to those goals, and what performance measures would be used; * descriptions of how the Bureau's approaches to human-capital management, organizational structure, IT acquisitions, and other internal functions are aligned with the performance goals; * an assessment of the risks associated with each significant decennial operation, including the interrelationships between the operations and a description of relevant mitigation plans; * detailed milestone estimates for each significant decennial operation, including estimated testing dates, and justification for any changes to milestone estimates; * detailed life-cycle cost estimates of the decennial census that are credible, comprehensive, accurate, and well-documented as stipulated by Office of Management and Budget and GAO guidance; and: * a detailed description of all significant contracts the Bureau plans to enter into and a justification for the contracts. A comprehensive road map could generate several important benefits. For example, it could help ensure a measure of transparency and facilitate a more collaborative approach to planning the next census. Specifically, an operational plan could function as a template for 2020 giving stakeholders a common framework to assess and comment on the design of the census and its supporting infrastructure, the resources needed to execute the design, and the extent to which it could lead to greater cost-effectiveness compared to alternative approaches. Further, it could be used to monitor the Bureau's progress in implementing its approach, and hold the agency accountable for results. Importantly, to the extent the plan--or aspects of it--are made available using social media tools, it could prompt greater and perhaps more constructive civic engagement on the census, by fostering an ongoing dialog involving individuals and communities of stakeholders throughout the decade. On December 8, 2010, the Senate approved a bill, the Census Oversight Efficiency and Management Reform Act of 2010.[Footnote 22] If enacted, this bill, among its other provisions, would require the Director of the Census to submit an annual comprehensive status report on the next decennial census, beginning with the 2020 decennial census, to the appropriate congressional committees. The specific requirements in the bill for the annual plan include most of the elements discussed above. Given the magnitude of the planning and transformation efforts facing the Bureau, another reexamination question is that of long-term stewardship governing the endeavor. Specifically, as the research, development, and testing efforts for 2020 will play out over the decade-long census life cycle, what is the optimal way to ensure continuity and accountability for an enterprise that takes years to complete and extends beyond the tenure of many elected political leaders? Although the Director of the Census Bureau can, in concept, provide a measure of continuity, of the 11 census directors that have served since July 1969 (not including the current director), the average tenure was around 3 years, and only one director has served more than 5 years. Moreover, in the decade leading up to the 2010 Census, the Bureau was led by four different directors and several acting directors. The turnover in the Bureau's chief executive officer position makes it difficult to develop and sustain efforts that foster change, produce results, mitigate risks, and control costs over the long term. Currently, census directors are nominated by the President with Senate confirmation. At the same time, the heads of a number of executive agencies serve fixed appointments, including the Director of the Office of Personnel Management (4 years), the Commissioner of Labor Statistics (4 years), and the Commissioner of Internal Revenue (5 years). The census bill, recently passed by the Senate and discussed above, includes a provision for a 5-year tenure for the Census Director. We believe that the continuity resulting from a fixed-term appointment could provide the following benefits to the Bureau: * Strategic vision. The Director needs to build a long-term vision for the Bureau that extends beyond the current decennial census. Strategic planning, human-capital succession planning, and life-cycle cost estimates for the Bureau all span the decade. * Sustaining stakeholder relationships. The Director needs to continually expand and develop working relationships and partnerships with governmental, political, and other professional officials in both the public and private sectors to obtain their input, support, and participation in the Bureau's activities. * Accountability. The life-cycle cost for a decennial census spans a decade, and decisions made early in the decade about the next decennial census guide the research, investments, and tests carried out throughout the entire 10-year period. Institutionalizing accountability over an extended period may help long-term decennial initiatives provide meaningful and sustainable results. Overall, the obstacles to conducting a cost-effective census have grown with each decade, and as the Bureau looks toward the next enumeration, it might confront its biggest challenge to date. As the Bureau's past experience has shown, early investments in planning can help reduce the costs and risks of its downstream operations. Therefore, while Census Day 2020 is 10 years away, it is not too early for stakeholders to start considering the reforms needed to help ensure the next headcount is as cost-effective as possible. Conclusions: Although the complete results of the 2010 Census are still some years away, several preliminary lessons learned for the next enumeration have already begun to emerge. They include the benefits of a replacement questionnaire, the removal of late mail returns from the NRFU workload, and hiring locally. Focus areas for improvement include revisiting the Bureau's staffing model, ensuring the Bureau emphasizes quality as well as production during NRFU, better IT management, and ensuring a high-quality address file is used to carry out VDC operations. That said, perhaps the most important lesson learned comes from the collective experience gained from the 1990, 2000, and now 2010 enumerations: the Bureau goes to great lengths each decade to improve specific census-taking activities, but these incremental modifications have not kept pace with societal changes that make the population increasingly difficult to locate and count cost-effectively. Therefore, as the Bureau looks toward 2020, it will be important for it to reexamine both the fundamental design of the enumeration, as well as its management and culture to ensure that the Bureau's business practices and systems enhance its capacity to conduct an accurate count, control costs, manage risks, and be more nimble in adapting to social, demographic, technological, and other changes that can be expected in the years ahead. The Bureau is taking some initial steps toward rethinking the census. At the same time, past experience has shown that the Bureau cannot plan and execute a successful enumeration on its own. Indeed, the noteworthy achievements of the 2010 Census occurred because of the shared efforts of the Bureau, and its parent organizations the Department of Commerce and the Economics and Statistics Administration, Congress, and thousands of other parties. It will be important for these and additional stakeholders to maintain their focus on the census throughout the decade in order to achieve desired results. Certain census reforms could require legislative changes, and any new procedures will need to be thoroughly vetted, tested, and refined. Although the next enumeration is 10 years away, the groundwork for building a new census infrastructure is already under way. The bottom line is that while the urgency of the 2010 Census has subsided, it is by no means any less important to the nation. Recommendations for Executive Action: As the Bureau plans for the next decennial census in 2020, in order to support efforts to reexamine the fundamental design of the decennial census, and help refine existing operations should they be used again in the 2020 Census, we recommend that the Secretary of Commerce direct the Under Secretary of the Economics and Statistics Administration, as well as the Census Director, to take the following six actions: * To help enhance the Bureau's performance and accountability, improve the transparency of the planning process, gauge whether the Bureau is on-track toward a more cost-effective 2020 Census, and foster greater public dialog about the census, the Bureau should develop an operational plan or road map for 2020 that integrates performance, budget, methodological, schedule, and other information that would be updated as needed and posted on the Bureau's Web site and other social media outlets, and develop a mechanism that allows for and harnesses input from census stakeholders and individuals. * To refine its approach to recruiting, the Bureau should evaluate current economic factors that are associated with and predictive of employee interest in census work, such as national and regional unemployment levels, and use these available data to determine the potential temporary workforce pool and adjust its recruiting approach. * To help ensure that the Bureau's procedures for NRFU result in the collection of high-quality data, the Bureau's procedures for the timely completion of NRFU should emphasize the collection of high- quality data and proper enumeration techniques as much as speed. * To improve the fingerprinting process of temporary workers, the Bureau should revise or modify training so that field staff are provided with numerous practice opportunities for collecting fingerprints prior to each operation. * To ensure that the Bureau improves its ability to manage future IT acquisitions, the Bureau should immediately establish and enforce a system-acquisition management policy that incorporates best practices in system-and software-acquisition management. * To help ensure the Bureau compiles an accurate address list for VDC operations in 2020, the Bureau should research how duplicate addresses were inadvertently included in the VDC workload. Agency Comments and Our Evaluation: The Secretary of Commerce provided written comments on a draft of this report on December 7, 2010. The comments are reprinted in appendix II. The Department of Commerce generally agreed with the overall findings and recommendations of the report. In addition, the Secretary of Commerce provided the Bureau's technical comments and suggestions where additional context might be needed, and we revised the report to reflect these comments where appropriate. The Bureau noted that our report did not acknowledge the steps it took to modify its recruiting plans prior to NRFU. However, we do discuss the Bureau's modifications to its recruiting plans. Specifically, we stated that "once the Bureau had an adequate pool of candidates for 2010, it attempted to limit the number of additional applicants, taking such steps as discontinuing the advertising of census jobs in mailed out census materials." The Bureau also commented that it wanted to discuss our analysis that found that the fast pace of NRFU was associated with the collection of less-complete household data, noting that its own analysis of a similar question did not yield the same finding. On December 7, 2010, we met with Bureau staff to discuss the methodologies and variables used in each analysis. After discussing our methodology and results, Bureau staff explained that their analysis was preliminary and not as comprehensive as our analysis. Further, they acknowledged that they used a different methodology and different variables. The Bureau, in commenting on our finding related to fingerprinting temporary workers, noted that it was unclear as to ways in which extending training, which usually requires spending more time and money, would streamline fingerprinting efforts. To clarify this section we changed the body of the report. The text now reads, "In looking forward to 2020, the Bureau should revise or modify training so that field staff are provided with numerous practice opportunities for collecting fingerprints prior to each operation." We are sending copies of this report to the Secretary of Commerce, the Under Secretary of Economic Affairs, the Director of the U.S. Census Bureau, and interested congressional committees. The report also is available at no charge on GAO's Web site at [hyperlink, http://www.gao.gov]. If you have any questions on matters discussed in this report, please contact Robert Goldenkoff at (202) 512-2757 or by e-mail at goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix III. Signed by: Robert Goldenkoff: Director: Strategic Issues: [End of section] Appendix I: Objectives, Scope, and Methodology: The objectives of this report were to assess the implementation of (1) nonresponse follow-up (NRFU), the largest and most costly census field operation, where the U.S. Census Bureau (Bureau) sends enumerators to collect data from households that did not mail back their census forms, and (2) other key follow-up field operations that were critical for ensuring a complete count; and (3) identify key questions and focus areas that will be important for the Bureau, Congress, and census stakeholders to consider going forward now that planning for the next enumeration is underway. To meet our objectives, we used a combination of approaches and methods to examine the conduct of these operations. These included statistical analyses; surveys of the Bureau's 494 local census office (LCO) managers; analysis of mail response and form check-in rates from Bureau cost and progress systems; interviews with key bureau headquarters officials and LCO managers and staff; observation of LCO's NRFU operations; and reviews of relevant documentation including our prior work on the planning and implementation of the 1990, 2000, and 2010 Censuses. To examine the factors that affected the implementation of NRFU and vacant/delete check operations (VDC), we interviewed LCO managers and other supervisory staff and observed operations at 28 LCOs we visited across the country. We selected LCOs because they were located in hard- to-count areas as determined by data from the 2000 Census. To make these selections, we also used other factors such as their percentage of rural population to obtain diversity in urban/rural populations, and proximity to hurricane-affected areas. Selections for VDC observations were based primarily on locations with high rates of vacant and delete classifications, and they were chosen to include a mix of urban, suburban, and rural LCO located in all regions of the country. (See below for a complete list of the offices we visited.) During these visits, which took place from April to July 2010, we observed office operations to see how office staff were processing questionnaires using the Paper-Based Operations Control System (PBOCS) and capturing fingerprints with live scanners, attended enumerator training, and observed enumerators in the field go door-to-door to collect census data for NRFU, NRFU reinterview, and VDC. Because offices were judgmentally selected, our findings from these visits cannot be projected to the universe of LCOs. To obtain a national perspective on the conduct of NRFU and other field data collection operations, we conducted a panel survey of all 494 LCO managers from March to August 2010 using six questionnaires. The survey was designed to examine (1) factors that affect the cost and performance of local data collection efforts, and (2) LCO managers' satisfaction with information technology (IT) systems and other management support functions. Response rate was at least 75 percent for each survey questionnaire. The practical difficulties of developing and administering a survey may introduce errors--from how a particular question is interpreted, for example, or from differences in the sources of information available to respondents when answering a question. Therefore, we included steps in developing and administering the questionnaire to minimize such errors. For instance, we conducted pretests to check that (1) the questions were clear and unambiguous, (2) terminology was used correctly, (3) the questionnaire did not place an undue burden on agency officials, (4) the information could feasibly be obtained, and (5) the survey was comprehensive and unbiased. Pretest sites were selected for each wave to emphasize variation among urban and rural LCOs. Pretests were conducted over the phone, mostly as cognitive pretests in which the respondent completed the survey during the pretest. We made changes to the content and format of the questionnaire after review by a GAO survey expert and after each of the pretests, based on the feedback we received. To examine whether the pace of NRFU was associated with the collection of less-complete data, in addition to the efforts described above, we analyzed Bureau proxy and closeout rates, and the time it took for an LCO to complete the NRFU workload. In order to determine whether the durations of 2010 NRFU production activities were associated with lower-quality work, we conducted regression analyses using data from the Bureau's Cost and Progress System, PBOCS, and Matching Review and Coding System (MaRCS). These analyses assessed whether indicators of lower-quality enumeration such as the collection of proxy data from a neighbor and closeout interviews, where a housing unit is occupied but no interview was obtained, were associated with the number of days that the LCO spent conducting NRFU production activities, after adjusting for other factors associated with the timeliness of completion and workload. We used two regression models: one model tested the association between the number of days it took each LCO to complete 100 percent of its workload and quality factors; the other regression model tested the association between quick completion and quality factors. We also analyzed cost data weekly for both NRFU and VDC to determine whether those operations were within their respective budgets. To assess the reliability of the data, we reviewed Bureau electronic documentation to gain information about the data and their sources. We examined data from the Bureau's Cost and Progress, PBOCS, and MaRCS systems to check for logical errors and inconsistencies, and followed up with agency officials knowledgeable about the data in cases where we had questions about potential errors or inconsistencies, and to inquire about the accuracy and completeness of the entry and processing of the data. Values are updated by the Bureau throughout the operations, and may be revised by the Bureau even after the operations close. On the basis of our efforts, we determined that the data were sufficiently reliable for the purposes of this engagement. Finally, to identify preliminary steps the Bureau can take to help transform its management and culture, we reviewed our prior work on governmentwide reexamination, as well as leading practices and attributes in the areas of IT management, organizational performance, collaboration, stewardship, and human capital. In addition, we reviewed census planning material, prior GAO work on census planning and development efforts, and spoke with Bureau officials about their needs and plans for management and cultural transformation. We conducted this performance audit from December 2009 until December 2010 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audits to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Local Census Offices Visited in This Review: Tucson, Arizona; Fresno and San Francisco, California; New Castle, Delaware; Fort Myers, Florida; Atlanta, Georgia; Chicago (2 locations), Illinois; New Orleans, Louisiana; Baltimore and Seat Pleasant, Maryland; Boston, Massachusetts; Detroit, Michigan; Meridian, Mississippi; Cape Girardeau and St. Louis, Missouri; Las Vegas (2 locations), Nevada; Albuquerque, New Mexico; Bronx, Brooklyn, and Manhattan, New York; Asheville and Fayetteville, North Carolina; Philadelphia, Pennsylvania; Dallas and Houston, Texas; Washington, District of Columbia. [End of section] Appendix II: Comments from the Department of Commerce: Note: Page numbers in the draft report may differ from those in this report. United States Department Of Commerce: The Secretary of Commerce: Washington, D.C. 20230: December 7, 2010: Mr. Robert Goldenkoff: Director: Strategic Issues: U.S. Government Accountability Office: Washington, DC 20548: Dear Mr. Goldenkoff: The Department of Commerce appreciates the opportunity to comment on the U.S. Government Accountability Office's draft report entitled "2010 Census: Data Collection Operations Were Generally Completed as Planned, But Longstanding Challenges Suggest Need for Fundamental Reforms" (GAO-11-193). The Department of Commerce's comments on this report are enclosed. Sincerely, Signed by: Gary Locke: Enclosure: [End of letter] Department of Commerce Comments on the United States Government Accountability Office Draft Report Entitled "2010 Census: Data Collection Operations Were Generally Completed as Planned, But Longstanding Challenges Suggest Need for Fundamental Reforms" (GA0-11-193) December 2010. The Department of Commerce (Department) would like to thank the U.S. Government Accountability Office (GAO) for its efforts in examining the 2010 Census Nonresponse Followup (NRFU) and other field operations to seek out improved approaches to securing greater participation from the public and to reduce extensive operating costs in the next census. The Department generally agrees with the overall findings and with the recommendations regarding items suggested for study for conducting the 2020 Census. The Census Bureau does, however, wish to provide a few comments about the statements and conclusions in this report: * Page 13, second paragraph: "With respect to staffing levels, the Bureau set a recruitment goal of nearly 3.7 million qualified applicants and achieved 104 percent of this goal..." Response: As clarification, the Census Bureau notes that our goal was 3.8 million total applicants, in order to yield a sufficient number of qualified applicants. Not all applicants we recruit qualify for census work. In Census 2000, only about 73 percent of the applicants ended up being qualified. For the 2010 Census, this figure was about 77 percent. This comment also applies to a similar statement on page 3 of the report. * Page 13, last paragraph: "According to the Bureau, based on past experience, it set its recruiting goal at five times the number of persons .... develop more accurate recruiting targets. It will also be important for the Bureau to adhere to recruiting goals so that additional costs are not incurred." Response: The Census Bureau agrees that initial plans for 2010 Census recruiting were driven by its Census 2000 experience; however, this section of the report does not acknowledge that, based on the state of the economy in late 2009 and early 2010, the Census Bureau took several steps to modify its recruiting plans prior to the peak recruiting for 2010 field operations, such as NRFU. The Census Bureau certainly agrees that monitoring economic conditions closely to develop and implement a recruiting strategy for the 2020 Census is important. * Page 16, first paragraph: "...if an enumerator was unable to collect data from either the household or a proxy respondent a "closeout interview" was used where household information was later determined using statistical methods based on neighborhood characteristics." and: Page 19, first paragraph: "... a closeout interview is one where no interview is conducted and household information is later determined using statistical methods." Response: The Census Bureau recommends some slight revisions to these sentences. As currently written, these statements might be read to imply that field staff was responsible for determining household characteristics based on the characteristics of neighboring units. Explaining that this imputation step takes place at headquarters during data processing would provide clarity and additional accuracy. Also, a statement that implies that no data are collected during a closeout interview is not correct. Field staff sometimes obtained a population count directly from a resident. * Page 18, last paragraph: "Significantly, our analysis of Bureau data found that the fast pace of NRFU was associated with the collection of less complete household data." Response: The Census Bureau would be interested in discussing these findings in more detail, because its analysis of a similar question did not yield the same finding. * Page 24, lines 16-17: "ln looking forward to 2020, the Bureau could streamline fingerprint taking by extending training sessions to allot more time for the process." Response: The Census Bureau would appreciate additional clarity regarding this recommendation. The Census Bureau is unclear as to ways in which extending training, which usually requires spending more time and money, would streamline fingerprinting efforts. * Page 47, second paragraph: "...the noteworthy achievements of the 2010 Census occurred because of the shared efforts of the Bureau, Congress and thousands of other parties." Response: This sentence should specifically include the Department of Commerce and the Economics and Statistics Administration (ESA). Particularly in 2009 and 2010, ESA played a significant role in helping to make the 2010 Census a success. * Page 48, second paragraph: "...we recommend that the Secretary of Commerce direct the Census Director to take the following six actions:..." Response: The Secretary of Commerce should direct the Under Secretary of the Economics and Statistics Administration (ESA) as well as the Census Director. ESA has management oversight responsibility of the Census Bureau and has been actively engaged in planning for the 2020 Census, including development of effective, efficient, and forward thinking integrated management approaches and systems that will result in successful and cost-effective operations across the bureau's programs and activities. * Page 50, second paragraph: "We are sending copies of this report to the Secretary of Commerce, the Director of the U.S. Census Bureau, and interested congressional committees." Response: Please also send a copy of the report to the Under Secretary for Economic Affairs (ESA). In conclusion, we want to acknowledge the GAO's extensive work in reviewing these activities, and appreciate its ongoing efforts to help us develop a successful evaluation plan for the 2020 Census. [End of section] Appendix III: GAO Contact and Staff Acknowledgments: GAO Contact: Robert Goldenkoff, (202) 512-2757 or goldenkoffr@gao.gov: Staff Acknowledgments: Other key contributors to this report include Lisa Pearson, Assistant Director; Mark Abraham; David Bobruff; Benjamin Crawford; Sara Daleski; Dewi Djunaidy; Vijay D'Souza; Elizabeth Fan; Ronald Fecso; Robert Gebhart; Richard Hung; Kristen Lauber; Andrea Levine; Ty Mitchell; Kathleen Padulchick; Michael Pahr; Tind Ryen; Jonathan Ticehurst; Timothy Wexler; Holly Williams; Monique Williams; and Kate Wulff. [End of section] Related GAO Products: 2010 Census: Key Efforts to Include Hard-to-Count Populations Went Generally as Planned; Improvements Could Make the Efforts More Effective for Next Census. [hyperlink, http://www.gao.gov/products/GAO-11-45]. Washington, D.C.: December 14, 2010. 2010 Census: Follow-up Should Reduce Coverage Errors, but Effects on Demographic Groups Need to Be Determined. [hyperlink, http://www.gao.gov/products/GAO-11-154]. Washington, D.C.: December 14, 2010. 2010 Census: Plans for Census Coverage Measurement Are on Track, but Additional Steps Will Improve Its Usefulness. [hyperlink, http://www.gao.gov/products/GAO-10-324]. Washington, D.C.: April 23, 2010. 2010 Census: Data Collection Is Under Way, but Reliability of Key Information Technology Systems Remains a Risk. [hyperlink, http://www.gao.gov/products/GAO-10-567T]. Washington, D.C.: March 25, 2010. 2010 Census: Key Enumeration Activities Are Moving Forward, but Information Technology Systems Remain a Concern. [hyperlink, http://www.gao.gov/products/GAO-10-430T]. Washington, D.C.: February 23, 2010. 2010 Census: Census Bureau Continues to Make Progress in Mitigating Risks to a Successful Enumeration, but Still Faces Various Challenges. [hyperlink, http://www.gao.gov/products/GAO-10-132T]. Washington, D.C.: October 7, 2009. 2010 Census: Census Bureau Should Take Action to Improve the Credibility and Accuracy of Its Cost Estimate for the Decennial Census. [hyperlink, http://www.gao.gov/products/GAO-08-554]. Washington, D.C.: June 16, 2008. 2010 Census: Census at Critical Juncture for Implementing Risk Reduction Strategies. [hyperlink, http://www.gao.gov/products/GAO-08-659T]. Washington, D.C.: April 9, 2008. Information Technology: Significant Problems of Critical Automation Program Contribute to Risks Facing 2010 Census. [hyperlink, http://www.gao.gov/products/GAO-08-550T]. Washington, D.C.: March 5, 2008. Information Technology: Census Bureau Needs to Improve Its Risk Management of Decennial Systems. [hyperlink, http://www.gao.gov/products/GAO-08-259T]. Washington, D.C.: December 11, 2007. Information Technology Management: Census Bureau Has Implemented Many Key Practices, but Additional Actions Are Needed. [hyperlink, http://www.gao.gov/products/GAO-05-661]. Washington, D.C.: June 16, 2005. 21st Century Challenges: Reexamining the Base of the Federal Government. [hyperlink, http://www.gao.gov/products/GAO-05-325SP]. Washington, D.C.: February 1, 2005. Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity. [hyperlink, http://www.gao.gov/products/GAO-04-394G]. Washington, D.C.: March 1, 2004. Comptroller General's Forum, High-Performing Organizations: Metrics, Means, and Mechanisms for Achieving High Performance in the 21st Century Public Management Environment. [hyperlink, http://www.gao.gov/products/GAO-04-343SP]. Washington, D.C.: February 13, 2004. Human Capital: Key Principles for Effective Strategic Workforce Planning. [hyperlink, http://www.gao.gov/products/GAO-04-39]. Washington, D.C.: December 11, 2003. [End of section] Footnotes: [1] 13 U.S.C. § 141(b). [2] See for example: GAO, Decennial Census: Preliminary 1990 Lessons Learned Indicate Need to Rethink Census Approach, [hyperlink, http://www.gao.gov/products/GAO/T-GGD-90-18] (Washington, D.C.: Aug. 8, 1990); and 2000 Census: Progress Made on Design, but Risks Remain, [hyperlink, http://www.gao.gov/products/GAO/GGD-97-142] (Washington, D.C.: July 14, 1997). [3] GAO, 2010 Census: Key Efforts to Include Hard-to-Count Populations Went Generally as Planned; Improvements Could Make the Efforts More Effective for Next Census, [hyperlink, http://www.gao.gov/products/GAO-11-45] (Washington, D.C.: Dec. 14, 2010); and 2010 Census: Follow-up Should Reduce Coverage Errors, but Effects on Demographic Groups Need to Be Determined, [hyperlink, http://www.gao.gov/products/GAO-11-154] (Washington, D.C.: Dec. 14, 2010). [4] See for example: GAO, Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity, [hyperlink, http://www.gao.gov/products/GAO-04-394G] (Washington, D.C.: March 2004); Human Capital: Key Principles for Effective Strategic Workforce Planning, [hyperlink, http://www.gao.gov/products/GAO-04-39] (Washington, D.C.: Dec. 11, 2003); Comptroller General's Forum, High-Performing Organizations: Metrics, Means, and Mechanisms for Achieving High Performance in the 21st Century Public Management Environment, [hyperlink, http://www.gao.gov/products/GAO-04-343SP] (Washington, D.C.: Feb. 13, 2004); 21st Century Challenges: Reexamining the Base of the Federal Government, [hyperlink, http://www.gao.gov/products/GAO-05-325SP] (Washington, D.C.: February 2005); and Results-Oriented Government: Practices That Can Help Enhance and Sustain Collaboration among Federal Agencies, [hyperlink, http://www.gao.gov/products/GAO-06-15] (Washington, D.C.: Oct. 21, 2005). [5] High-risk areas are areas GAO has called special attention to because of their vulnerability to mismanagement or their broad need for reform. GAO, Information Technology: Significant Problems of Critical Automation Program Contribute to Risks Facing 2010 Census, [hyperlink, http://www.gao.gov/products/GAO-08-550T] (Washington, D.C.: Mar. 5, 2008). [6] [hyperlink, http://www.gao.gov/products/GAO-04-343SP]. [7] In the 2000 Census, the Bureau mailed out both long-and short-form questionnaires. The short-form questionnaire had a higher response rate because it had fewer questions. For the 2010 Census, the Bureau used only a short-form questionnaire. For this report we use the 2000 Census short-form mail response rate when comparing 2000 and 2010 mail- back response rates. [8] GAO, 2010 Census: Census Bureau Should Take Action to Improve the Credibility and Accuracy of Its Cost Estimate for the Decennial Census, [hyperlink, http://www.gao.gov/products/GAO-08-554] (Washington, D.C.: June 16, 2008). [9] In order to determine whether the pace of the 2010 Census NRFU was associated with lower-quality work, we conducted regression analysis using Census data to assess whether indicators of lower-quality work were associated with NRFU completion time among the 494 LCOs after adjusting for other factors associated with the timeliness of completion and workload. [10] The National Crime Prevention and Privacy Compact, enacted in 1998, generally requires that fingerprints be submitted with all requests for criminal history record checks for noncriminal justice purposes; 42 U.S.C. § 14616. For the 2000 Census, the Federal Bureau of Investigation (FBI) did not have the capacity to timely process the fingerprints of the Census's temporary workforce, so employees were subject to only a name background check. [11] A well-defined enterprise architecture provides a clear and comprehensive picture of an entity, whether it is an organization (e.g., a federal department) or a functional or mission area that cuts across more than one organization (e.g., personnel management). This picture consists of snapshots of both the enterprise's current or "As Is" environment and its target or "To Be" environment, as well as a capital-investment road map for transitioning from the current to the target environment. [12] GAO, Information Technology Management: Census Bureau Has Implemented Many Key Practices, but Additional Actions Are Needed, [hyperlink, http://www.gao.gov/products/GAO-05-661] (Washington, D.C.: June 16, 2005). [13] GAO, Census 2010: Census at Critical Juncture for Implementing Risk Reduction Strategies, [hyperlink, http://www.gao.gov/products/GAO-08-659T] (Washington, D.C.: Apr. 9, 2008); Information Technology: Census Bureau Needs to Improve Its Risk Management of Decennial Systems, [hyperlink, http://www.gao.gov/products/GAO-08-259T] (Washington, D.C.: Dec. 11, 2007); and GAO-08-550T. [14] GAO, 2010 Census: Data Collection Is Under Way, but Reliability of Key Information Technology Systems Remains a Risk, [hyperlink, http://www.gao.gov/products/GAO-10-567T] (Washington, D.C.: Mar. 25, 2010); 2010 Census: Key Enumeration Activities Are Moving Forward, but Information Technology Systems Remain a Concern, [hyperlink, http://www.gao.gov/products/GAO-10-430T] (Washington, D.C.: Feb. 23, 2010); and 2010 Census: Census Bureau Continues to Make Progress in Mitigating Risks to a Successful Enumeration, but Still Faces Various Challenges, [hyperlink, http://www.gao.gov/products/GAO-10-132T] (Washington, D.C.: Oct. 7, 2009). [15] See for example: [hyperlink, http://www.gao.gov/products/GAO-05-661]; GAO, Census Bureau: Important Activities for Improving Management of Key 2010 Decennial Acquisitions Remain to be Done, [hyperlink, http://www.gao.gov/products/GAO-06-444T] (Washington, D.C.: Mar. 1, 2006); Information Technology: Census Bureau Needs to Improve Its Risk Management of Decennial Systems, [hyperlink, http://www.gao.gov/products/GAO-08-79] (Washington, D.C.: Oct. 5, 2007); and Information Technology: Census Bureau Testing of 2010 Decennial Systems Can Be Strengthened, [hyperlink, http://www.gao.gov/products/GAO-09-262] (Washington, D.C.: Mar. 5, 2009). [16] [hyperlink, http://www.gao.gov/products/GAO-08-554]. [17] GAO, 2010 Census: Plans for Census Coverage Measurement Are on Track, but Additional Steps Will Improve Its Usefulness, [hyperlink, http://www.gao.gov/products/GAO-10-324] (Washington, D.C.: Apr. 23, 2010). [18] [hyperlink, http://www.gao.gov/products/GAO/GGD-97-142]. [19] See for example: [hyperlink, http://www.gao.gov/products/GAO-04-394G], [hyperlink, http://www.gao.gov/products/GAO-04-39], [hyperlink, http://www.gao.gov/products/GAO-04-343SP, [hyperlink, http://www.gao.gov/products/GAO-05-325SP]], and [hyperlink, http://www.gao.gov/products/GAO-06-15]. [20] GAO, Transition Series: Commerce Issues, [hyperlink, http://www.gao.gov/products/OCG-89-11TR] (Washington, D.C.: Nov. 1, 1988). [21] GAO, 2010 Census: Cost and Design Issues Need to Be Addressed Soon, [hyperlink, http://www.gao.gov/products/GAO-04-37] (Washington, D.C.: Jan. 15, 2004). [22] S. 3167, 111th Cong. § 2 (2010). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each weekday, GAO posts newly released reports, testimony, and correspondence on its Web site. To have GAO e-mail you a list of newly posted products every afternoon, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s Web site, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: E-mail: fraudnet@gao.gov: Automated answering system: (800) 424-5454 or (202) 512-7470: Congressional Relations: Ralph Dawn, Managing Director, dawnr@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, D.C. 20548: Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, D.C. 20548: