This is the accessible text file for GAO report number GAO-15-98 entitled 'Nuclear Regulatory Commission: NRC Needs to Improve Its Cost Estimates by Incorporating More Best Practices' which was released on December 16, 2014. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to Congressional Requesters: December 2014: Nuclear Regulatory Commission: NRC Needs to Improve Its Cost Estimates by Incorporating More Best Practices: GAO-15-98: GAO Highlights: Highlights of GAO-15-98, a report to congressional requesters. Why GAO Did This Study: The March 2011 accident at Japan's Fukushima Daiichi nuclear power plant resulted in a release of radioactive material. NRC considered several new requirements for U.S. plants based on lessons learned from Fukushima, such as the installation of a filtered venting system. Nuclear industry stakeholders raised concerns that NRC's cost estimate for filtered venting systems did not reflect accurate costs of implementation. GAO was asked to review NRC's cost estimating procedures. This report examines (1) the extent to which NRC's cost estimating procedures support the development of reliable cost estimates and (2) the extent to which NRC staff's 2012 cost estimate for proposed filtered venting systems is reliable. GAO compared both NRC's cost estimating procedures and the 2012 cost estimate to four characteristics of reliable cost estimates: comprehensive, well-documented, accurate, and credible. These characteristics relate to specific best practices identified in GAO's Cost Guide. GAO excluded some best practices that were not relevant to NRC's role as an independent regulator. What GAO Found: The Nuclear Regulatory Commission's (NRC) overall cost estimating procedures incorporate some best practice characteristics identified in the GAO Cost Estimating and Assessment Guide (Cost Guide), but not others, and do not adequately support the creation of reliable cost estimates (see table). For example, while the procedures include the best practice of developing an estimating plan by outlining the processes for selecting alternatives and performing a cost-benefit analysis, the procedures do not require the cost estimating methods to be identified. In addition, while the NRC's procedures include guidance for conducting sensitivity analyses, which examine how the cost estimate is affected by a change in a cost driver's value, the procedures do not call for an independent cost estimate to be performed. An independent cost estimate can provide decision makers with an objective and unbiased assessment of whether the estimate can be achieved. NRC staff said they are currently updating their cost estimating procedures as part of a multiyear effort. NRC staff's 2012 cost estimate for proposed filtered venting systems is not reliable because it did not sufficiently follow related best practices to fully or substantially meet any of the four characteristics of a reliable cost estimate. For example, while the estimate included life cycle costs of filtered venting systems, covering a 25-year license term, the estimate did not include documentation explaining step by step how the estimate was developed. In addition, the estimate did not incorporate a risk and uncertainty analysis to assess variability in estimates due to factors such as a lack of knowledge about the future. Without a reliable cost estimate, decision makers do not have adequate cost information to make decisions on filtered venting systems. Table: Extent NRC's Overall Cost Estimating Procedures and 2012 Cost Estimate for Proposed Filtered Venting Systems Met Best Practices: Best practice characteristic: Comprehensive; NRC's overall cost estimating procedures: Partially meets; 2012 cost estimate for proposed filtered venting system: Partially meets. Best practice characteristic: Well-documented; NRC's overall cost estimating procedures: Substantially meets; 2012 cost estimate for proposed filtered venting system: Partially meets. Best practice characteristic: Accurate; NRC's overall cost estimating procedures: Minimally meets; 2012 cost estimate for proposed filtered venting system: Partially meets. Best practice characteristic: Credible; NRC's overall cost estimating procedures: Partially meets; 2012 cost estimate for proposed filtered venting system: Does not meet. Source: GAO analysis of NRC data. GAO-15-98. Note: A characteristic is not met when none of the underlying best practices are satisfied; minimally met when a small portion of the underlying best practices are satisfied; partially met when about half of the underlying best practices are satisfied; substantially met when a large portion of the underlying best practices are satisfied; and fully met when the underlying best practices are completely satisfied. [End of table] What GAO Recommends: GAO recommends that NRC align its cost estimating procedures with relevant cost estimating best practices identified in the GAO Cost Guide and ensure that future cost estimates are prepared in accordance with relevant cost estimating best practices. NRC generally agreed with GAO's recommendation. View GAO-15-98. For more information, contact Frank Rusco at 202-512- 3841 or ruscof@gao.gov. [End of section] Contents: Letter: Background: NRC's Overall Cost Estimating Procedures Do Not Support the Creation of Reliable Estimates: NRC Staff's Specific 2012 Cost Estimate for Proposed Filtered Venting Systems Is Not Reliable: Conclusions: Recommendation for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope and Methodology: Appendix II: Boiling Water Reactor with Proposed Filtered Venting System (Corresponds to Fig. 1): Appendix III: Assessment of NRC's Overall Cost Estimating Procedures: Appendix IV: Assessment of NRC Staff's 2012 Cost Estimate for Proposed Filtered Venting Systems: Appendix V: Comments from the Nuclear Regulatory Commission: Appendix VI: GAO Contact and Staff Acknowledgments: Tables: Table 1: Extent to Which NRC's Overall Cost Estimating Procedures Met Best Practices: Table 2: Extent to Which NRC Staff's 2012 Cost Estimate for Proposed Filtered Venting Systems Met Best Practices: Table 3: Description of Diagram Components: Table 4: GAO's Assessment of NRC's Overall Cost Estimating Procedures Against Best Practices: Table 5: GAO's Assessment of NRC Staff's 2012 Cost Estimate for Proposed Filtered Venting Systems Against Best Practices: Figure: Figure 1: A Simplified Cross Section Diagram of a Boiling Water Reactor (BWR) with Proposed Filtered Venting System: Abbreviations: BWR: boiling water reactor: Cost Guide: GAO Cost Estimating and Assessment Guide: NRC: Nuclear Regulatory Commission: OMB: Office of Management and Budget: WBS: work breakdown structure: [End of section] United States Government Accountability Office: GAO: 441 G St. N.W. Washington, DC 20548: December 12, 2014: The Honorable David Vitter: Ranking Member: Committee on Environment and Public Works: United States Senate: The Honorable Fred Upton: Chairman: Committee on Energy and Commerce: House of Representatives: The March 2011 accident at Japan's Fukushima Daiichi nuclear power plant destroyed three nuclear reactors and resulted in the most extensive release of radioactive material at a nuclear power plant since the 1986 Chernobyl disaster. The tsunami that flooded the Fukushima Daiichi plant destroyed its power supplies and emergency equipment, which, in turn, prevented operators from cooling the reactors. Heat and pressure built up in the reactors, eventually damaging three reactor containment structures and releasing radioactive material into the environment. The accident led to the evacuation of over one hundred thousand residents from the area around the plant, is expected to cost Japan tens of billions of dollars, and has led to a review of civilian nuclear power programs worldwide. In response to the accident, the U.S. Nuclear Regulatory Commission (NRC)--the nuclear regulatory body of the United States--formed the Near-Term Task Force to conduct a comprehensive assessment of NRC's processes and regulations to determine whether the agency should make changes to its regulatory system to better ensure reactor safety in the United States. The task force issued a report in 2011 that made several recommendations, including one focused on boiling water reactors (BWR), the same type of reactor involved in the Fukushima Daiichi accident. Specifically, the task force recommended that NRC require operators of BWRs with smaller containment structures, known as Mark I and Mark II designs, to install hardened vents.[Footnote 1] Under severe accident conditions, such as those at Fukushima, hardened vents could be opened to release heat and pressure building in a reactor's containment structure to prevent reactor core damage and radioactive leaks. Opening the vents, however, would potentially release radioactive material into the atmosphere. One means of limiting the release of radioactive material in such instances would be to install filtered venting systems to capture radioactive materials released through the hardened vents. In considering whether to require licensees of BWRs with Mark I and Mark II designs to upgrade hardened vents, NRC staff also analyzed the costs and benefits of installing filtered venting systems to the hardened vents, among other options. In its 2012 regulatory analysis,[Footnote 2] NRC staff presented options for the Commissioners[Footnote 3] to consider and concluded that filtered venting systems were a cost-justified, substantial safety enhancement that should be imposed on operators of BWRs with Mark I and II designs.[Footnote 4] Among the quantitative costs considered was an implementation cost estimate for the filtered venting systems of $15 to $20 million per reactor unit.[Footnote 5] While their analysis of filtered venting systems showed that the quantitative benefits did not exceed quantitative costs, when the staff factored in various qualitative factors, such as improving defense in depth,[Footnote 6] they concluded that the benefits overall exceeded the costs and were therefore cost-justified. NRC staff then recommended that the Commissioners require operators of BWRs with Mark I and Mark II designs to install filtered venting systems. However, after reviewing the staff's analysis and presentation of options, the Commissioners did not order operators to install filtered venting systems as the staff recommended and instead directed the staff to pursue filtering through the formal rulemaking process. In addition, industry stakeholders expressed concerns that implementation costs could be two to three times higher than NRC's estimate. NRC acknowledged that given uncertainties in its cost estimate, the actual cost of the filtered venting systems could be higher than its estimate. This uncertainty raised questions about NRC's cost estimating procedures.[Footnote 7] NRC's cost estimating procedures are important because the agency performs cost estimating, as part of its regulatory analyses, to ensure that it makes sound decisions on requirements related to safety improvements at nuclear power plants. In March 2009, we issued the GAO Cost Estimating and Assessment Guide (Cost Guide),[Footnote 8] a compilation of cost estimating best practices drawn from across industry and government. Specifically, the Cost Guide identifies the four characteristics of a reliable cost estimate as: comprehensive, well-documented, accurate, and credible. In addition, the guide identifies 12 best practices that, when performed correctly, should produce reliable cost estimates. You asked us to review NRC's cost estimating procedures. This report examines (1) the extent to which NRC's cost estimating procedures support the development of reliable cost estimates and (2) the extent to which NRC staff's 2012 cost estimate for proposed filtered venting systems is reliable. For both objectives, we analyzed NRC's cost estimating procedures and cost estimate for proposed filtered venting systems against the best practices outlined in the Cost Guide. The Cost Guide contains best practices that can be used to assess both cost estimating procedures and cost estimates themselves. For cost estimating procedures or a specific cost estimate to be considered reliable, they would need to substantially or fully meet each of the four characteristics. The extent to which the characteristics are met is determined by the extent to which the underlying best practices for each characteristic are incorporated. GAO developed the Cost Guide to assist government agencies as they develop, manage, and evaluate the costs of capital projects. Although NRC does not directly implement or oversee implementation of capital projects at reactors, such as the potential addition of filtered venting systems, the agency develops cost estimates as part of its decision-making process and needs reliable cost estimates to inform the Commissioners' decisions. Most of the best practices for producing reliable cost estimates identified in the Cost Guide apply to NRC's situation as a regulatory decision maker, but we determined that certain best practices do not apply to NRC's situation. For example, while one cost estimating best practice is to update the estimate to reflect actual costs and changes, we neither assessed NRC's procedures nor its specific estimate for filtered venting systems against this best practice because we determined that it did not apply to NRC's situation as regulatory decision maker. Additionally, we did not compare NRC staff's 2012 cost estimate to actual costs of installing filtered venting systems in U.S. plants because no BWRs in the U.S. have installed such systems to date. For both objectives we reviewed relevant laws, regulations, as well as documentation pertaining to the cost of filtered venting systems from NRC and stakeholders, such as industry groups. We interviewed NRC staff and stakeholders from industry, public safety, and environmental groups. We also conducted a site visit to a boiling water reactor that could be required to install a filtered venting system in order to understand implementation issues and how these might affect costs. We provided our draft analyses to NRC for their early review and as an opportunity to provide additional documentation. Appendix I contains more detailed information on our objectives, scope, and methodology. We conducted this performance audit from December 2013 to December 2014, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: In its role as the independent nuclear regulatory agency of the United States, NRC is charged with protecting public health and safety, promoting common defense and security, and protecting the environment. When the Commissioners believe that a modification to a nuclear power plant is necessary to ensure that the facility provides adequate protection to public health and safety or is in accord with the common defense and security, it can order licensed operators to modify or make additions to a reactor or aspects of a plant's facilities without consideration of economic costs. Alternatively, if NRC determines that there is a substantial increase in the overall protection of the public health and safety or the common defense and security to be derived from a modification and that the costs of implementation for that facility are justified in view of this increased protection, NRC may order licensed operators to modify a reactor or aspect of the facility, among other actions. Specifically, in making this determination, NRC conducts a regulatory analysis focusing on an estimation and evaluation of costs and benefits.[Footnote 9] NRC conducts its regulatory analyses according to its internal procedures outlined in Regulatory Analysis Guidelines of the U.S. Nuclear Regulatory Commission and Regulatory Analysis Technical Evaluation Handbook.[Footnote 10] These documents establish the agency's procedures for preparing cost estimates to be used in its regulatory analyses, according to NRC staff.[Footnote 11] In 2012, NRC staff conducted regulatory analysis comparing the costs and benefits of requiring upgraded hardened vents and upgraded hardened vents with filtered venting systems for BWRs with Mark I and Mark II designs.[Footnote 12] Public safety advocates had raised the issue of requiring the installation of filtered venting systems that would reduce radioactive releases in the event of a severe accident. Filtered venting systems are an established technology that have been installed in many European plants that could allow operators to vent pressure from a containment structure during a severe accident while reducing releases of radioactive materials.[Footnote 13] See figure 1, an interactive graphic, for a simplified diagram of a BWR with a Mark I design that includes the proposed filtered venting system. (See appendix II for a printable version.) Figure 1: A Simplified Cross Section Diagram of a Boiling Water Reactor (BWR) with Proposed Filtered Venting System: [Refer to PDF for image: interactive graphic] Instructions: Online, roll your mouse over each number in the figure for more information. For printable version, see appendix II, page 26. Note: The diagram above is a generic representation of how a filtered venting system relates to existing boiling water reactors (BWR) with Mark I containment designs. The principles are similar for a Mark II containment design, but the containment structures of Mark IIs have different specific design features, such as having the wetwell directly below the drywell floor rather than in a free-standing, torus- shaped vessel outside the footprint of the drywell. Source: GAO. GAO-15-98. [End of figure] Based on their regulatory analysis, NRC staff recommended that the Commissioners order the installation of upgraded hardened vents with filtered venting systems. While the Commissioners issued an order modifying licenses held by operators of BWRs with Mark I and Mark II containment designs to require the installation of upgraded reliable hardened vents as a cost-justified safety enhancement in 2013, they did not require operators to implement filtered venting systems. Instead, the Commissioners directed NRC staff to pursue filtering through the rulemaking process. According to NRC, the rulemaking process is ongoing and has involved numerous public meetings. NRC staff are expected to issue a draft regulatory basis--the first step in NRC's process to develop a new rule--in December 2014.[Footnote 14] According to NRC staff, this draft will include another regulatory analysis considering the costs and benefits of filtered venting systems compared with other filtering strategies. Such filtering strategies could involve making use of existing equipment in the plant, such as the wetwell, to limit the release of radioactive materials. GAO developed the Cost Guide to assist federal agencies in developing reliable cost estimates and also as a tool for evaluating existing cost estimating procedures. To develop the Cost Guide, GAO cost experts assessed measures applied by cost estimating organizations throughout the federal government and industry and considered best practices for the development of reliable cost estimates. While the Cost Guide has a focus on developing cost estimates in the context of government acquisition programs, it outlines best practices that are generally applicable to cost estimation in a variety of circumstances. These best practices can be used to assess both an agency's cost estimating procedures and the specific cost estimates an agency develops to determine whether they meet the four characteristics-- comprehensive, well-documented, accurate, and credible--for being reliable. For example, an agency's cost estimating procedures can be assessed over whether they provide guidance on how to conduct a sensitivity analysis and a specific cost estimate developed by an agency could be reviewed to see if they performed a sensitivity analysis in accordance with best practices.[Footnote 15] Following are some of the criteria that the cost estimating guide cites as central to meeting these characteristics. An estimate is: * comprehensive when it accounts for all possible costs associated with a project, is structured in sufficient detail to ensure that costs are neither omitted nor double-counted, and the estimating teams' composition is commensurate with the assignment; * well-documented when supporting documentation is accompanied by a narrative explaining the process, sources, and methods used to create the estimate and contains the underlying data used to develop the estimate; * accurate when it is not overly conservative or too optimistic and based on an assessment of the costs most likely to be incurred; and: * credible when it has been cross-checked with independent cost estimates, the level of confidence associated with the point estimate has been identified, and a sensitivity analysis has been conducted-- that is, the project has examined the effect of changing one assumption related to each project activity while holding all other variables constant in order to identify which variable most affects the cost estimate. The Cost Guide identifies best practices for cost estimating procedures and underlying tasks associated with each of the four characteristics of reliable cost estimates. For example, developing an estimating plan and determining an estimating structure are two best practices associated with the comprehensive characteristic, and there are several underlying tasks associated with both best practices. Each best practice is related to one of the four characteristics of a reliable cost estimate. See appendix III for more information on the best practices that can be used to assess an agency's cost estimating procedures. See appendix IV for more information on the best practices that can be used to assess specific cost estimates. NRC's Overall Cost Estimating Procedures Do Not Support the Creation of Reliable Estimates: NRC's overall cost estimating procedures incorporate some best practices, but not others, and do not adequately support the creation of reliable cost estimates. For cost estimating procedures to support the creation of reliable cost estimates, they must substantially or fully meet each of the four characteristics in GAO's Cost Guide-- comprehensive, well-documented, accurate, and credible--based on the extent to which the procedures incorporate the underlying best practices for each characteristic. While NRC's overall procedures incorporate some best practices for each of the four characteristics, we found that NRC's cost estimating procedures satisfied only one characteristic. Specifically, we found that NRC's cost estimating procedures substantially met the well-documented characteristic, partially met the comprehensive and credible characteristics, and minimally met the accurate characteristic. Our conclusion that NRC's cost estimating procedures do not meet the characteristics for producing reliable cost estimates is based on the following observations: * Comprehensive. NRC's cost estimating procedures partially met the comprehensive characteristic because the procedures incorporate some elements of the two associated best practices. The procedures include the best practice of developing an estimating plan. For example, NRC's procedures outline the process for selecting alternatives and performing a cost-benefit analysis. NRC's procedures also include estimated durations of each step in the process. However, we found that NRC's cost estimating procedures do not include the best practice of providing guidance for deconstructing a program's end product into successive levels with smaller specific elements until the work is subdivided to a level suitable for management control; this is known as a product-oriented work breakdown structure.[Footnote 16] A product- oriented work breakdown structure also defines in detail the work necessary to accomplish a program's objectives and identifies project costs at a detailed level to ensure that all relevant costs are included. In addition, having a standardized work breakdown structure is considered a best practice because, among other things, it allows data to be shared across organizations. Not having a standardized work breakdown structure can make it very difficult to compare elements of different cost estimates. Industry stakeholders told us that it would be useful if NRC used standardized cost categories that would allow industry and NRC to identify what is driving possible differences between the cost estimates developed by each. * Well-documented. NRC's cost estimating procedures substantially met the well-documented characteristic because the procedures incorporate a large number of related best practices. NRC's procedures fully include tasks related to the underlying best practices of defining the purpose of the estimate, as well as identifying the ground rules and assumptions. In addition, NRC's procedures include the best practice of documenting the estimate by requiring that staff provide documentation that demonstrates that the analysis conducted is based on the best scientific, technical, and economic information that is reasonably attainable, relies on peer-reviewed literature, and identifies sources for all original information. However, NRC's procedures do not include tasks to ensure the consideration of historical data or cost drivers, which are factors that can influence a program's cost and are essential for ensuring that the right data are captured. We also found that, while NRC's procedures provide guidance regarding conducting and documenting the regulatory analysis, they do not require documenting all parameters, descriptions, or calculations used to develop the cost estimate. According to the GAO Cost Guide, rigorous documentation that identifies the primary methods, calculations, results, rationales, assumptions, and sources of the data used to generate each cost element, increases an estimate's credibility and helps support an organization's decision making. * Accurate. NRC's cost estimating procedures minimally met the accurate characteristic because they only minimally incorporate tasks associated with the one relevant underlying best practice.[Footnote 17] Specifically, NRC's procedures discuss the development of estimates, but they do not discuss estimate methodologies. Best practices identified in the GAO Cost Guide state that the estimating methodology and rationale used to derive each work breakdown structure element's cost should be described in detail. However, NRC's procedures do not call for estimating methods to be identified. * Credible. NRC's cost estimating procedures partially met the credible characteristic because the procedures incorporate about half of the associated best practices. For example, we found that NRC's procedures include guidance for conducting sensitivity analyses, which examine how the cost estimate is affected by a change in a cost driver's value. We also found that NRC's procedures include some guidance for conducting a risk and uncertainty analysis, which best practices indicate is necessary to quantify the imperfectly understood risks in the estimate and to identify the effects of changing key cost driver assumptions and factors. However, NRC's procedures do not require uncertainty analyses to always be performed, whereas best practices identified in the GAO Cost Guide always call for uncertainty analyses. In addition, we found that NRC's procedures do not call for an independent cost estimate to be performed by an organization that is outside and at a higher-level than the organization that developed the baseline cost estimate. Including an independent cost estimate is a best practice because it can provide decision makers with an objective and unbiased assessment of whether the estimate can be achieved. Because NRC's cost estimating procedures do not fully incorporate relevant best practices for developing comprehensive, well-documented, accurate, and credible estimates, the cost estimates it produces may not be reliable. In a May 2014 public meeting, NRC staff reported that they are in a two-phase process of updating their cost estimating procedures. The first phase will focus on administrative changes, such as the consolidation of guidance documents, and should be issued as a draft for public comment in the summer of 2015. The second phase will focus on addressing potential changes in policy and methodology and will be a multiyear effort. In addition, NRC staff said they recognize the need to improve the accuracy of the agency's quantitative estimates and indicated that improving their guidance and practices related to quantifying the costs and benefits of proposed regulatory actions is an overarching priority for the staff.[Footnote 18] Table 1 summarizes our assessment of the extent to which NRC's cost estimating procedures met the four characteristics in GAO's Cost Guide and incorporate the best practices underlying each characteristic. Appendix III summarizes our detailed assessment of NRC's procedures. Table 1: Extent to Which NRC's Overall Cost Estimating Procedures Met Best Practices: Characteristic: Comprehensive; Overall assessment: Partially meets; Best practices for cost estimating procedures: Develop the estimating plan; Detailed assessment by best practice[A]: Substantially met. Best practices for cost estimating procedures: Determine the estimating structure; Detailed assessment by best practice[A]: Minimally met. Characteristic: Well-documented; Overall assessment: Substantially meets; Best practices for cost estimating procedures: Define estimate's purpose; Detailed assessment by best practice[A]: Fully met. Best practices for cost estimating procedures: Define the program's characteristics; Detailed assessment by best practice[A]: Partially met. Best practices for cost estimating procedures: Identify ground rules and assumptions; Detailed assessment by best practice[A]: Fully met. Best practices for cost estimating procedures: Obtain the data; Detailed assessment by best practice[A]: Substantially met. Best practices for cost estimating procedures: Document the estimate; Detailed assessment by best practice[A]: Partially met. Best practices for cost estimating procedures: Present estimate to management for approval; Detailed assessment by best practice[A]: Substantially met. Characteristic: Accurate; Overall assessment: Minimally meets; Best practices for cost estimating procedures: Develop the point estimate; Detailed assessment by best practice[A]: Minimally met. Best practices for cost estimating procedures: Update the estimate to reflect actual costs/changes; Detailed assessment by best practice[A]: Not applicable[B]. Characteristic: Credible; Overall assessment: Partially meets; Best practices for cost estimating procedures: Compare the point estimate to an independent cost estimate; Detailed assessment by best practice[A]: Minimally met. Best practices for cost estimating procedures: Conduct a sensitivity analysis; Detailed assessment by best practice[A]: Fully met. Best practices for cost estimating procedures: Conduct a risk and uncertainty analysis; Detailed assessment by best practice[A]: Partially met. Source: GAO analysis of NRC data. GAO-15-98. [A] The ratings we used in this analysis are as follows: "Not met" means we found no evidence that satisfies the best practice. "Minimally met" means we found evidence that satisfies a small portion of the best practice. "Partially met" means we found evidence that satisfies about half of the best practice. "Substantially met" means we found evidence that satisfies a large portion of the best practice. "Fully met" means we found complete evidence that satisfies the entire best practice. [B] As explained previously, we did not assess NRC's procedures against the best practice of updating a cost estimate to reflect actual costs and changes because we determined that this practice is not applicable to cost estimates used as part of a regulatory analysis to support a specific regulatory decision. For additional information see appendix I. [End of table] NRC Staff's Specific 2012 Cost Estimate for Proposed Filtered Venting Systems Is Not Reliable: The 2012 cost estimate for proposed filtered venting systems is not reliable because it did not fully or substantially meet any of the four characteristics of a reliable cost estimate. When we assessed the 2012 cost estimate for proposed filtered venting systems against the four characteristics derived from GAO's Cost Guide, we found that the cost estimate partially met the comprehensive, well-documented, and accurate characteristics because the estimate satisfied about half of the best practices associated with these three characteristics. However, the estimate was developed with minimal or no use of the best practices underlying the credible characteristic, and so the cost estimate did not meet this characteristic. Our conclusion that the cost estimate for proposed filtered venting systems is not reliable is based on the following observations: * Comprehensive. The cost estimate partially met the comprehensive characteristic because it was developed following some of the underlying best practices, but it did not follow or partially followed others. For example, the estimate included life cycle costs of filtered venting systems, such as industry implementation and operational costs covering a 25-year license term. However, the estimate developed was not based on a product-oriented work breakdown structure, a cost estimating best practice that involves identifying project costs at a detailed level to ensure that all relevant costs are included. Without a product-oriented work breakdown structure, NRC cannot be certain it captured all of the relevant costs necessary to produce a comprehensive estimate. In addition, the estimate outlined the proposed technical requirements for filtered venting systems as part of its cost estimate, according to NRC staff, but it did not consider the risks associated with those technical requirements. NRC staff disagreed with our application of the best practice requiring a work breakdown structure to assess their 2012 cost estimate. The staff explained that not all cost estimating techniques require the development of work breakdown structures and detailed estimates are not usually possible when there are the characteristics of poor program definition, changing requirements, and scarce cost data, characteristics which they said are reflective of the context in which they are creating cost estimates. NRC staff also noted that they are dependent upon the availability of information, including cost estimates, submitted to them by NRC licensees and applicants, who might withhold information that is proprietary. However, we consider a work breakdown structure to be in accordance with best practices for all cost estimates, and a characteristic of the work breakdown structure is to clearly outline the end product and major work that comprises the program being estimated. In addition, a work breakdown structure can be set up to reflect the amount of data available to estimate the program costs by expanding or collapsing the various levels within the hierarchical structure of a work breakdown structure. Cost estimates that do not adequately address project risks do not provide decision makers with sufficient information about the risks' potential effect on costs and the likelihood of project success. * Well-documented. The estimate partially met the well-documented characteristic because NRC staff performed the underlying work for the estimate but did not provide documentation of the work with the estimate. NRC staff explained to us the specific steps used to adjust the cost data to develop the estimate,[Footnote 19] but this information was not included in the documentation accompanying the cost estimate itself. Insufficient documentation can hinder understanding of the estimate. The estimate also did not include documentation explaining step by step how the estimate was developed, such as what calculations NRC staff performed. These steps would be important for any cost analyst unfamiliar with the program to be able to understand what was done and replicate it. Moreover, a well- documented cost estimate provides the estimate additional credibility by explaining the rationale for the estimate's methodology. NRC staff provided additional explanation to support the estimate, but they did not provide complete documentation supporting their cost estimate for our review. They did provide foreign plant costs for filtered venting systems and the inflation factors used, which allowed us to verify that the estimate was appropriately adjusted for inflation. As we previously noted, NRC's overall cost estimating procedures substantially met the well-documented characteristic because the procedures incorporated a large number of best practices. However, for the specific filtered venting system estimate, NRC staff did not provide documentation called for by best practices, even though NRC staff conducted the underlying work. * Accurate. The estimate partially met the accurate characteristic because it was developed following some of the underlying best practices[Footnote 20] but only partially followed or did not follow others.[Footnote 21] For example, NRC staff adjusted their cost data for inflation, and the estimate appeared to be free of substantive errors. However, the estimate was not developed by making fully appropriate use of cost estimating techniques. NRC staff stated they used two techniques: an analogy estimate based on actual implementation and operation costs of filtered venting systems at foreign plants and an informal engineering build-up estimate[Footnote 22] to cross-check the analogy estimate. However, the engineering build-up estimate did not include a work breakdown structure so there was no assurance that the estimate captured all of the relevant costs accurately. Moreover, the estimate was developed without an uncertainty analysis to determine risk levels. Without a risk and uncertainty analysis, decision makers will not be aware of risks that could affect costs, making it impossible to determine if the estimate is overly conservative or optimistic. As we previously stated, NRC's overall cost estimating procedures minimally met the accuracy characteristic because they do not call for estimating methods to be identified. However, for the 2012 cost estimate on proposed filtered venting systems, NRC staff identified and implemented specific methods. Consequently, the estimate received a higher rating for the accurate characteristic than NRC's procedures. * Credible. The cost estimate did not meet the credible characteristic because the estimate was developed making minimal or no use of related best practices. Specifically, the estimate did not include an independent cost estimate to provide an unbiased test of whether the original estimate was reasonable, a formal sensitivity analysis to examine the effects of changing assumptions and ground rules, or a risk and uncertainty analysis to assess variability in point estimates due to factors such as a lack of knowledge about the future or errors resulting from historical data inconsistencies. In regard to the development of an independent cost estimate, NRC staff told us that their cost estimates undergo quality assurance reviews by other analysts, although these reviews are not necessarily done by an independent group. According to the Cost Guide, an independent cost estimate is considered one of the best and most reliable validation methods as it provides an independent view of expected program costs that test the program office's estimate for reasonableness. Therefore, an independent cost estimate can provide decision makers with additional insight into a program's potential costs--in part, because it frequently uses different methods and is less burdened with organizational bias. In addition, the estimate was developed making minimal use of crosschecks of major costs through the use of an engineering build-up estimate. NRC staff stated they conducted the engineering build-up as a cross-check of the top-level estimate, but described it as a "back of the envelope" estimate with no formal documentation that could be reviewed. As we previously noted, NRC's overall cost estimating procedures partially met the credible characteristic because the procedures incorporate about half of the associated best practices. For example, NRC's procedures do not require uncertainty analyses to always be performed, whereas best practices identified in the GAO Cost Guide always call for uncertainty analyses. For this cost estimate, NRC staff chose not to perform risk or sensitivity analyses called for by best practices. By not fully incorporating best practices underlying the characteristics of a reliable cost estimate, NRC does not have assurance that its proposed filtered venting systems cost estimate is reliable. Without a reliable cost estimate, decision makers do not have adequate information on which to make decisions on filtered venting systems. Although NRC staff said they generally agreed with our evaluation of areas where they could improve their cost estimates, they did not think that the Cost Guide's best practices were appropriate criteria for evaluating regulatory analysis cost estimates. NRC staff explained that NRC, as an independent regulatory agency imposing new regulatory requirements on its licensees, develops cost estimates, such as the one for filtered venting systems, to enhance the soundness of decision making by NRC managers and the Commissioners. They also did not think the Cost Guide applied to their cost estimate for filtered venting systems because the filtered venting systems were not a capital investment that NRC itself would oversee or pay for but rather one that licensees would acquire and manage. We concluded that the Cost Guide best practices did apply to NRC staff's estimate for filtered venting systems as it was an estimate for a capital investment, regardless of who would acquire and manage it, and therefore certain best practices in the Cost Guide are applicable. We also concluded that the majority of the Cost Guide's best practices were applicable to an independent regulatory agency such as NRC and excluded from our analysis those best practices we determined were not applicable. Table 2 summarizes our assessment of the extent to which NRC staff's cost estimate for proposed filtered venting systems met the characteristics for being a reliable estimate. Additional information about our assessment of the cost estimate for proposed filtered venting systems is provided in appendix IV. Table 2: Extent to Which NRC Staff's 2012 Cost Estimate for Proposed Filtered Venting Systems Met Best Practices: Characteristic: Comprehensive; Overall assessment: Partially met; Best practices for developing cost estimates: Estimate includes all life cycle costs; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Substantially met. Best practices for developing cost estimates: Estimate completely defines program, reflects current schedule, and is technically reasonable; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Partially met. Best practices for developing cost estimates: Estimate work breakdown structure is product-oriented, traceable to the statement of work/ objective, and at an appropriate level of detail to ensure that cost elements are neither omitted nor double-counted; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Minimally met. Best practices for developing cost estimates: Estimate documents all cost-influencing ground rules and assumptions; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Partially met. Characteristic: Well-Documented; Overall assessment: Partially met; Best practices for developing cost estimates: Documentation captures the source data used, reliability of the data, and how the data were normalized; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Partially met. Best practices for developing cost estimates: Documentation describes in sufficient detail the calculations performed and the estimating methodology used to derive each element's cost; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Partially met. Best practices for developing cost estimates: Documentation describes step-by-step how the estimate was developed so that an analyst unfamiliar with the program could understand what was done and replicate it; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Minimally met. Best practices for developing cost estimates:Documentation discusses the technical baseline description and data in the baseline are consistent with the estimate; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Minimally met. Best practices for developing cost estimates: Documentation demonstrates that the estimate was reviewed and accepted by management; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Partially met. Characteristic: Accurate[B]; Overall assessment: Partially met; Best practices for developing cost estimates: Estimate results are not overly conservative or optimistic and are based on an assessment of most likely costs; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Not met. Best practices for developing cost estimates: Estimate has been properly adjusted for inflation; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Substantially met. Best practices for developing cost estimates: Estimate contains few, if any, minor mistakes; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Substantially met. Best practices for developing cost estimates: Estimate is regularly updated to reflect significant changes in the program so as to always reflect current status; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Not applicable[C]. Best practices for developing cost estimates: Variances between planned and actual costs are documented, explained, and reviewed; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Not applicable[C]. Best practices for developing cost estimates: Estimate is based on a historical record of cost estimating and actual experiences from other comparable programs; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Partially met. Best practices for developing cost estimates: Estimating technique for each cost element was used appropriately; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Partially met. Characteristic: Credible; Overall assessment: Not met; Best practices for developing cost estimates: Estimate includes a sensitivity analysis that identifies a range of possible costs based on varying major assumptions, parameters, and data inputs; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Not met. Best practices for developing cost estimates: Estimate includes a risk and uncertainty analysis that quantifies imperfectly understood risks and the effects of changing key cost driver assumptions and factors; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Not met. Best practices for developing cost estimates: Major cost elements were cross checked to see whether results were similar; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Minimally met. Best practices for developing cost estimates: An independent cost estimate was conducted by a group outside the acquiring organization to determine whether other estimating methods produce similar results; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Not met. Source: GAO analysis of NRC staff's filtered venting system cost estimate. GAO-15-98. [A] The ratings we used in this analysis are as follows: "Not met" means we found no evidence that satisfies the best practice. "Minimally met" means we found evidence that satisfies a small portion of the best practice. "Partially met" means we found evidence that satisfies about half of the best practice. "Substantially met" means we found evidence that satisfies a large portion of the best practice. "Fully met" means we found complete evidence that satisfies the entire best practice. [B] According to the Cost Guide, cost estimates are accurate when they are not overly conservative or too optimistic, based on an assessment of most likely costs, adjusted properly for inflation, and contain few, if any, minor mistakes. [C] As explained previously, we did not assess NRC's procedures against best practices related to updating a cost estimate. For additional information see appendix I. [End of table] Conclusions: NRC performs cost estimating, as part of its regulatory analyses, to ensure that the agency makes sound decisions regarding actions needed to protect the health and safety of the public or the common defense and security. NRC states that these analyses ensure that the agency bases its decisions on adequate information, and that the staff arrives at its decisions by following a systematic and disciplined process that is also open and transparent. However, we found that NRC's procedures for developing the cost estimates do not fully incorporate relevant cost estimating best practices. Without procedures that adhere to cost estimating best practices, NRC does not have assurance that it is producing reliable cost estimates. Thus, the Commissioners may not have adequate information on which to base their decisions. To its credit, NRC has acknowledged the need to revise and consolidate its cost estimating procedures. However, unless NRC revises its procedures by incorporating relevant best practices for developing reliable cost estimates, the process may continue to yield unreliable cost estimates that could negatively affect a decision- making process that is intended to ensure public safety. Recommendation for Executive Action: To improve the reliability of its cost estimates, we recommend that, as NRC revises its cost estimating procedures, the NRC Chairman ensures that the agency aligns the procedures with relevant cost estimating best practices identified in the GAO Cost Estimating and Assessment Guide and ensure that future cost estimates are prepared in accordance with relevant cost estimating best practices. Agency Comments and Our Evaluation: We provided a draft of this report to the Chairman of the Nuclear Regulatory Commission for review and comment. In written comments, which are reproduced in appendix V, NRC generally agreed with our recommendation that NRC could improve its regulatory analyses and said they have efforts underway to do so. However, NRC said that they do not believe GAO's Cost Guide is appropriate criteria for assessing their program. NRC stated that they use OMB Circular A-4 as a reference for their guidance on regulatory analysis, which they believe is more appropriate for assessing cost estimates. NRC explained that, in their view, the Cost Guide is more appropriate for acquisitions of major systems, which they said NRC rarely does. NRC noted that the agency regulates and licenses private entities that construct and operate their own nuclear facilities. However, in our view, the Cost Guide is the most appropriate criteria to assess the reliability of cost estimates. OMB Circular A-4 discusses cost estimating at a very high level and does not go into great detail, whereas the Cost Guide provides much greater detail in terms of assessing the reliability of cost estimates. In addition, the Cost Guide includes best practices, from the private and public sectors, in cost estimating for capital assets. A filtered venting system consists of equipment and structures and is a type of capital asset. The fact that the filtered venting systems will be acquired by licensees, and not NRC, does not change the importance of producing reliable cost estimates. Furthermore, as we noted in the report, we did not assess NRC's estimate against best practices that we determined did not apply to an estimate performed for regulatory purposes. Based on all of these factors, we continue to assert that the Cost Guide is both sufficient and reasonable criteria for assessing NRC's cost estimating procedures and specific cost estimates. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Chairman of the NRC, and other interested parties. In addition, the report will be available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you or your staff members have any questions regarding this report, please contact me at (202) 512-3841 or ruscof@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VI. Signed by: Frank Rusco Director, Natural Resources and Environment: [End of section] Appendix I: Objectives, Scope and Methodology: Our review assessed: (1) the extent to which the Nuclear Regulatory Commission's (NRC) overall cost estimating procedures support the development of reliable cost estimates and (2) the extent to which NRC staff's 2012 cost estimate for proposed filtered venting systems is reliable. To address these objectives, we reviewed relevant laws, regulations, NRC guidance documents, as well as reports on the Fukushima Daiichi accident and filtered venting systems. We interviewed NRC staff and stakeholders from two nuclear power industry groups, one environmental group, and one public safety group. We also conducted a site visit to a boiling water reactor in Pennsylvania that may be required to install a filtered venting system to understand implementation issues and how these affect costs. For both objectives, we relied on the best practices contained in the Cost Guide[Footnote 23] as our criteria for assessment. GAO developed the Cost Guide to assist government agencies as they develop, manage, and evaluate the costs of their capital projects. Although NRC does not directly implement or oversee implementation of capital projects at reactors, such as the potential addition of filtered venting systems, the agency develops cost estimates as part of its decision- making process and needs reliable cost estimates to inform the Commissioners' decisions. Most of the best practices for producing reliable cost estimates identified in the Cost Guide apply to NRC's situation as a regulatory decision maker, but we determined that certain best practices do not apply to NRC's situation. For example, we neither assessed NRC's overall procedures nor its specific estimate for filtered venting systems, against the best practice of regularly updating a cost estimate to reflect significant changes in the program because we determined that this practice is not applicable to cost estimates used as part of a regulatory analysis to support a specific regulatory decision. Updating cost estimates through the life of a project is to identify where actual costs vary from the estimate so that project managers can make adjustments when necessary and identify cost overruns. Because NRC is using cost estimates to make a regulatory decision and not to manage the costs of a specific project, this step is not applicable and, as a result, we excluded this best practice from our assessment. We also did not assess NRC staff's 2012 estimate for filtered venting systems against the best practice of documenting, explaining, and reviewing variances between planned and actual costs for the same reasons described previously. We assessed NRC's overall cost estimating procedures and specific cost estimate for filtered venting systems against relevant best practices within the following four characteristics: (1) comprehensive, (2) well- documented, (3) accurate, and (4) credible. Each characteristic is associated with specific best practices criteria. We used a five point scale for these assessments: not met = 1, minimally met = 2, partially met = 3, substantially met = 4, and fully met = 5.[Footnote 24] After assessing NRC's procedures and cost estimate against each relevant best practice criterion, we calculated the average of the best practice assessment ratings to determine the overall rating for each of the four characteristics. The resulting average became the overall assessment as follows: not met = 0 to 1.4; minimally met = 1.5 to 2.4; partially met = 2.5 to 3.4; substantially met = 3.5 to 4.4; and fully met = 4.5 to 5.0. After conducting our initial assessments of NRC's procedures and cost estimate, we shared our draft analysis with NRC staff to provide them an opportunity to comment. Based on their comments and additional information provided, we revised our draft assessment to produce the final assessment. To determine the extent to which NRC's cost estimating procedures are consistent with best practices for producing reliable cost estimates, we compared NRC's cost estimating procedures with best practices contained in the GAO Cost Guide. We reviewed NRC's cost estimating procedures contained in the Regulatory Analysis Guidelines of the U.S. Nuclear Regulatory Commission[Footnote 25] and the Regulatory Analysis Technical Evaluation Handbook.[Footnote 26] We interviewed NRC staff responsible for cost estimation to understand how the procedures would be applied. We also reviewed NRC documentation related to its ongoing effort to update its cost benefit guidance. We used the information collected to determine whether NRC's procedures incorporate best practices using the five-point rating system and scoring method described above. To examine the extent to which NRC staff's specific cost estimate for proposed filtered venting systems was reliable, we evaluated whether the cost estimate was generated according to best practices outlined in the GAO Cost Guide. We reviewed NRC's Regulatory Analysis and Backfitting for the Consideration of Additional Requirement for Containment Venting Systems for Boiling Water Reactors with Mark I and Mark II Containments,[Footnote 27] as well as other documentation associated with the filtered venting system estimate. We also reviewed external reports on filtered venting systems and relevant documentation of system costs obtained from stakeholders, such as industry groups. We interviewed NRC staff responsible for preparing the filtered venting system cost estimate. We attended and reviewed summaries of public meetings on NRC's ongoing rulemaking related to filtering strategies and on cumulative effects of regulation. We used this information to determine, whether the cost estimate was developed using best practices, rating the estimate against individual best practices criteria using the five-point rating system and scoring method described above. In order for a cost estimate to be considered reliable, the estimate must have "substantially" or "fully" met each of the four characteristics. We did not generate our own independent cost estimate for filtered venting systems for comparison to the NRC staff's cost estimate. We conducted this performance audit from December 2013 to December 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Boiling Water Reactor with Proposed Filtered Venting System (Corresponds to Figure 1): This appendix provides additional details on filtered venting systems, including rollover information, on figure 1. Table 3 shows descriptions of the diagram components depicted in figure 1. Table 3: Description of Diagram Components: Number and label: 1-Reactor vessel; Text box description: Reactor vessel; This contains the reactor core, including the reactor's fuel, where the fission reaction occurs. In an accident involving the loss of cooling systems, the vessel may be damaged by high heat and pressure allowing radioactive material and steam to enter the drywell. Number and label: 2-Drywell; Text box description: Drywell; This is part of the containment system, which is an open space intended to prevent the escape of radioactive materials and steam in an accident. The drywell is enclosed by a steel and reinforced concrete shell. Number and label: 3-Wetwell; Text box description: Wetwell; This is part of the containment system, which has a large pool of water to absorb excess energy from the reactor. In an accident, steam flows through a set of pipes from the drywell into the pool of water, reducing pressure in the drywell. Number and label: 4-Hardened vent; Text box description: Hardened vent; This system of hardened pipes can vent to the outside atmosphere to relieve pressure on the containment system, preventing a less controlled breach of the containment from occurring. Certain boiling water reactors are in the process of upgrading these vents to meet new standards. Number and label: 5a-Release from hardened vent; Text box description: Release from hardened vent; Releasing pressure from the containment system helps minimize the effects of an accident. The hardened vent provides a path to release this pressure, but would potentially release radioactive materials to the atmosphere. NRC is currently exploring various filtering strategies to prevent the release of radioactive materials, some of which involve installing a separate filter and some of which do not. Number and label: 5b-Release from filter; Text box description: Release from filter; Releasing pressure from the containment system helps minimize the effects of an accident. The hardened vent provides a path to release this pressure and the proposed filtered venting system creates an additional layer of protection by capturing radioactive materials before they enter the atmosphere. Number and label: 6-Proposed filtered venting system; Text box description: Proposed filtered venting system; This system adds an external filter to the hardened pipes used to vent to the outside atmosphere. Venting from the containment system during an accident would pass through the filter which would capture radioactive materials in a pool of water using several filtering techniques. Source: GAO. GAO-15-98. [End of table] [End of section] Appendix III: Assessment of NRC's Overall Cost Estimating Procedures: The Cost Guide can be used to assess cost estimating procedures to determine whether they meet the four characteristics--comprehensive, well documented, accurate, and credible--for being reliable. Table 4 presents our assessment of NRC's overall cost estimating procedures against the four characteristics and related best practices. Table 4: GAO's Assessment of NRC's Overall Cost Estimating Procedures Against Best Practices: Characteristic: Comprehensive; Overall assessment: Partially met; Related best practices for cost estimating procedures: Develop the estimating plan; Detailed assessment by best practice[A]: Substantially met. NRC's procedures detail the steps for selecting alternatives and performing a value-impact analysis. While the procedures do not detail schedules for estimate development, they do establish steps for the process including estimated durations of each step. Related best practices for cost estimating procedures: Determine the estimating structure; Detailed assessment by best practice[A]: Minimally met. NRC's procedures do not provide guidance for the development of a work breakdown structure (WBS), a best practice that involves deconstructing a program's end product into successive levels with smaller specific elements until the work is subdivided to a level suitable for management control. Instead, it provides 18 attributes that are not the same as a WBS. These attributes minimally meet the best practice of having a WBS that facilitates a reflection of requirements and provides a basis for developing a cost estimate. Characteristic: Well-documented; Overall assessment: Substantially met; Related best practices for cost estimating procedures: Define estimate's purpose; Detailed assessment by best practice[A]: Fully met. NRC procedures are in line with best practices to define the purpose of a cost estimate. Related best practices for cost estimating procedures: Define the program's characteristics; Detailed assessment by best practice[A]: Partially met. NRC's procedures detail the steps necessary to have an adequate understanding of the program. However, there was no guidance to capture changes to key assumptions or to update the technical baseline to reflect changes. Related best practices for cost estimating procedures: Identify ground rules and assumptions; Detailed assessment by best practice[A]: Fully met. NRC's procedures are specific about identifying ground rules and assumptions. Related best practices for cost estimating procedures: Obtain the data; Detailed assessment by best practice[A]: Substantially met. NRC's procedures state that staff should provide documentation that the analysis is based on the best reasonably attainable scientific, technical, and economic information available and should rely on peer- reviewed literature, when available, and provide the source for all original information; however it did not contain guidance that historical data should be studied or that cost drivers should be considered. Related best practices for cost estimating procedures: Document the estimate; Detailed assessment by best practice[A]: Partially met. NRC's procedures provide guidance regarding conducting and documenting the regulatory analysis, but they do not require documentation of all parameters, descriptions, or calculations used to develop the cost estimate. Related best practices for cost estimating procedures: Present estimate to management for approval; Detailed assessment by best practice[A]: Substantially met. NRC's procedures do not require the estimate alone to be presented to management; but, they do require management review of rulemaking packages, which include the regulatory analysis and cost estimates. Characteristic: Accurate; Overall assessment: Minimally met; Related best practices for cost estimating procedures: Develop the point estimate; Detailed assessment by best practice[A]: Minimally met. NRC's procedures did not contain guidance or direction to identify the methods used to develop a point estimate. Related best practices for cost estimating procedures: Update the estimate to reflect actual costs/changes; Detailed assessment by best practice[A]: Not applicable. NRC procedures were not assessed against this best practice because it is inapplicable to regulatory analysis cost estimates. NRC does not update estimates with actual costs as projects progress. Characteristic: Credible; Overall assessment: Partially met; Related best practices for cost estimating procedures: Compare the point estimate to an independent cost estimate; Detailed assessment by best practice[A]: Minimally met. NRC's procedures did not contain guidance or direction to compare the point estimate to an independent cost estimate. The procedures state that cost analysts review the estimate during the estimate's development, but it does not require the review to be done by an independent organization. Related best practices for cost estimating procedures: Conduct a sensitivity analysis; Detailed assessment by best practice[A]: Fully met. NRC's procedures provided adequate guidance for conducting a sensitivity analysis. Related best practices for cost estimating procedures: Conduct a risk and uncertainty analysis; Detailed assessment by best practice[A]: Partially met. NRC's procedures provide some guidance for conducting a risk analysis but they did not specifically address doing a Monte Carlo simulation to determine a level of confidence and associated contingency reserve needed to address potential risks. However, NRC's procedures do not require uncertainty analyses to always be performed, whereas best practices always call for uncertainty analyses. Source: GAO analysis of NRC's cost estimating procedures. GAO-15-98. [A] The ratings we used in this analysis are as follows: "Not met" means we found no evidence that satisfies the best practice. "Minimally met" means we found evidence that satisfies a small portion of the best practice. "Partially met" means we found evidence that satisfies about half of the best practice. "Substantially met" means we found evidence that satisfies a large portion of the best practice. "Fully met" means we found complete evidence that satisfies the entire best practice. [End of table] [End of section] Appendix IV: Assessment of NRC Staff's 2012 Cost Estimate for Proposed Filtered Venting Systems: The Cost Guide can be used to assess specific cost estimates to determine whether they meet the four characteristics--comprehensive, well-documented, accurate, and credible--for being reliable. Table 5 presents our assessment of NRC staff's 2012 cost estimate for proposed filtered venting systems against the four characteristics and related best practices. Table 5: GAO's Assessment of NRC Staff's 2012 Cost Estimate for Proposed Filtered Venting Systems Against Best Practices: Characteristic: Comprehensive; Overall assessment: Partially met; Related best practices for developing cost estimates: The cost estimate includes all life cycle costs; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Substantially met. The estimate covered implementation and operational costs for both industry and NRC over a 25-year period, but it did not include decommissioning costs. Related best practices for developing cost estimates: The cost estimate completely defines the program, reflects the current schedule, and is technically reasonable; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Partially met. The estimate documented proposed technical requirements for filtered venting systems, but these requirements did not address project risks. Related best practices for developing cost estimates: Partially met: The cost estimate work breakdown structure is product-oriented, traceable to the statement of work/objective, and at an appropriate level of detail to ensure that cost elements are neither omitted nor double-counted; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Minimally met. The estimate did not include a work breakdown structure, a best practice that breaks down product-oriented elements into a hierarchical structure that shows how the elements relate to one another as well as the overall end product. NRC staff noted that the estimate incorporated cost elements based on NRC's guidance, but these cost elements were not product-oriented and were not at a sufficient level of detail. Related best practices for developing cost estimates: The estimate documents all cost-influencing ground rules and assumptions; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Partially met. The estimate included cost-influencing assumptions, but did not address risks related to these assumptions. Characteristic: Well-Documented; Overall assessment: Partially met; Related best practices for developing cost estimates: The documentation should capture the source data used, the reliability of the data, and how the data were normalized; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Partially met. The estimate documentation described the data sources, but did not provide detail on how the data were used and normalized. Related best practices for developing cost estimates: The documentation describes in sufficient detail the calculations performed and the estimating methodology used to derive each element's cost; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Partially met. The estimate employed analogy and engineering build up methods to calculate the cost, but it did not describe the approach or provide calculations in the documentation. Related best practices for developing cost estimates: The documentation describes step by step how the estimate was developed so that a cost analyst unfamiliar with the program could understand what was done and replicate it; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Minimally met. NRC staff described the steps to GAO; however, the estimate did not include step-by-step explanations of how the estimate was developed, and NRC did not provide the cost model with their calculations. Related best practices for developing cost estimates: The documentation discusses the technical baseline description and the data in the baseline are consistent with the estimate; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Minimally met. The estimate did not include sufficient detail to determine if it was consistent with the technical baseline because it did not always provide details on how specific technical requirements would connect to costs. Related best practices for developing cost estimates: The documentation provides evidence that the cost estimate was reviewed and accepted by management; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Partially met. The estimate was approved by NRC management, but there is limited evidence of detailed review. Characteristic: Accurate[B]; Overall assessment: Partially met; Related best practices for developing cost estimates: The cost estimate results are not overly conservative or optimistic and based on an assessment of most likely costs; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Not met. The estimate did not include a cost uncertainty analysis to determine risk levels. Without this analysis, it cannot be determined if the estimate is overly conservative or optimistic. Related best practices for developing cost estimates: The estimate has been adjusted properly for inflation; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Substantially met. The estimate appeared to adjust cost elements for inflation, but calculation of the adjustments was not included in estimate documentation. Related best practices for developing cost estimates: The estimate contains few, if any, minor mistakes; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Substantially met. The estimate underwent an internal quality control process and appeared to have no substantive errors. Related best practices for developing cost estimates: The cost estimate is regularly updated to reflect significant changes in the program so that it is always reflecting current status; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Not applicable. This best practice is not applicable because it does not apply to regulatory analysis cost estimates. Related best practices for developing cost estimates: Variances between planned and actual costs are documented, explained, and reviewed; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Not applicable. This best practice is not applicable because it does not apply to regulatory analysis cost estimates. Related best practices for developing cost estimates: Not met: The estimate is based on a historical record of cost estimating and actual experiences from other comparable programs; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Not met: Partially met. The estimate from the analogy method relied on foreign nuclear plant data, but these plant level cost data were estimated by engineers and may have been a part of larger projects that included the installation of filters. Related best practices for developing cost estimates: The estimating technique for each cost element was used appropriately; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Partially met. The estimate employed an analogy method and an engineering build up method, but it did not include a work breakdown structure or any documentation of the engineering build up method. Characteristic: Credible; Overall assessment: Not met; Related best practices for developing cost estimates: The cost estimate includes a sensitivity analysis that identifies a range of possible costs based on varying major assumptions, parameters, and data inputs; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Not met. The estimate did not include any sensitivity analysis of cost assumptions, parameters, and data inputs. Related best practices for developing cost estimates: A risk and uncertainty analysis was conducted that quantified the imperfectly understood risks and identified the effects of changing key cost driver assumptions and factors; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Not met. The estimate did not include any risk and uncertainty analysis to determine the risk level associated with the cost estimates. Related best practices for developing cost estimates: Major cost elements were cross-checked to see whether results were similar; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Minimally met. The estimate included an engineering build- up as a cross-check of the analogy method estimate, but NRC staff stated they had no documentation of the estimate. Related best practices for developing cost estimates: An independent cost estimate was conducted by a group outside the acquiring organization to determine whether other estimating methods produce similar results; Detailed assessment of NRC staff's filtered venting system cost estimate[A]: Not met. No independent cost estimate was prepared. Source: GAO analysis of NRC staff's filtered venting system cost estimate. GAO-15-98. [A] The ratings we used in this analysis are as follows: "Not met" means we found no evidence that satisfies the best practice. "Minimally met" means we found evidence that satisfies a small portion of the best practice. "Partially met" means we found evidence that satisfies about half of the best practice. "Substantially met" means we found evidence that satisfies a large portion of the best practice. "Fully met" means we found complete evidence that satisfies the entire best practice. [B] According to the Cost Guide, cost estimates are accurate when they are not overly conservative or too optimistic, based on an assessment of most likely costs, adjusted properly for inflation, and contain few, if any, minor mistakes. [End of table] [End of section] Appendix V: Comments from the Nuclear Regulatory Commission: United States Nuclear Regulatory Commission: Washington, D.C. 20555-0001: November 18, 2014: Mr. Franklin Rusco, Director: Natural Resources and Environment: U.S. Government Accountability Office: 441 G Street, NW: Washington, DC 20548: Dear Mr. Rusco: Thank you for providing the U.S. Nuclear Regulatory Commission (NRC) the opportunity to review and comment on the U.S. Government Accountability Office's (GAO's) draft report GAO-15-98, "Nuclear Regulatory Commission: NRC Needs to Improve its Cost Estimates by Incorporating More Best Practices." The NRC staff appreciates the time and effort that you and your staff have taken to review this topic. The NRC staff generally agrees with the GAO's recommendation that the NRC could improve its regulatory analyses and has efforts underway to do so. These efforts have been discussed with GAO and are further described below. The NRC does not believe however, that the standards used by GAO to assess our program are appropriate. The Office of Management and Budget (OMB) has issued specific guidance on the attributes of a regulatory analysis (OMB Circular A-4, "Regulatory Analysis"). The NRC uses OMB Circular A-4 as a reference for the NRC guidance on regulatory analysis. The staff believes that OMB Circular A-4 is more appropriate for determining the adequacy of the NRC's cost estimates in regulatory analysis than the GAO's Cost Estimating and Assessment Guide, which is more appropriate to acquisitions of major systems. As noted in the staff's prior comments, it is rare when the NRC acquires major systems. The NRC's statutorily designated work consists of regulating and licensing private entities that construct and operate their own nuclear facilities, or otherwise use and possess radioactive material. The staff believes that if the OMS Circular A-4 standards were applied , the GAO's conclusions would confirm that there are no systemic deficiencies with respect to the quality and accuracy of the staffs analytical work. Consistent with OMB Circular A-4, the NRC staff develops regulatory analyses for its rulemakings and other regulatory actions, such as orders and regulatory guides. The regulatory analyses undergo an intra- agency review and are typically released for public notice and comment along with the proposed rulemaking or other regulatory actions. Comments, when provided, are considered and the regulatory analyses are revised , as appropriate, to accompany final rules. In many cases, the regulatory analyses are also reviewed by the NRC's Advisory Committee on Reactor Safeguards (ACRS), a statutorily mandated committee under Section 29 of the Atomic Energy Act of 1954, as amended, through which experts representing many technical perspectives can provide independent advice that is factored into the Commission's decision-making process. The staff relies on the best available information to calculate the cost and benefits of its proposed regulatory actions and uses the ACRS, as well as the public comment process, to further refine that work. The agency constantly strives to improve the quality and transparency of its regulatory analyses and looks forward to using GAO's insights in its ongoing plans to further improve the NRC's cost estimates. The NRC staff also agrees with the GAO regarding the importance of regulatory analyses being comprehensive, well-documented, accurate, and credible. The specific example cited in the GAO report involved the NRC's consideration of filtered venting systems for bailing-water reactors (BWRs) with Mark I and Mark II containment designs. The staff used the OMB guidance in its evaluation of this issue and provided cost-benefit information to the Commission to consider in its deliberations. The Commission considered the quantitative information, including the cost estimates, and qualitative factors, and directed the staff to further assess filtering strategies and severe accident management of BWR Mark I and II containments using the agency's rule making procedures. While the staff agrees that" should pursue improvements to our cost-estimating procedures, the NRC must meet its statutory obligations and the use of the OMB guidance and our existing processes best support this objective. Recently, the Commission directed the NRC staff to engage the nuclear industry to perform case studies on specific regulations to assess the accuracy of cost and schedule estimates used in the NRC's regulatory analyses in assessing the cumulative effects of regulation (CER). This effort is ongoing, as described in SECY-14-0002, "Plan for Updating the U.S. Nuclear Regulatory Commission's Cost-Benefit Guidance- (Agencywide Documents Access and Management System Accession No. ML 13274A495). Also, described in SECY-14-0002, the staff's plan focuses on updating the content and structure of the NRC's cost-benefit guidance documents, primarily NUREG/BR-0058, "Regulatory Analysis Guidelines of the U.S. Nuclear Regulatory Commission," and NUREG/BR-0184, "Regulatory Analysis Technical Handbook." As it pursues these efforts, the NRC staff will consider the applicable best practices from the GAO's Cost Estimating and Assessment Guide, as discussed in the draft GAO report. The NRC staff appreciates the opportunity to comment on the draft GAO report. Should you have any questions or need further information, please contact Jesse Arildsen at 301-415-1785. Sincerely, Signed by: [Illegible], for: Mark A. Satorius: Executive Director for Operations: [End of section] Appendix VI: GAO Contact and Staff Acknowledgments: GAO Contact: Frank Rusco, (202) 512-3841 or ruscof@gao.gov: Staff Acknowledgments: In addition to the individual named above, Ned Woodward (Assistant Director), Brian Bothwell, Richard Burkard, Eric Charles, Juaná Collymore, John Delicath, R. Scott Fletcher, Cindy Gilbert, Tom Laetz, Angela Miles, Karen Richey, and Jack Wang made significant contributions to this report. [End of section] Footnotes: [1] Thirty-one of the 35 BWRs currently operating in the United States are Mark I or Mark II designs. In its 1989 Generic Letter 89-16, NRC recommended that BWRs with Mark I designs install a reliable wetwell vent to provide a path for releasing pressure in order to both prevent and mitigate the consequences of severe accidents. A hardened vent is a system of hardened pipes to the outside atmosphere that can relieve pressure on the containment system. After the Fukushima accident, NRC issued Order EA-12-050 in 2012 that required BWRs with Mark I and Mark II designs to install reliable hardened vents. NRC issued order EA-13- 109 in 2013, which superseded EA-12-050, and required the affected licensees to upgrade or replace their hardened vents to remain functional during severe accident conditions. These requirements include that the vent shall be capable of being operated manually and of operating for at least 24 hours following the loss of power. [2] Nuclear Regulatory Commission, Consideration of Additional Requirements for Containment Venting Systems for Boiling Water Reactors with Mark I and Mark II Containments, SECY-12-0157 (Rockville, MD: Nov. 26, 2012). [3] The Commission--made up of five Commissioners appointed by the President and confirmed by the Senate--formulates policies, develops regulations governing nuclear reactor and nuclear material safety and security, issues orders to licensees, and adjudicates legal matters. NRC staff support the commission's policy making efforts, including conducting regulatory analysis for the commission on specific policy matters. [4] Under its regulations, NRC may modify the license of a nuclear facility operator to require the operator to modify or improve a structure or design of a facility, among other actions. This type of modification, or "backfitting," may be based on a determination by NRC that there is a substantial increase in the overall protection of the public health and safety or the common defense and security to be derived from the backfit and that the costs of implementation for that facility are justified in view of this increased protection. 10 C.F.R. § 50.109(a)(3), Alternatively, NRC may require a licensee to modify a facility without regard to cost if it determines that the action is necessary to ensure that the facility provides adequate protection to the health and safety of the public and is in accord with the common defense and security. 10 C.F.R. § 50.109(a)(4). [5] For this regulatory analysis, NRC also produced a cost estimate of approximately $2 million per reactor unit to upgrade hardened vents on BWRs with Mark I and II designs. We did not assess NRC's cost estimate for upgrading the hardened vents. [6] NRC defines defense in-depth as an approach to designing and operating nuclear facilities that prevents and mitigates accidents that release radiation or hazardous materials. The key is creating multiple independent and redundant layers of defense to compensate for potential human and mechanical failures so that no single layer, no matter how robust, is exclusively relied upon. Defense-in-depth includes the use of access controls, physical barriers, redundant and diverse key safety functions, and emergency response measures. [7] As of October 2014, the NRC has not taken action to require U.S. nuclear plants to install filtered venting systems. [8] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, [hyperlink, http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: March 2009). [9] NRC conducts a value-impact (benefit cost) analysis as a substantial part of its regulatory analysis. The NRC regulatory guidelines define values as the beneficial aspects anticipated from a proposed regulatory action such as, but not limited to, the (1) enhancement of health and safety, (2) protection of the natural environment, (3) promotion of the efficient functioning of the economy and private markets, and (4) elimination or reduction of discrimination or bias. Impacts are defined as the costs anticipated from a proposed regulatory action such as, but not limited to, the (1) direct costs to the NRC and Agreement States in administering the proposed action and to licensees and others in complying with the proposed action; (2) adverse effects on health, safety, and the natural environment; and (3) adverse effects on the efficient functioning of the economy or private markets. [10] According to NRC, there is no statute, NRC regulation, or executive order requiring NRC to conduct a regulatory analysis, but the agency has been voluntarily performing them since 1976. [11] NRC staff told us that they use the Office of Management and Budget's (OMB) Circular A-4 Regulatory Analysis as a reference for their own guidance on regulatory analysis. Circular A-4 provides OMB 's guidance to federal agencies on the development of regulatory analysis. [12] The regulatory analysis also included an option to continue with the implementation of order EA-12-050 for reliable hardened vents and take no additional action and an option to require licensees to develop severe accident confinement strategies for their plants. [13] Filters can use different approaches, and one major approach is to use water to capture radioactive particles. While filtering technology can capture a significant number of radioactive particles, it has no ability to capture some radioactive particles, such as noble gases. [14] According to NRC staff, the regulatory basis describes the technical, legal, and policy information that support the staff's recommendation to the Commission that a current regulation should be amended or a new regulation promulgated. The regulatory basis includes an analysis of the costs and benefits associated with such rulemaking. The regulatory basis provides a natural point in the process to allow for an informed decision regarding whether a regulatory action is justified such that a rulemaking effort should commence. [15] Sensitivity analysis examines how the cost estimate is affected by a change in an assumption's value. As a best practice, sensitivity analysis should be included in all cost estimates because it examines the effects of changing assumptions and ground rules. [16] According to the Cost Guide, a work breakdown structure breaks down product-oriented elements into a hierarchical structure that shows how elements relate to one another, as well as to the overall end product. It also allows a program to track cost and schedule by defined deliverables, such as hardware or software components. It also allows a program manager to more precisely identify which components are causing cost or schedule overruns and to more effectively mitigate the root cause of the overruns. [17] We did not assess NRC's procedures against the best practice of updating a cost estimate to reflect actual costs and changes because we determined that this practice is not applicable to cost estimates used as part of a regulatory analysis to support a specific regulatory decision. Updating cost estimates through the life of a project is to identify where actual costs vary from the estimate so that project managers can make adjustments when necessary and identify cost over runs. Because NRC is using cost estimates to make a regulatory decision and not to manage the costs of a specific project, this step is not applicable and as a result we excluded this best practice from our assessment. [18] In March 2013, the Commission directed NRC staff to engage the industry to seek volunteer facilities to perform case studies to review the accuracy of cost and schedule estimates used in NRC's regulatory analysis. In response to this request, the industry analyzed the cost and schedule estimates in the NRC's regulatory analyses supporting three regulations. The industry presented their results during a January 28, 2014 public meeting. [19] Cost data may be adjusted in a number of ways to ensure that cost estimators are working with a consistent and comparable data set including adjusting cost data for inflation or grouping major costs with similar missions, characteristics or operating environments. [20] As with our assessment of NRC's cost estimating procedures, we excluded best practices related to updating a cost estimate. For additional explanation, please see appendix I. [21] According to the Cost Guide, cost estimates are accurate when they are not overly conservative or too optimistic, based on an assessment of most likely costs, adjusted properly for inflation, and contain few, if any, minor mistakes. [22] An engineering build-up estimate provides an estimate of the total cost of all individual cost elements, such as labor and specific materials. [23] [hyperlink, http://www.gao.gov/products/GAO-09-3SP]. [24] The ratings we used in this analysis are as follows: "Not met" means we found no evidence that satisfies the best practice. "Minimally met" means we found evidence that satisfies a small portion of the best practice. "Partially met" means we found evidence that satisfies about half of the best practice. "Substantially met" means we found evidence that satisfies a large portion of the best practice. "Fully met" means we found complete evidence that satisfies the entire best practice. [25] U.S. Nuclear Regulatory Commission, Regulatory Analysis Guidelines of the U.S. Nuclear Regulatory Commission, NUREG/BR-0058 (Rockville, MD: 2004). [26] U.S. Nuclear Regulatory Commission, Regulatory Analysis Technical Evaluation Handbook, NUREG/BR-OI84 (Rockville, MD: 1997). [27] Nuclear Regulatory Commission, Consideration of Additional Requirements for Containment Venting Systems for Boiling Water Reactors with Mark I and Mark II Containments, SECY-12-0157 (Rockville, MD: Nov. 26, 2012). [End of section] GAO's Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO's commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO's website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select "E-mail Updates." Order by Phone: The price of each GAO publication reflects GAO's actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO's website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov: (202) 512-4400: U.S. Government Accountability Office: 441 G Street NW, Room 7125: Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov: (202) 512-4800: U.S. Government Accountability Office: 441 G Street NW, Room 7149: Washington, DC 20548. [End of document]