This is the accessible text file for GAO report number GAO-04-386SP entitled 'Best Practices: Using A Knowledge-Based Approach To Improve Weapon Acquisition' which was released on January 30, 2004. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. GAO-04-386SP: Best Practices: Using A Knowledge-Based Approach To Improve Weapon Acquisition: This booklet describes the knowledge-based approach successful organizations generally follow to deliver sophisticated products in less time and at lower costs. The Department of Defense (DOD) is on the threshold of several major investments in acquisition programs that are likely to dominate budget and doctrinal debates well into the next decade. Over the next 5 years alone, DOD’s overall investments are expected to average $150 billion a year as DOD works to keep legacy systems as well as transform our national defense capabilities for the future. To meet this challenge, it is essential that sound foundations for investments in systems be laid now so that the resulting programs can be executed within estimates of available resources. At the request of the Congress, we have been examining ways DOD can optimize its investment in weapon systems, drawing on lessons learned from the best, mostly commercial, product development efforts. Leading commercial firms we have studied have developed increasingly sophisticated products in less time and at lower cost. Key to their success is their knowledge-based approach to the acquisition of new products. A knowledge-based approach is supported by incentives that encourage realism and candor. This booklet highlights the results of our work to date. We continue to explore additional facets of the acquisition process to identify best practices. More details on our work can be found in the reports cited in this brochure. GAO’s best practices work is done under the direction of Katherine V. Schinasi. For more on this work, go to [hyperlink, http://www.gao.gov/bestpractices]. If you have questions or would like to discuss our reviews, please contact Ms. Schinasi at (202) 512-4841 or schinasik@gao.gov. Signed by: Jack L. Brock Jr. Managing Director: Acquisition and Sourcing Management: [End of section] A Knowledge-Based Approach Puts Acquisition Programs In A Better Position To Succeed: Leading commercial firms expect that their managers will deliver high quality products on time and within budgets. Doing otherwise could result in losing a customer in the short term and losing the company in the long term. Thus, these firms have adopted practices that put their individual program in a good position to succeed in meeting these expectations on individual products. Collectively, these practices ensure that a high level of knowledge exists about critical facets of the product at key junctures during its development and is used to deliver capability as promised. DOD has recognized the need to adopt ways of doing business that enable it to achieve similar results for its weapons acquisitions. Our reviews have shown that there are three critical junctures at which firms must have knowledge to make large investment decisions. First, before a product development is started, a match must be made between the customers’ needs and the available resources—technical and engineering knowledge, time, and funding. Second, a product’s design must demonstrate its ability to meet performance requirements and be stable about midway through development. Third, the developer must show that the product can be manufactured within cost, schedule, and quality targets and is demonstrated to be reliable before production begins. If the knowledge attained at each juncture does not confirm the business case on which the acquisition was originally justified, the program does not go forward. In applying the knowledge-based approach, the most-leveraged decision point of the three junctures is matching the customer’s needs with the developer’s resources. This initial decision sets the stage for the eventual outcome—desirable or problematic. The match is ultimately achieved in every development program, but in successful development programs, it occurs before product development. In successful programs, negotiations and tradeoffs occur before product development is started to ensure that a match exists between customer expectations and developer resources. Leading firms thus make an important distinction between technology development and product development. Technologies that are not mature continue to be developed in the technology base—they are not included in a product development. With achievable requirements and commitment of sufficient investment to complete the development, programs are better able to deliver products at cost and on schedule. When knowledge lags, a number of risks are introduced into the acquisition process that can result in cost overruns, schedule delays, and inconsistent product performance. An approach that enables organizations to achieve a match between needs and resources is evolutionary product development. Under this approach, basic requirements are achieved first, with additional capabilities planned for future generations of the product. Because product development is incremental, achieving knowledge is more manageable. Commercial companies have found that trying to capture the knowledge needed to stabilize the design of a product with considerable new technical content is an unwieldy task—especially if the goal is to reduce development cycle times and get the product to the marketplace as quickly as possible. With evolutionary development, a product’s design uses only components and subsystems whose reliability has been proven through past use or testing. Knowledge Points at a Glance: The knowledge-based process followed by leading organizations is highlighted and further discussed below. Figure: Knowledge Points at a Glance: [See PDF for image] This figure is an illustration of knowledge points, as follows: Technology Development: Leads to: Product Development: Program launch and Knowledge Point 1 occur at the beginning of product development; Knowledge point 2 occurs in the mid- point of product development, which leads to: Production: Knowledge point 3 occurs at the beginning of production. * Knowledge point 1: Resources and needs match. Knowledge point 1 occurs when a sound business case is made for the product—that is, a match is made between the customer’s requirements and the product developer’s available resources in terms of knowledge, time, and money. To determine their available resources, successful companies rely on current and valid information from predecessor programs, new technologies that have demonstrated a high level of maturity, system engineering data, and experienced people. Successful companies also communicate extensively with customers to match their wants and needs with the firm’s available resources and with its ability to manufacture an appropriate product. * Knowledge point 2: Product design is stable. Knowledge point 2 occurs when a company determines that a product’s design is stable—that is, it will meet customer requirements and cost and schedule targets. A best practice is to achieve design stability at the product’s critical design review, usually held midway through development. In DOD, the critical design review occurs when the first phase of product development—product integration—has been completed and the second phase—product demonstration—is about to begin. Completion of at least 90 percent of engineering drawings at the critical design review provides tangible evidence that the design is stable. * Knowledge point 3: Production processes are mature. This level of knowledge is achieved when it has been demonstrated that the product can be manufactured within cost, schedule, and quality targets. A best practice is to ensure that all key manufacturing processes are in statistical control—that is, they are repeatable, sustainable, and capable of consistently producing parts within the product’s quality tolerances and standards—at the start of production. It is important that the product’s reliability be demonstrated before production begins, as investments can increase significantly if defective parts need to be repaired or reworked. [End of figure] Measuring Success at Key Knowledge Points: The organizations we have studied apply a variety of tools and measures to gauge the level of knowledge that they have attained before making major investment decisions. * Measuring technology readiness: Many programs now use an analytical tool—technology readiness levels (TRL)—that can assess the maturity level of technology as well as the risk that maturity poses if the technology is included in a product development. The experiences of the DOD and commercial technology development cases we reviewed indicate that demonstrating a high level of maturity before allowing new technologies into product development programs puts those programs in a better position to succeed. Simply put, the more mature technology is at the start of the program, the more likely the program will succeed in meeting its objectives. There are nine TRLs, each denoting a level of demonstrated performance, beginning with concept—the lowest level of readiness—to application, where the technology has been “flight proven” under mission conditions. The higher the TRL, the smaller the gap between the technology’s maturity and the product’s requirements, and the lower the risk of including the technology in the product’s development. Our best practices work has shown that TRL 7—demonstration of a technology in an operational environment—is the level of technology needed to minimize risks when launching an acquisition program. Using TRLs to assess a technology’s maturity helps decisionmakers make informed choices about product development. If a technology lacks maturity, decisionmakers can choose to delay product development until the technology has matured sufficiently or lower the product’s requirements so that a less advanced but proven technology can be used. The more a technology has been proven, the more likely the product is to meet its objectives and the more likely the company can minimize unexpected problems and avoid potential schedule delays and cost overruns. * Measuring design readiness: The organizations we studied also understood the importance of having disciplined design reviews and getting agreement from the stakeholders that the product’s design had been demonstrated to meet requirements before beginning initial manufacturing. Each organization had a design review process that began at the component level, continued through the subsystem level, and culminated with a critical design review of the integrated system to determine if the product was ready to progress to the next phase of development. In addition to design engineers, a cross-functional team of stakeholders in the process included key suppliers, manufacturing representatives, and service and maintenance representatives. From past experience, leading organizations have discovered that cross-functional teams provide a complete perspective of the product. At critical design review, the leading organizations we studied generally require that at least 90 percent of the engineering drawings be completed. They consider engineering drawings to be a good measure of the demonstrated stability of a product’s design because the drawings represent the language used by engineers to communicate to the manufacturers the details of a new product design—what it looks like, how its components interface, how it functions, how to build it, and what critical materials and processes are required to fabricate and test it. Other Important Tools and Practices: Our reports have identified other tools and practices that enable successful implementation of the approach. These include: * Systems engineering for identifying gaps between requirements and resources so that they can be reconciled through effective trade-offs before product development. * Employment of integrated product teams to bring together in a single organization the different functions needed to design and manufacture a product, such as engineering, finance, test and evaluation, and manufacturing. * Supplier management approaches that optimize relationships with both contractors and subcontractors. * Using earned value management techniques to track a projects progress and assess its ability to meet cost and scheduling goals. * The use of a variety of testing and evaluation tools and techniques to validate a product’s performance early and throughout development. * The use of targeted, hands-on methods to ensure that program offices are trained on key practices. * Collection of statistical process control data to ensure manufacturing processes are consistently producing parts within quality standards. * Considering reasonable operating and support costs and the readiness or availability of equipment as requirements equal in importance to other performance characteristics. Coupling the knowledge-based approach with sound tools and techniques for oversight, strengthening the workforce, and ensuring the right knowledge is attained at the right times, increases an organization’s potential to meet cost, scheduling, and performance targets. It is still essential, however, that the right incentives and resources be in place to encourage program managers to employ such measures. Following A Knowledge-Based Acquisition Process Is Critical For Weapon Acquisitions: The majority of weapon system acquisitions we have reviewed over the past several decades experienced problems during acquisition that drove up costs and schedules and increased technical risks. Many programs have been restructured by DOD in the face of delays and cost growth; a few have been canceled. We have found that these problems are largely rooted in the failure to match customer’s needs with the developer’s resources—technical knowledge, timing, and funding--when starting product development. In other words, commitments were made to delivering capability without knowing whether technologies being pursued could really work as intended. Time and costs were consistently underestimated. Problems that surfaced early cascaded throughout development and magnified the risks facing the program. When Knowledge Is Not Attained At Key Junctures: * Launching an acquisition program before requirements and available resources are matched can result in a product that fails to perform as expected, costs more, or takes longer to develop. * Failure to ensure design stability about halfway through product development can result in design changes that are more costly to correct later in development or after the product is fielded. * Entering production before manufacturing processes are under control can result in product defects that require additional resources to rework or scrap—a costly and inefficient practice. On a number of programs, DOD has shown that it can manage its weapon system acquisition process to ensure important knowledge about technology and requirements, design, and manufacturing is captured and used to make informed and timely decisions before committing to substantial development and production investments. Moreover, over the past few years, DOD has made constructive changes to its acquisition policy to embrace best practices—especially those related to technology maturity and separating technology development from product development. If faithfully implemented, DOD’s policies of evolutionary acquisition and phased requirements will make it easier to match resources with needs before starting a new product development. Funding pressures and the need to modernize systems across the Department will continue to make it paramount for DOD to adopt practices that can save money and deliver new capabilities quicker. However, a number of incentives—many tied to funding—continue to undermine DOD’s ability to achieve a match between needs and resources at the onset of weapon acquisition programs. Unlike the commercial world where the focus is on delivering a product to market, DOD’s system focuses on competing for resources. In the competition for funding, managers are encouraged to launch product developments before technologies are mature. Because a proven way to win support for a new weapons acquisition program is to promote unprecedented performance features and design characteristics, managers have implicit incentives to rely on immature technologies. Moreover, because funding is competitive and DOD’s forecasts of cost, schedule, and performance are largely based on immature technologies and other unknowns, estimates tend to be squeezed into insufficient profiles of available funding. Other factors, such as short tenures and career pressures, discourage program managers from saying no to requirements that are later discovered to have been unreasonable. Thus, to meet the investment challenge of modernizing its forces, DOD will need not only to implement policies that embrace evolutionary, knowledge-based acquisitions, it will need to instill incentives that encourage realism and candor in the acquisition process and sustain its commitment to improving business practices. [End of section] Best Practices Reports: These reports can be found on the GAO Web site at [hyperlink, http://www.gao.gov]. Defense Acquisitions: Assessments of Major Weapon Programs, GAO-03-476, Washington, D.C.: May 2003. Best Practices: Setting Requirements Differently Could Reduce Weapon Systems’ Total Ownership Costs. GAO-03-57. Washington, D.C.: February 11, 2003. Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 2002. Defense Acquisitions: DOD Faces Challenges in Implementing Best Practices. GAO-02-469T. Washington, D.C.: February 27, 2002. Best Practices: DOD Teaming Practices Not Achieving Potential Results. GAO-01-510, Washington, D.C.: April 2001. Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 2001. Best Practices: A More Constructive Test Approach Is Key to Better Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 2000. Defense Acquisitions: Employing Best Practices Can Shape Better Weapon System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000. Best Practices: DOD Training Can Do More to Help Weapon System Programs Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 16, 1999. Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 1999. Best Practices: DOD Can Help Suppliers Contribute More to Weapon System Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998. Best Practices: Successful Application to Weapon Acquisition Requires Changes in DOD’s Environment. GAO/NSIAD-98-56. Washington, D.C.: February 24, 1998. Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 1996.