This is the accessible text file for GAO report number GAO-06-218 
entitled 'NASA: Implementing a Knowledge-Based Acquisition Framework 
Could Lead to Better Investment Decisions and Project Outcomes' which 
was released on January 23, 2006. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to Congressional Requesters: 

United States Government Accountability Office: 

GAO: 

December 2005: 

NASA: 

Implementing a Knowledge-Based Acquisition Framework Could Lead to 
Better Investment Decisions and Project Outcomes: 

GAO-06-218: 

GAO Highlights: 

Highlights of GAO-06-218, a report to congressional requesters: 

Why GAO Did This Study: 

The National Aeronautics and Space Administration (NASA) plans to spend 
over $100 billion on capabilities and technologies to achieve the 
initial goals of the President’s 2004 Vision for Space Exploration. In 
the past, NASA has had difficulty meeting cost, schedule, and 
performance objectives for some of its projects because it failed to 
adequately define project requirements and quantify resources. NASA 
will be further challenged by a constrained federal budget and a 
shrinking experienced NASA workforce. To help face these challenges and 
manage projects with greater efficiency and accountability, NASA 
recently updated its program and project management policy and is 
developing an agencywide systems engineering policy. 

GAO has issued a series of reports on the importance of obtaining 
critical information and knowledge at key junctures in major system 
acquisitions to help meet cost and schedule objectives. This report (1) 
evaluates whether NASA's policy supports a knowledge-based acquisition 
approach and (2) describes how NASA centers are implementing the 
agency’s acquisition policies and guidance. 

What GAO Found: 

While NASA’s revised policy for developing flight systems and ground 
support projects incorporates some of the best practices used by 
successful developers, it lacks certain key criteria and major decision 
reviews that support a knowledge-based acquisition framework. For 
example, NASA’s policy requires projects to conduct a major decision 
review before moving from formulation to implementation. Further, 
before moving from formulation to implementation, projects must 
validate requirements and develop realistic cost and schedule 
estimates, human capital plans, a preliminary design, and a technology 
plan—key elements for matching needs to resources. However, NASA’s 
policies do not require projects to demonstrate technologies at high 
levels of maturity before program start. By not establishing a minimum 
threshold for technology maturity, NASA increases the risk that design 
changes will be required later in development, when such changes are 
typically more costly to make. In addition, although NASA’s policy does 
require project managers to establish a continuum of technical and 
management reviews, it does not specify what these reviews should be, 
nor does it require major decision reviews at other key points in a 
product’s development. Acquiring knowledge at key junctures will become 
increasingly important as NASA proceeds to implement elements of the 
Vision. Without a major decision review at key milestones to ensure 
that the appropriate level of knowledge has been achieved to proceed to 
the next phase, the risk of cost and schedule overruns, as well as 
performance shortfalls, increases. 

NASA centers have varying approaches for implementing the agency’s 
policies and guidance. Some centers have established product 
development criteria that are similar to the criteria used in a 
knowledge-based acquisition, while other centers have not. As a result, 
each center reports a different level and type of knowledge about a 
project at key decision points. Centers also rely on project managers 
and systems engineers to employ good project management and systems 
engineering practices. However, given the loss of experienced project 
managers and the decline of in-house systems engineering and technical 
capabilities, that reliance could be problematic. These situations make 
it difficult for decision makers to evaluate projects on the same basis 
and make sound investment decisions and tradeoffs based on those 
evaluations. A standardized, knowledge-based approach would prepare 
NASA to face competing budgetary priorities and better position the 
agency to make difficult decisions regarding the investment in and 
termination of projects. 

What GAO Recommends: 

GAO is making several recommendations to help ensure NASA uses a 
knowledge-based acquisition approach in making informed investment 
decisions. NASA concurred with GAO’s recommendations. 

www.gao.gov/cgi-bin/getrpt?GAO-06-218. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Allen Li at (202) 512-
4841 or lia@gao.gov. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

NASA's Revised Policy Does Not Fully Support a Knowledge-Based Approach 
to Acquisitions: 

Lack of NASA-Wide Project Management Criteria May Result in Investment 
Decisions Based on Inconsistent Information: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Scope and Methodology: 

Appendix II: Activities That Enable the Capture of Design and 
Manufacturing Knowledge: 

Appendix III: Comments from the National Aeronautics and Space 
Administration: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

Related GAO Products: 

Tables: 

Table 1: Examples of Problems With NASA Programs/Projects: 

Table 2: KP1 Criteria Compared to NASA Policy Criteria: 

Table 3: Activities to Capture Design Knowledge and Make Decisions: 

Table 4: Activities to Capture Manufacturing Knowledge and Make 
Decisions: 

Figures: 

Figure 1: Conceptual Drawing of NASA Crew Vehicle Docked with Lunar 
Lander and Departure Stage in Earth Orbit: 

Figure 2: Project Categorization Scheme and Oversight Authorities from 
NPR 7120.5C: 

Figure 3: Knowledge-Based Acquisition Life Cycle: 

Figure 4: Life Cycle Phases for NASA's Flight Systems and Ground 
Support Projects: 

Figure 5: Technology Maturity Levels for Product Development: 

Figure 6: Comparison of NASA's Life Cycle with a Knowledge-Based 
Acquisition Life Cycle: 

Abbreviations: 

CADRe: Cost analysis data requirement: 

CDR: Critical design review: 

EVM: Earned value management: 

GPMC: Governing Program Management Committee: 

GSFC: Goddard Space Flight Center: 

ITA: Independent Technical Authority: 

JPL: Jet Propulsion Laboratory: 

JSC: Johnson Space Center: 

KP1: knowledge point 1: 

KP2: knowledge point 2: 

KP3: knowledge point 3: 

MSFC: Marshall Space Flight Center: 

NAR: Non-Advocate Review: 

NASA: National Aeronautics and Space Administration: 

NPD: NASA Policy Directive: 

NPR: NASA Procedural Requirement: 

PDR: Preliminary design review: 

Pre-NAR: Preliminary Non-Advocate Review: 

TRL: Technology readiness level: 

United States Government Accountability Office: 

Washington, DC 20548: 

December 21, 2005: 

The Honorable Bart Gordon: 
Ranking Minority Member: 
Committee on Science: 
House of Representatives: 

The Honorable Mark Udall: 
Ranking Minority Member: 
Subcommittee on Space and Aeronautics: 
Committee on Science: 
House of Representatives: 

The National Aeronautics and Space Administration (NASA) plans to spend 
over $100 billion to develop new capabilities and technologies critical 
to supporting the initial goals outlined in the President's 2004 Vision 
for Space Exploration (see fig. 1 for a conceptual drawing of NASA's 
proposed crew vehicle). The Vision, which NASA has characterized as 
bold, includes plans to explore the moon, Mars, and beyond.[Footnote 1] 
Despite many successes, such as landing the Pathfinder and Exploration 
Rovers on Mars, NASA has had difficulty carrying out a number of other 
missions, such as the X-33, a technology demonstrator for future 
reusable launch vehicles, because the agency was overly optimistic in 
what could be achieved within available resources. NASA's failure to 
define requirements adequately and quantify the resources needed to 
meet those requirements resulted in some projects costing more, taking 
longer, and achieving less than originally planned. In addition to 
these project management challenges, a constrained federal budget and a 
shrinking experienced project manager and technical workforce, well- 
versed in systems engineering and architecture design, will present 
further challenges to NASA in the years ahead. 

Figure 1: Conceptual Drawing of NASA Crew Vehicle Docked with Lunar 
Lander and Departure Stage in Earth Orbit: 

[See PDF for image] 

[End of figure] 

To meet these challenges, NASA recently updated its program and project 
management policy[Footnote 2] and is developing an agencywide systems 
engineering policy. NASA expects that these new polices will help the 
agency manage its projects with greater efficiency, responsibility, and 
accountability. We have issued a series of reports on the importance of 
obtaining critical information and knowledge at key junctures in major 
system acquisitions before additional investments are made. This work 
has shown that following a knowledge-based approach helps developers 
meet cost and schedule objectives in developing new and more 
sophisticated products--the kinds of results that NASA seeks. 

Given the major endeavor that NASA is about to undertake, the agency's 
history of cost and schedule overruns and technical problems, and the 
significant challenges the agency currently faces and will likely face, 
you asked us to (1) evaluate whether NASA's policies support an 
acquisition approach consistent with best practices identified in GAO's 
work on system acquisitions and (2) describe how NASA-wide acquisition 
policies are implemented at the various NASA centers.[Footnote 3] 

To conduct our work, we reviewed and analyzed NASA-wide program and 
project management policies as they relate to flight systems and ground 
support projects and systems engineering guidance and compared the 
policies and guidance with criteria contained in GAO's best practices 
work on systems acquisition, including space systems. We also 
interviewed NASA headquarters officials from the Office of the Chief 
Engineer responsible for the policy and guidance. In addition, we 
reviewed NASA center-specific program and project management policies 
and systems engineering policies, as they relate to flight systems and 
ground support projects, and interviewed officials responsible for 
implementing those policies. We focused primarily on Goddard Space 
Flight Center (GSFC), the Jet Propulsion Laboratory (JPL), Johnson 
Space Center (JSC), and Marshall Space Flight Center (MSFC), which 
manage the majority of NASA's flight systems and ground support 
projects. Complete details of our scope and methodology can be found in 
appendix I. We performed our work from May 2005 to November 2005 in 
accordance with generally accepted government auditing standards. 

Results in Brief: 

NASA's revised policy for developing flight systems and ground support 
projects incorporates some of the best practices used by successful 
developers, but the policy lacks certain key criteria and decision 
reviews necessary to fully support the knowledge-based acquisition 
framework. For example, NASA's policy requires projects to conduct a 
major decision review before moving from formulation to implementation. 
Further, before moving from formulation to implementation, projects 
must validate requirements and develop realistic cost and schedule 
estimates, human capital plans, a preliminary design, and a technology 
plan--all key elements for matching needs to resources before a 
commitment to major investment is made at project start. However, 
NASA's policy does not require that projects demonstrate technologies 
at high levels of maturity at that point. By not establishing a minimum 
threshold for technology maturity, NASA increases the risk that 
requirements will not be met and design changes will be required later 
in development, when such changes are typically more costly to make. In 
addition, although NASA's policy does require project managers to 
establish a continuum of technical and management reviews, the policy 
does not specify what these reviews should be, nor does it require 
major decision reviews at other key points in a product's development. 
Acquiring knowledge at key junctures will become increasingly important 
as NASA proceeds to implement various elements of the Vision. Without a 
major decision review at key milestones to ensure that the appropriate 
level of knowledge has been achieved to proceed to the next phase, NASA 
increases the risk of cost and schedule overruns as well as performance 
shortfalls. 

NASA centers have varying approaches to implementing agencywide project 
management policy and systems engineering guidance for flight systems 
and ground support projects. Some centers use criteria at key decision 
points that are similar to the criteria required to ensure a knowledge- 
based approach is followed, while others lack such criteria. As a 
result, each center reports a different level and type of knowledge 
about a project at key decision points. Centers also rely on project 
managers and systems engineers to employ good project management and 
systems engineering practices. However, given the loss of experienced 
project managers and the decline of in-house systems engineering and 
technical capabilities agencywide, that reliance could be problematic. 
These situations make it difficult for NASA decision makers to evaluate 
center projects on a common foundation of knowledge and make sound 
investment decisions and tradeoffs based on those evaluations. A 
standardized knowledge-based approach would prepare NASA to face 
competing budgetary priorities and better position the agency to make 
difficult decisions regarding the investment in and termination of 
projects. 

To ensure NASA uses a knowledge-based acquisition approach in making 
informed investment decisions, we are making several recommendations to 
improve the agency's policies. We are recommending that NASA (1) 
require the capture of specific knowledge to be used as criteria for 
allowing projects to enter implementation and proceed through 
development and to support informed investment decisions and (2) 
institute additional reviews for flight system and ground support 
projects during project implementation, which result in recommendations 
to the appropriate decision authority. In written comments on a draft 
of this report, NASA agreed with our recommendations. NASA's comments 
are included in their entirety in appendix III. 

Background: 

Over the past decade, NASA has experienced significant problems with 
several of its projects, which GAO and others have reported on (see 
table 1 for some examples of such problems). In addition, in 2002 we 
reported on several sources of failures in NASA programs, including 
underestimating complexity and technology maturity, and inadequate 
review and systems engineering processes. Further, we reported that the 
sources of these problems were not new and that NASA failed to 
consistently apply lessons previously learned.[Footnote 4] The failures 
identified in our 2002 report were in part the result of the "faster, 
better, cheaper" approach to managing its major acquisitions, which 
NASA adopted in the 1990s.[Footnote 5] 

Table 1: Examples of Problems With NASA Programs/Projects: 

X-33: 

Program/Project description: The X-33 was to be an unmanned technology 
demonstrator, which tested a range of technologies needed for future 
reusable launch vehicles; 
Reported problems: In 1999, GAO reported that technical problems on the 
X-33 project led to cost overruns of $75 million dollars and over a 
year's delay in schedule. These problems resulted from NASA developing 
unrealistic cost estimates in the early stages of the X-33 project and 
not following its own policies with regard to project and risk 
management. In February 2001, NASA announced it would provide no 
additional funding for the X-33 program. 

Space Launch Initiative (SLI): 

Program/Project description: Approved in February 2001, SLI was 
designed as a $4.8 billion, 6-year effort to fly a half-scale flight 
demonstrator intended to validate advanced technologies that would 
dramatically reduce the cost of putting a pound of payload into space-
-from $10,000 to $1,000; 
Reported problems: In 2002, both the NASA Inspector General and GAO 
released reports assessing SLI with particular emphasis on requirement 
definition and implementation of management controls; The NASA 
Inspector General found NASA did not verify and validate the basic 
requirements for its second-generation space transportation system. GAO 
found NASA could not implement key management controls until it defined 
the SLI's basic requirements. In October 2002, in response to the GAO 
report, NASA indefinitely postponed the System Readiness Review for 
SLI. 

Comet Nucleus Tour (COUNTOUR): 

Program/Project description: NASA's CONTOUR project was intended to 
visit at least two comets to provide the first detailed look at the 
differences between primitive building blocks of the solar system and 
answer questions about how comets act and evolve; Reported problems: A 
May 31, 2003 report released by a NASA Mishap Investigation Board (MIB) 
found that inadequate systems engineering processes and an inadequate 
review function were root causes of the December 2002 loss of the $159 
million NASA COUNTOUR spacecraft. Further, based on the findings 
related to the CONTOUR project, the MIB board recommended that NASA 
establish clear standards for conducting and documenting engineering 
work and associated peer and independent reviews and reevaluate its 
oversight and review requirements. 

Prometheus 1: 

Program/Project description: The Prometheus 1 project is part of NASA's 
Prometheus Nuclear Systems and Technology project to develop nuclear 
power technologies capable of providing power and propulsion for a new 
generation of missions. It is being designed to use nuclear power and 
electric propulsion technologies to explore the outer reaches of the 
solar system. The Jupiter Icy Moons Orbiter mission was to be a 4-to 6- 
year study of three of Jupiter's moons; 
Reported problems: In February 2005, GAO issued a report on NASA's 
Prometheus I project which found that the agency faced challenges 
preparing preliminary requirements and cost estimates and that the 
critical technologies supporting the project would require extensive 
advancement before they could satisfy requirements. According to 
Prometheus 1 project management, the approved funding profile for the 
project was inadequate to support the planned mission, a 2015 launch to 
Jupiter's Icy Moons. GAO recommended that NASA identify the level of 
resources the agency is committing to the project and direct project 
officials to develop requirements based on this resource constraint. 
NASA has since deferred the Jupiter Icy Moons Orbiter mission, citing 
concerns over cost and technical complexity. 

Source: GAO and NASA. 

[End of table] 

These problems, along with others, highlighted the need for the agency 
to reevaluate its approach to cost and schedule estimating, risk 
assessments, technology development, project reviews, and systems 
engineering. In 1998, NASA adopted a new program and project management 
policy, which was revised in 2002. The policy provided significant 
flexibility, including allowing tailoring and projects to opt out of 
requirements at the discretion of the project manager. In March 2005-- 
following a series of internal and external assessments of NASA that 
showed that the agency faced significant problems with project 
management--NASA again revised its program and project management 
policy, which includes policy requirements for the development of 
flight systems and ground support projects.[Footnote 6] According to 
NASA officials, further changes to the policy are anticipated in light 
of new agency leadership.[Footnote 7] 

The March 2005 policy document, NASA Procedural Requirement (NPR) 
7120.5C, differs from previous versions of the policy in that it 
delineates requirements based on four NASA investment areas: Basic and 
Applied Research, Advanced Technology Development, Flight Systems and 
Ground Support, and Institutional Infrastructure. The policy also 
reinstitutes a life cycle phased approach to product development and 
institutes a project categorization scheme, based on project cost and 
priority, which denotes the oversight authorities[Footnote 8] and the 
level of detail that is needed to support project planning documents. 
See figure 2 below. 

Figure 2: Project Categorization Scheme and Oversight Authorities from 
NPR 7120.5C: 

[See PDF for image] 

[End of figure] 

NASA has also attempted to address some of its cost-estimating 
weaknesses by instituting a cost analysis data requirement (CADRe) and 
establishing thresholds for the use of earned value management 
(EVM).[Footnote 9] Another major change to the policy is the 
establishment of the Independent Technical Authority (ITA). According 
to the policy, the purpose of ITA is to establish sound technical 
requirements and decisions for safe and reliable system operations 
separate from the project management and reporting chain. Finally, 
agency officials told us that while the requirements in previous 
versions of the policy were easy to tailor, projects now must document 
compliance with the requirements in a "compliance matrix" and must 
request and have approved any deviations and/or waivers to 
requirements. 

Although NPR 7120.5C contains some mandatory requirements with regard 
to systems engineering, according to agency officials, NASA has never 
had an agencywide systems engineering policy to inform the development 
of flight systems and ground support projects. Since 1995, in the 
absence of a policy on systems engineering, project managers and 
systems engineers have relied on information contained in NASA's 
Systems Engineering Handbook to guide their systems engineering 
approach on projects.[Footnote 10] Project managers and systems 
engineers, however, are not required to follow the handbook. 
Recognizing the need for a more structured and rigorous approach to 
systems engineering agencywide, NASA's Office of the Chief Engineer is 
currently leading the effort to develop a systems engineering policy. 
The policy is in draft form. 

Knowledge-Based Acquisition Framework: 

Over the last several years, we have undertaken a body of work on how 
leading developers in industry and government use a knowledge-based 
approach to deliver high quality products on time and within 
budget.[Footnote 11] A knowledge-based approach to product development 
efforts enables developers to be reasonably certain, at critical 
junctures or "knowledge points" in the acquisition life cycle, that 
their products are more likely to meet established cost, schedule, and 
performance baselines and, therefore provides them with information 
needed to make sound investment decisions. See figure 3 for a depiction 
of a knowledge-based acquisition life cycle. 

Figure 3: Knowledge-Based Acquisition Life Cycle: 

[See PDF for image] 

[End of figure] 

* Knowledge point 1 (KP1): Resources and needs match. Knowledge point 1 
occurs when a sound business case is made for the product--that is, a 
match is made between the customer's requirements and the product 
developer's available resources in terms of knowledge, time, workforce, 
and money. To determine available resources, successful developers rely 
on current and valid information from predecessor projects, new 
technologies that have demonstrated a high level of maturity, system 
engineering data, and experienced people. Successful developers also 
communicate extensively with customers to match their wants and needs 
with available resources and with the developers ability to manufacture 
an appropriate product. 

* Knowledge point 2 (KP2): Product design is stable. Knowledge point 2 
occurs when a developer determines that a product's design is stable-- 
that is, it will meet customer requirements and cost and schedule 
targets. A best practice is to achieve design stability at the 
product's critical design review (CDR), usually held midway through 
development. Completion of at least 90 percent of engineering drawings 
at the CDR provides tangible evidence to decision makers that the 
design is stable. 

* Knowledge point 3 (KP3): Production processes are mature. This level 
of knowledge is achieved when it has been demonstrated that the product 
can be manufactured within cost, schedule, and quality targets. A best 
practice is to ensure that all key manufacturing processes are in 
statistical control--that is, they are repeatable, sustainable, and 
capable of consistently producing parts within the product's quality 
tolerances and standards--at the start of production. It is important 
that the product's reliability be demonstrated before production 
begins, as investments can increase significantly if defective parts 
need to be repaired or reworked. 

If the knowledge attained at each juncture does not confirm the 
business case on which the initial investment was originally justified, 
the project should not go forward and additional resources should not 
be committed. Product development efforts that do not follow a 
knowledge-based approach can be frequently characterized by poor cost, 
schedule, and performance outcomes. 

NASA's Revised Policy Does Not Fully Support a Knowledge-Based Approach 
to Acquisitions: 

NASA's revised acquisition policy for developing flight and ground 
support systems incorporates some of the elements of a knowledge-based 
acquisition approach. However, it lacks specific key criteria and 
decision reviews necessary to fully support such an approach. NASA's 
policy defines a phased life cycle approach and requires a major 
decision review to move from project formulation to implementation. The 
policy requirements for this review address many of the key elements 
necessary to match needs to resources, such as requirements to 
establish project baselines. However, the policy does not require that 
projects demonstrate technologies at high levels of maturity before 
launching a project and investing a large amount of resources. In 
addition, NASA's policy does not require any further major decision 
reviews following the formulation phase of the project. Major decision 
reviews in the implementation phase based on specific evaluation 
criteria at the final design and fabrication, assembly, and testing 
milestones--critical decision points in any product development--could 
help NASA ensure that sufficient knowledge has been gained to warrant 
moving forward in the development process. 

NASA's Policy Establishes a Phased Life Cycle: 

The life cycle for all NASA projects is divided into two major phases-
-formulation and implementation. Because flight systems and ground 
support projects are particularly complex and have long life cycles, 
NASA has further divided the formulation and implementation phases for 
these projects to allow managers to assess management and engineering 
progress (see fig. 4). 

Flight systems and ground support projects must successfully complete 
two major decision reviews: a Preliminary-Non Advocate Review (Pre-NAR) 
between Phases A and B and a Non-Advocate Review (NAR) between Phases B 
and C. At these reviews, the Governing Program Management Committee 
(GPMC) evaluates the cost, schedule, safety, and technical content of 
the project to ensure that the project is meeting commitments specified 
in key management documents.[Footnote 12] Following each of these 
reviews, the GPMC recommends to the appropriate decision 
authority[Footnote 13] whether the project should be authorized to 
proceed.[Footnote 14] 

Figure 4: Life Cycle Phases for NASA's Flight Systems and Ground 
Support Projects: 

[See PDF for image] 

[End of figure] 

As with KP1 in a knowledge-based acquisition life cycle, the NAR marks 
the official project approval point in the life cycle. After approval 
at the NAR, projects are included as part of implementation reviews of 
their parent program. These implementation reviews are conducted 
biennially and are not tied to any design or production milestones. 
After entering the implementation phase, the GPMC is notified if cost 
or schedule performance exceeds the baselines established at the NAR by 
10 percent or when key performance criteria are not met. Exceeding the 
NAR baselines can result in the GPMC conducting a termination review to 
determine whether or not to continue the project. 

NASA's Policy Lacks Requirements for Maturing Technologies: 

To help ensure project requirements do not outstrip resources, leading 
developers obtain the right knowledge about a new product's technology, 
design, and production at the right time. NASA's policy emphasizes many 
elements needed at the NAR (or KP1) to match needs to resources, such 
as validating requirements, developing realistic cost and schedule 
estimates and human capital plans, and establishing a preliminary 
design. The policy does not, however, require projects to demonstrate 
technologies at high levels of maturity before launching a project. 
Table 2 compares KP1 criteria and NASA's policy criteria. 

Table 2: KP1 Criteria Compared to NASA Policy Criteria: 

KP1 Criteria (resources and needs match): High level of technology 
maturity; NPR 7120.5C NAR requirements: A technology plan describing 
the technology needed for the project, including plans for technology 
maturation, validation, and insertion, and quantifiable milestones, 
decision gates, and resources required. 

KP1 Criteria (resources and needs match): Informed requirements; 
NPR 7120.5C NAR requirements: A set of requirements that are well 
formed (clear and unambiguous), complete (agrees with customer and 
stakeholder expectations), and consistent (conflict free); and each 
requirement is verifiable and traceable to higher level requirements. 

KP1 Criteria (resources and needs match): Realistic cost and schedule 
estimates; 
NPR 7120.5C NAR requirements: A cost analysis data requirement 
(CADRe)[A] documents the programmatic, technical, and life cycle cost 
information. The CADRe includes a life cycle cost estimate, integrated 
master project schedule, and a work breakdown structure (WBS).[B]  

KP1 Criteria (resources and needs match): Human capital in place; 
NPR 7120.5C NAR requirements: Plans to staff the project with personnel 
with the appropriate skills, abilities, and experience, and provide 
integrated team training to successfully execute the project. The 
project manager is required to negotiate to obtain the needed resources 
identified in the plan. 

KP1 Criteria (resources and needs match): Preliminary system design; 
NPR 7120.5C NAR requirements: A preliminary system design is required 
and is reviewed at the NAR. 

KP1 Criteria (resources and needs match): Conduct a decision review 
before project launch; 
NPR 7120.5C NAR requirements: The GPMC determines the project's 
readiness to proceed to implementation and recommends a course of 
action to the appropriate decision authority. 

Source: NASA and GAO. 

[A] A CADRe is required for Category I and Category II flight system 
and ground support projects. 

[B] A WBS is a product-oriented division of hardware, software, 
services, and project unique tasks that organizes and defines the 
product to be developed and serves as the basis for estimating both 
cost and schedule. 

[End of table] 

While NASA requires projects to develop plans that describe how 
technologies will be matured and to provide alternative development 
strategies for technologies that do not mature as expected, it does not 
establish a minimum threshold for technology maturity. Consequently, 
projects can enter the implementation phase with immature technologies 
and embark on a risky path of having to build technology, design, and 
production knowledge concurrently. Our best practices work has shown 
that maturing technologies during the preliminary design phase and 
before entering product development is a key element of matching needs 
to resources and that there is a direct relationship between the 
maturity of technologies and the risk of cost and schedule 
growth.[Footnote 15] Allowing technology development to carry over into 
the product development phase increases the risk that significant 
problems will be discovered late in development. Addressing such 
problems at this stage may require extensive retrofitting and redesign 
as well as retesting, which can jeopardize performance and result in 
more time and money to fix. This approach also makes it more difficult 
for projects to demonstrate the same level of design stability in later 
phases of implementation since technology and design activities will be 
done concurrently. 

Technology readiness levels (TRL)--a concept developed by NASA--can be 
used to gauge the maturity of individual technologies. The higher the 
TRL, the more the technology has been proven and the lower the risk of 
performance problems and cost and schedule overruns (see fig. 5). TRL 
7--demonstrating a technology as a fully integrated prototype in an 
operational environment--is the level of maturity preferred by product 
developers to minimize risks when entering product development. 

Figure 5: Technology Maturity Levels for Product Development: 

[See PDF for image] 

[End of figure] 

Successful developers will not commit to undertaking product 
development, and more importantly investing resources, unless they have 
high confidence that they have achieved a match between what the 
customer wants and what the project can deliver. Technologies that are 
not mature continue to be developed in an environment that is focused 
solely on technology development. Once matured, these technologies can 
be transitioned to projects. This puts developers in a better position 
to succeed because they can focus on integrating the technologies and 
testing and proving the product design. 

NASA's Policy Does Not Support Informed Design and Production 
Decisions: 

Our prior work has shown that successful developers establish specific 
criteria to ensure that requisite knowledge has been attained before 
moving forward from final design into the latter stages of 
development.[Footnote 16] Before making significant increases in 
investments to fabricate, assemble, and test the product, these 
developers conduct a decision review to determine if the design is 
stable and performs as expected and the project is ready to enter the 
next phase. To make this determination and reach KP2, successful 
developers use specific, knowledge-based standards and criteria. (See 
app. II for more information on the specific knowledge-based standards 
and criteria successful developers use to judge readiness to proceed 
beyond detailed design activities.) 

Successful developers also demand proof that manufacturing processes 
are in control and product reliability goals are attained before 
committing to production. To determine whether they have achieved this 
knowledge point, KP3, successful developers conduct another mandatory 
decision review in which they use specific, knowledge-based standards 
and criteria to determine if the product can be produced within cost, 
schedule, and quality targets. (See app. II for more information on 
specific knowledge-based standards and criteria used by successful 
developers to judge readiness to enter into production.) 

Contrary to these best practices, the NAR at the end of the preliminary 
design phase--KP1--is the last major decision review in the NASA 
project life cycle. (See fig. 6.) Although NPR 7120.5C requires that 
projects document in the project plan a continuum of technical and 
management reviews, such as a PDR and CDR, it does not require any 
specific reviews.[Footnote 17] In addition, NASA's March 2005 policy 
does not require a NAR-type decision review to ensure a project has 
obtained the knowledge needed to proceed beyond the final design phase 
into the fabrication, assembly, and test phase, which serves as both 
the demonstration and production phase of the NASA life cycle.[Footnote 
18] According to NASA officials, projects conduct a CDR at the end of 
the final design phase to ensure adequate information is available 
about product design and producibility before entering the fabrication, 
assembly, and test phase. The CDR, however, is a technical review--not 
a major decision review like the NAR. Furthermore, the policy does not 
establish criteria as to what constitutes successful completion of a 
CDR. 

Figure 6: Comparison of NASA's Life Cycle with a Knowledge-Based 
Acquisition Life Cycle: 

[See PDF for image] 

[End of figure] 

NASA's policy also does not require a major decision review before 
beginning manufacturing. (See fig. 6 above.) Therefore, the transition 
from final design to fabrication, assembly, and test often marks a de 
facto production decision. According to NASA officials, the agency 
rarely enters a formal production phase due to the small quantities of 
space systems that they build. However, due to the high cost of failure 
associated with NASA projects and the costs and risks involved in 
repairing a system in-orbit, a major decision review at production that 
assesses product reliability is essential even for these limited 
production systems. In addition, although NASA's production quantities 
are typically low, in some instances NASA does produce larger 
quantities of a system or subsystem, such as the external tanks for the 
Space Shuttle. Furthermore, NASA's plans indicate that the agency may 
be increasing production for elements of their future systems. For 
example, NASA's Exploration Systems Architecture Study indicates that 
NASA plans to build several new crew exploration vehicles, with 
disposable elements, such as the lunar lander, solid rocket boosters, 
and space shuttle main engines that will require higher numbers of 
production runs. 

Rather than establish specific criteria by which all projects are 
judged, NASA's policy requires that projects manage to baselines and 
plans established in key management documents and approved at the 
project NAR. The baselines and plans serve as their primary tools for 
measuring project progress and as the primary basis for judgment at 
project reviews. While the plans may include some information that 
addresses knowledge-based criteria for design and production, the 
instructions for preparing them leaves the establishment of thresholds 
and success criteria to the discretion of the project manager. For 
example, NASA policy requires that projects include, as part of the 
project plan, a Verification and Validation Sub Plan that describes the 
project's approach to verifying and validating hardware and software as 
part of the project plan. The policy, however, includes no instruction 
as to what constitutes a sufficient approach to testing. In other 
words, there are no requirements concerning the fidelity of test 
articles or the realism of the test environment. Similarly, NASA's 
policy requires that projects include, as part of the project plan, a 
Systems Engineering Sub Plan that describes the project's approach to 
systems engineering and the technical standards that are applicable, 
including metrics that verify the processes. The policy, however, does 
not identify the types of metrics appropriate to verify the process or 
establish any threshold criteria. 

The absence of major decision reviews, along with specific criteria in 
the fabrication, assembly, and test phase, could result in concurrent 
design and manufacturing activities, a practice our past work has found 
increases risk in acquisition programs.[Footnote 19] Furthermore, 
lacking a major decision review to ensure that projects have gained the 
appropriate levels of knowledge at KP2 and KP3, NASA decision makers 
cannot be provided a high level of certainty that the project will meet 
cost, schedule, and performance requirements and have no assurance that 
the provisions of the key management documents required by NPR 7120.5C 
are being executed after the NAR. 

Lack of NASA-Wide Project Management Criteria May Result in Investment 
Decisions Based on Inconsistent Information: 

NASA centers have varying approaches to implementing project management 
policies and systems engineering guidance for flight systems and ground 
support projects. Some centers use criteria at key decision points that 
are similar to the criteria required to ensure a knowledge-based 
approach is followed, while others lack such criteria. As a result, 
each center reports a different level and type of knowledge about a 
project at key decision points. Centers also rely on project managers 
and systems engineers to employ good project management and systems 
engineering practices. However, given the loss of experienced project 
managers and the decline of in-house systems engineering and technical 
capabilities agencywide, that reliance could be problematic. These 
situations make it difficult for NASA decision makers to evaluate 
center projects on a common foundation of knowledge and make sound 
investment decisions and tradeoffs based on those evaluations. A 
standardized, knowledge-based approach would prepare NASA to face 
competing budgetary priorities and make difficult decisions regarding 
the investment in and termination of projects. 

Individual Centers Tailor Implementation of Agencywide Project 
Management Policies and Systems Engineering Guidance: 

While NASA centers are given discretion about how they implement 
agencywide policies, they are expected to have procedures and 
guidelines in place for implementing those policies. Some centers have 
developed center-specific policies and criteria for implementing NASA's 
project management policies and system engineering guidance, while 
others have not. Centers also rely on project managers and systems 
engineers to implement the requirements of NPR 7120.5C and to use 
NASA's Systems Engineering Handbook as guidance for good systems 
engineering practices.[Footnote 20] Some NASA centers have also 
developed criteria in their policies that are similar to the criteria 
used to ensure a knowledge-based approach is followed; other centers 
lack such criteria. 

Because of their varying policies and criteria, each center requires a 
different level of knowledge at the same point in a project's 
development cycle. For example, GSFC requires its projects to mature 
technologies to TRL 6 by the preliminary design review--before entering 
the implementation phase. On the other hand, JPL, MSFC, and JSC 
policies do not require projects to mature technology to a particular 
level before entering implementation, leaving the determination of 
needed technology maturity up to the project manager. 

Requirements for assessing design maturity also vary across the 
centers. While most of the center policies require a CDR to enter 
"Phase D--Fabrication, Assembly, and Test"--the criteria used to assess 
projects at this point vary. For example, both GSFC and MSFC require 
projects to have completed a percentage of design drawings at this 
review. MSFC requires that 90 percent of design drawings to be complete 
by CDR, which is consistent with best practices. While GSFC establishes 
a minimum threshold of drawings to be complete by CDR--greater than 80 
percent--neither JSC nor JPL establish a minimum percentage drawing 
requirement. Instead, JSC requires the design be complete and drawings 
ready to begin production, and JPL requires that the design be mature 
and provide confidence in the integrity of the flight system design. 

Almost none of the center policies include a requirement to assess the 
maturity of production processes. According to NASA officials, due to 
the low quantities of systems generally produced by the agency, most of 
the center policies do not require a review before beginning 
manufacturing. Only JSC requires a Production Readiness Review to 
ensure that production plans, facilities, and personnel are in place 
and ready to begin production. However, the criteria do not specify 
quantifiable thresholds to measure production readiness at the review. 
JPL, MSFC, and GSFC do not have policies that outline requirements for 
such a review. 

In addition to individual policy requirements, centers may rely on 
project managers and systems engineers to employ good project 
management and systems engineering practices. For example, an 
experienced project manager at JPL told us that although JPL policy 
does not require a particular TRL at PDR to enter implementation, he 
required that all technologies for his project be around a TRL 6 in 
order to separate technology development from systems development. 
Reliance on project managers to implement good practices, however, 
could be problematic given the diminishing number of experienced 
project managers available to lead projects and the decline of in-house 
systems engineering and technical capabilities agencywide caused by 
increasing retirements and outsourcing. 

Informed Investment Decisions Require Consistent Knowledge: 

In a knowledge-based process, the achievement of each successive 
knowledge point builds on the preceding one, giving decision makers the 
information they need, when they need it, to make decisions about 
whether to invest significant additional funds to move forward with 
product development. Our work has shown that successful product 
development efforts are marked by adherence to a disciplined process 
that establishes and uses common and consistent criteria for decision 
making at these key points. With varying project management and systems 
engineering criteria, NASA centers' technical reviews, such as PDR, 
provide different levels of knowledge to support NASA's major decision 
reviews, such as the NAR, which NASA uses to support its major 
investment decisions for flight systems and ground support projects. 

In the near future, NASA will need to determine the resources necessary 
to develop the systems and supporting technologies to achieve the 
President's Vision for Space Exploration and structure its investment 
strategy accordingly. Initial implementation of the Vision as explained 
in NASA's Exploration Systems Architecture Study calls for completing 
the International Space Station, developing a new crew exploration 
vehicle, and returning to the moon no later than 2020. NASA estimates 
that it will cost approximately $104 billion over the next 13 years to 
accomplish these initial goals. These priorities, along with NASA's 
other missions, will be competing within NASA for funding. It will 
likely be difficult for NASA managers to agree on which projects to 
invest in and which projects to terminate. The NASA Administrator has 
acknowledged that NASA faces difficult choices about its missions in 
the future--for example, between human space flight, science, and 
aeronautics missions. 

Using consistent criteria to evaluate all NASA projects would help 
ensure that the same level and type of knowledge is available about 
individual projects at key decision points. Analogous information about 
all flight systems and ground support projects would allow decision 
makers to make apples-to-apples comparisons across projects and make 
investment decisions, and trade-offs, based upon these comparisons. 
Further, policies with consistent criteria can provide inexperienced 
project managers and systems engineers with the necessary guidance to 
implement good project management and systems engineering practices and 
ensure that the right knowledge is available for decision makers. 

NASA officials within the Chief Engineer's Office acknowledged the need 
for more consistency in criteria across the various NASA centers in 
order to successfully achieve NASA's Vision for Space Exploration. Some 
NASA centers, however, are resistant to standardized criteria because 
they feel it could be overly prescriptive. These centers indicated that 
because of the unique nature of the work done at each of the 10 NASA 
centers, it would be unrealistic to hold every project to the same 
criteria. Nonetheless, our work has shown that the process used by 
successful product developers to develop leading-edge technology and 
products does not differ based upon the type of product being 
developed. Further, these developers adhere to a disciplined process 
that establishes and uses common and consistent criteria for decision 
making--regardless of the type of product or technology being 
developed. Using consistent criteria can allow NASA decision makers to 
assess the likely return on competing investment priorities and to 
reevaluate alternatives and make investment decisions across projects 
to increase the likelihood of attaining the strategic goals of the 
agency. 

Conclusions: 

Several of NASA's major acquisitions have been marked by cost, 
schedule, and performance problems. Yet the challenges NASA faces in 
the future are likely to far exceed those it has faced in the past. The 
complex technical requirements associated with fulfilling the 
President's Vision, the fiscal constraints under which NASA will be 
required to operate, the diminished number of experienced project 
managers and systems engineers, and the potential for increased 
production make following a knowledge-based approach for flight systems 
and ground support projects all the more critical. While NASA has made 
improvements to its policies governing project management, the lack of 
major decision reviews beyond the initial project approval gate leaves 
decision makers with little knowledge about the progress of the 
agency's projects. Further, without a standard set of criteria to 
measure projects at crucial phases in the development life cycle, NASA 
cannot be assured that its decisions will result in the best possible 
return on its investments. Since NASA is currently in the process of 
revising its program and project management policies and developing an 
agencywide systems engineering policy, we believe this presents a 
unique opportunity for the agency to correct some of the problems 
identified during our review. 

Recommendations for Executive Action: 

In order to close the gaps between NASA's current acquisition 
environment and best practices on knowledge-based acquisition, we 
recommend that NASA take steps to ensure NASA projects follow a 
knowledge-based approach for product development. Specifically, we 
recommend that the NASA Administrator direct the Office of the Chief 
Engineer to take the following two actions: 

* In drafting its systems engineering policy, incorporate requirements 
in the policy for flight systems and ground support projects to capture 
specific product knowledge by key junctures in project development. The 
demonstration of this knowledge should be used as exit criteria for 
decision making at the following key junctures: 

* Before projects are approved to transition from formulation to 
implementation, the policy should require that projects demonstrate 
that key technologies have reached a high maturity level. 

* Before projects are approved to transition from final design to 
fabrication, assembly, and test, the policy should require that 
projects demonstrate that the design is stable. 

* Before projects are approved to transition into production, the 
policy should require projects to demonstrate that the design can be 
manufactured within cost, schedule, and quality targets. 

* Revise NPR 7120.5C to institute additional major decision reviews 
following the NAR for flight systems and ground support projects, which 
result in recommendations to the appropriate decision authority. These 
reviews should be tied to the key junctures during project development 
mentioned above in order to increase the likelihood that cost, 
schedule, and performance requirements of the project will be met. 

Agency Comments and Our Evaluation: 

We provided a draft of this report to NASA for review and comment. In 
written comments, NASA indicated that it agreed with our 
recommendations and outlined specific actions that the agency plans to 
take to address them. 

The actions that NASA plans to take to address our recommendations are 
a positive step toward achieving successful project outcomes and 
ensuring that decision makers are appropriately investing the agency's 
resources. We are pleased to hear that many knowledge-based practices 
identified in our report are currently being practiced agencywide in 
the management and development NASA systems. The addition of such 
practices to NASA's policies will only strengthen their use agencywide 
and ensure that these practices continue to be utilized by less 
experienced project managers and systems engineers as the more 
experienced workforce retires. The effectiveness of such practices, 
however, will be limited if project officials are not held accountable 
for demonstrating a high level of knowledge, consistent with the 
success criteria that NASA plans to require in its policies, at key 
junctures in development. It is critical that project officials not 
only have a high level of knowledge about a project at key junctures, 
but also that this information is used by decision makers to make 
decisions on whether to invest additional resources and allow a project 
to proceed through the development life cycle. 

NASA's comments are reprinted in appendix III. NASA also provided 
technical comments, which we addressed throughout the report as 
appropriate. 

As agreed with your offices, unless you announce its contents earlier, 
we will not distribute this report further until 30 days from its date. 
At that time, we will send copies to NASA's Administrator and 
interested congressional committees. We will make copies available to 
others upon request. In addition, the report will be available at no 
charge on the GAO Web site at http://www.gao.gov. 

If your or your staff have any questions concerning this report, please 
contact me at (202) 512-4841 or lia@gao.gov. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this report. Key contributors to this report are 
acknowledged in appendix IV. 

Signed by: 

Allen Li: 
Director: 
Acquisition and Sourcing Management: 

[End of section] 

Appendix I: Scope and Methodology: 

To determine the extent to which NASA's current policies support an 
acquisition approach consistent with best practices identified in GAO's 
work on system acquisitions, we reviewed and analyzed NASA-wide program 
and project management policies and systems engineering guidance. Our 
review and analysis of the policy focused on requirements for flight 
systems and ground support projects. We interviewed NASA Headquarters 
officials from the Office of the Chief Engineer who are responsible for 
the policy and guidance. We compared NASA's policy on program and 
project management with criteria contained in GAO best practices work 
on systems acquisition and space system acquisitions. We concentrated 
on whether the policy provides a framework for a knowledge-based 
process and the criteria necessary to carry out this intent. 

To determine how NASA-wide acquisition policies are implemented across 
the various NASA centers, we reviewed NASA center-specific program and 
project management policies and systems engineering policies and 
interviewed officials responsible for implementing those policies. 
While we interviewed officials from all centers, we focused on the 
centers that manage the majority of NASA's flight systems and ground 
support projects--GSFC, JPL, JSC, and MSFC. This approach included site 
visits with GSFC, JPL, JSC, and MSFC and teleconferences with the 
remaining centers. We compared examples of the centers' implementation 
of the policies and specific criteria included in these policies with 
our best practices work on systems acquisition. 

We performed our work from May 2005 to November 2005 in accordance with 
generally accepted government auditing standards. 

[End of section] 

Appendix II: Activities That Enable the Capture of Design and 
Manufacturing Knowledge: 

Table 3: Activities to Capture Design Knowledge and Make Decisions: 

Knowledge: Design is stable and performs as expected (knowledge point 
2); 
Decision: Product is ready for initial manufacturing and system 
demonstration phase; 
Key indicator: 90 percent of engineering drawings completed. 

Activities to achieve stable design knowledge; 
* Limit design challenge--The initial design challenge is limited to a 
product that can be developed and delivered quickly and provide the 
user with an improved capability. A time-phased plan is used to develop 
improved products--future generations--in increments as technologies 
and other resources become available; 
* Demonstrate design meets requirements-- The product's design is 
demonstrated to meet the user's requirements. For a new product that is 
not based on an existing product, prototypes are built and tested. If 
the product is a variant of an existing product, companies often used 
modeling and simulation or prototypes at the component or subsystem 
level to demonstrate the new product's design; 
* Complete critical design reviews--Critical design reviews are used to 
assess whether a product's design meets requirements and is ready to 
start initial manufacturing. Reviews are conducted for the system, 
subsystems, and components to assess design maturity and technical 
risk; 
* Stakeholders agree drawings complete and producible-
-The agreement by stakeholders (engineers, manufacturers, and other 
organizations) is used to signify confidence that the design will work 
and the product can be built; 
* Executive level review to begin initial manufacturing--Corporate 
stakeholders meet and review relevant product knowledge, including 
design stability, to determine whether a product is ready to initiate 
manufacturing of production representative prototypes used during 
system demonstrations. The decision is tied to the capture of 
knowledge. 

[End of table] 

Source: GAO. 

Table 4: Activities to Capture Manufacturing Knowledge and Make 
Decisions: 

Knowledge: Product can be produced within cost, schedule, and quality 
targets (knowledge point 3); 
Decision: Product is ready for production and will be reliable; 
Key indicator: Critical processes in statistical control and product 
reliability demonstrated. 

Activities to Achieve Manufacturing Knowledge; 
* Identify key system characteristics and critical manufacturing 
processes--Key product characteristics and critical manufacturing 
processes are identified. Because there can be thousands of 
manufacturing processes required to build a product, companies focus on 
the critical processes--those that build parts that influence the 
product's key characteristics such as performance, service life, or 
manufacturability; 
* Determine processes in control and capable--Statistical process 
control is used to determine if the processes are consistently 
producing parts. Once control is established, an assessment is made to 
measure the process's ability to build a part within specification 
limits as well as how close the part is to that specification. A 
process is considered capable when it has a defect rate of less than 1 
out of every 15,152 parts produced; 
* Conduct failure modes and effects analysis--Bottom- up analysis is 
done to identify potential failures for product reliability. It begins 
at the lowest level of the product design and continues to each higher 
tier of the product until the entire product has been analyzed. It 
allows early design changes to correct potential problems before 
fabricating hardware; 
* Set reliability growth plan and goals--A product's reliability is its 
ability to perform over an expected period of time without failure, 
degradation, or need of repair. A growth plan is developed to mature 
the product's reliability over time through reliability growth testing 
so that it has been demonstrated by the time production begins; 
* Conduct reliability growth testing--Reliability growth is the result 
of an iterative design, build, test, analyze, and fix process for a 
product's design with the aim of improving the product's reliability 
over time. Design flaws are uncovered and the design of the product is 
matured; 
* Conduct executive level review to begin production--Corporate 
stakeholders meet and review relevant product knowledge, including 
manufacturing and reliability knowledge, to determine whether a product 
is ready to begin production. The decision is tied to the capture of 
knowledge. 

Source: GAO. 

[End of table] 

[End of section] 

Appendix III: Comments from the National Aeronautics and Space 
Administration: 

National Aeronautics and Space Administration: 
Office of the Administrator: 
Washington, DC 20546-0001: 

December 15, 2005: 

Mr. Allen Li: 
Director, Acquisition and Sourcing Management: 
United States Government Accountability Office: 
Washington, DC 20548: 

Dear Mr. Li: 

NASA appreciates the opportunity to comment on your draft report 
(General Accountability Office (GAO) 06-218) entitled "Implementing a 
Knowledge-Based Acquisition Framework Could Lead to Better Investment 
Decisions and Project Outcomes." 

Per the National Aeronautics and Space Act (Pub. L. No 85-568), NASA's 
charter is "lo provide for research into problems of flight within and 
outside the Earth's atmosphere, and for other purposes." NASA is a 
research and development organization, and its projects face greater 
uncertainty further into the life cycle than traditional high-volume 
acquisitions. Most NASA missions fabricate one-of-a-kind spacecraft 
using a proto-flight or very small quantity approach. 

Projects are primarily governed by two NASA Procedural Requirements 
(NPRs). NPR 7120.5C, NASA Program and Project Management Processes and 
Requirements, establishes the management system which governs the 
formulation, approval, implementation, and evaluation of NASA programs 
and projects. Requirements for performing, supporting, and evaluating a 
systems approach applied to all elements and levels of a system over 
the complete project life cycle are established for the implementing 
organizations in NPR 7123, NASA Systems Engineering Processes and 
Requirements. NPR 7123 is in the final approval cycle. 

Many of the best practices you recommend are currently being practiced 
Agencywide but may not be apparent in our current policy. Issuance of 
NPR 7123 will establish a consistent minimum set of project reviews 
that are already practiced within NASA (i.e., Systems Requirement 
Review (SRR), Preliminary Design Review (PDR), Critical Design Review 
(CDR)). While NPR 7120.5C does not depict decision reviews at key 
junctures beyond the Non-Advocate Review (NAR), NASA gains continuous 
knowledge throughout the implementation phase. Monthly or quarterly 
status reviews are typically held utilizing the hierarchy of governing 
program management councils. This process provides for decision points 
throughout the life cycle. 

The following paragraphs provide the GAO recommendations and the NASA 
response addressing these recommendations: 

First GAO Recommendation: 

In order to close the gaps between NASA's current acquisition 
environment and best practices on knowledge-based acquisition, we 
recommend that NASA take steps to ensure NASA projects follow a 
knowledge-based approach for product development. Specifically, we 
recommend that the NASA Administrator direct the Office of the Chief 
Engineer to take the following two actions: 

* In drafting its systems engineering policy, incorporate requirements 
in the policy for flight systems and ground support projects to capture 
specific product knowledge by key junctures in project development. The 
demonstration of this knowledge should be used as exit criteria for 
decision making at the following key junctures: 

- Before projects are approved to transition from formulation to 
implementation, the policy should require that projects demonstrate 
that key technologies have reached a high maturity level. 

- Before projects are approved to transition from final design to 
fabrication, assembly, and test, the policy should require that 
projects demonstrate that the design is stable. 

- Before projects are approved to transition into production, the 
policy should require projects to demonstrate that the design can be 
manufactured within cost, schedule and quality targets. 

NASA agrees, with the following comments: 

NASA takes a risk-based approach to technology readiness. NASA will 
define the applicable technology readiness level for each phase of the 
life cycle and include these definitions in NPR 7123. NASA will also 
strengthen the current wording in NPR 7120.5C to ensure that a design 
solution incorporating sufficiently mature technology will be available 
at PDR This does not preclude the program/project manager from pursuing 
a parallel design approach utilizing a higher performance but less 
mature solution as long as the mature technology remains available and 
appropriate risk mitigation processes are employed. However, exceptions 
should be made in the cases where the primary mission objective is to 
mature and demonstrate technologies to reduce risk on future missions 
(i.e., New Millennium Program). 

NASA implements a review process that includes peer reviews both within 
NASA and at the supplier. Per NPR 7123, CDRs are one of a minimum set 
of flight systems and ground support technical reviews held for the 
purpose of demonstrating "that the maturity of the design is 
appropriate to support proceeding with full scale fabrication, 
assembly, integration, and test, and that the technical effort is on 
track to complete the flight and ground system development and mission 
operations in order to meet mission performance requirements within the 
identified cost and schedule constraints." NASA will establish 
requirements for the CDR success criteria in NPR 7123 to ensure that 
the design is stable and that the system can be fabricated within cost, 
schedule, and performance specifications. In addition, NASA will 
establish requirements for the success criteria in NPR 7123 for other 
required technical reviews in the life cycle. 

Very few NASA projects enter into a traditional production process. 
Most NASA missions fabricate one-of-a-kind spacecraft. For this class 
of mission, the CDR will thoroughly review the readiness to proceed to 
fabrication of the flight system. CDR success criteria will be reviewed 
and updated to include any additional fabrication information required 
to enable a knowledge-based decision (see above paragraph). In the 
event that systems or elements are intended for large scale production, 
including major components of unique systems, NASA will conduct a 
Production Readiness Review (PRR) that will meet required success 
criteria. This review and its success criteria will be added to NPR 
7123. 

Second GAO Recommendation: 

Revise NPR 7120.5C to institute additional major decision reviews 
following the NAR for flight systems and ground support projects, which 
result in recommendations to the appropriate decision authority. These 
reviews should be tied to the key junctures during project development 
mentioned above in order to increase the likelihood that cost, 
schedule, and performance requirements of the project will be met. 

NASA agrees, with the following comments: 

The CDR results will provide the information needed for the decision 
authority. NASA will revise NPR 7120.5C to require the results of the 
CDR to be out-briefed to the appropriate decision authority in a timely 
manner for a decision to proceed. An additional decision point will be 
added, as applicable, to out-brief the results of a PRR in the case of 
large scale production lots to the appropriate decision authority for a 
decision to proceed. 

Thank you for the opportunity to comment on this draft report. 

Sincerely, 

Signed by: 

Shana Dale: 
Deputy Administrator: 

[End of section] 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Allen Li (202) 512-4841: 

Staff Acknowledgments: 

In addition to the individual named above, James L. Morrison, Assistant 
Director; Valerie Colaiaco, Tom Gordon, Alison Heafitz, Shelby S. 
Oakley, Ron Schwenn, Karen Sloan, and John S. Warren, Jr., made key 
contributions to this report. 

[End of section] 

Related GAO Products: 

Space Reports: 

Space Acquisitions: Stronger Development Practices and Investment 
Planning Need to Address Continuing Problems. GAO-05-891T. Washington, 
D.C.: July 12, 2005. 

Defense Acquisitions: Incentives and Pressures That Drive Problems 
Affecting Satellite and Related Acquisitions. GAO-05-570R. Washington, 
D.C.: June 23, 2005. 

Defense Acquisitions: Space-Based Radar Effort Needs Additional 
Knowledge before Starting Development. GAO-04-759. Washington, D.C.: 
July 23, 2004. 

Defense Acquisitions: Risks Posed by DOD's New Space Systems 
Acquisition Policy. GAO-04-379R. Washington, D.C.: January 29, 2004. 

Space Acquisitions: Committing Prematurely to the Transformational 
Satellite Program Elevates Risks for Poor Cost, Schedule, and 
Performance Outcomes. GAO-04-71R. Washington, D.C.: December 4, 2003. 

Defense Acquisitions: Improvements Needed in Space Systems Acquisition 
Policy to Optimize Growing Investment in Space. GAO-04-253T. 
Washington, D.C.: November 18, 2003. 

Defense Acquisitions: Despite Restructuring, SBIRS High Program Remains 
at Risk of Cost and Schedule Overruns. GAO-04-48. Washington, D.C.: 
October 31, 2003. 

Defense Acquisitions: Improvements Needed in Space Systems Acquisition 
Management Policy. GAO-03-1073. Washington, D.C.: September 15, 2003. 

Military Space Operations: Common Problems and Their Effects on 
Satellite and Related Acquisitions. GAO-03-825R. Washington, D.C.: June 
2, 2003. 

Military Space Operations: Planning, Funding, and Acquisition 
Challenges Facing Efforts to Strengthen Space Control. GAO-02-738. 
Washington, D.C.: September 23, 2002. 

Polar-Orbiting Environmental Satellites: Status, Plans, and Future Data 
Management Challenges. GAO-02-684T. Washington, D.C.: July 24, 2002. 

Defense Acquisitions: Space-Based Infrared System-Low at Risk of 
Missing Initial Deployment Date. GAO-01-6. Washington, D.C.: February 
28, 2001. 

Best Practices Reports: 

Defense Acquisitions: Assessments of Selected Major Weapon Programs. 
GAO-05-301. Washington, D.C.: March 31, 2005. 

Defense Acquisitions: Stronger Management Practices Are Needed to 
Improve DOD's Software-Intensive Weapon Acquisitions. GAO-04-393. 
Washington, D.C.: March 1, 2004. 

Defense Acquisitions: Assessments of Selected Major Weapon Programs. 
GAO-04-248. Washington, D.C.: March 31, 2004. 

Defense Acquisitions: DOD's Revised Policy Emphasizes Best Practices, 
but More Controls Are Needed. GAO-04-53. Washington, D.C.: November 10, 
2003. 

Defense Acquisitions: Assessments of Selected Major Weapon Programs. 
GAO-03-476. Washington, D.C.: May 15, 2003. 

Best Practices: Setting Requirements Differently Could Reduce Weapon 
Systems' Total Ownership Costs. GAO-03-57. Washington, D.C.: February 
11, 2003. 

Best Practices: Capturing Design and Manufacturing Knowledge Early 
Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.: July 15, 
2002. 

Defense Acquisitions: DOD Faces Challenges in Implementing Best 
Practices. GAO-02-469T. Washington, D.C.: February 27, 2002. 

Best Practices: Better Matching of Needs and Resources Will Lead to 
Better Weapon System Outcomes. GAO-01-288. Washington, D.C.: March 8, 
2001. 

Best Practices: A More Constructive Test Approach Is Key to Better 
Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.: July 31, 
2000. 

Defense Acquisition: Employing Best Practices Can Shape Better Weapon 
System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26, 2000. 

Best Practices: DOD Training Can Do More to Help Weapon System Program 
Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.: August 
16, 1999. 

Best Practices: Better Management of Technology Development Can Improve 
Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30, 
1999. 

Defense Acquisitions: Best Commercial Practices Can Improve Program 
Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999. 

Defense Acquisition: Improved Program Outcomes Are Possible. GAO/T- 
NSIAD-98-123. Washington, D.C.: March 18, 1998. 

Best Practices: Successful Application to Weapon Acquisition Requires 
Changes in DOD's Environment. GAO/NSIAD-98-56. Washington, D.C.: 
February 24, 1998. 

Major Acquisitions: Significant Changes Underway in DOD's Earned Value 
Management Process. GAO/NSIAD-97-108. Washington, D.C.: May 5, 1997. 

Best Practices: Commercial Quality Assurance Practices Offer 
Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.: August 26, 
1996. 

FOOTNOTES 

[1] The Vision includes a return to the moon that is intended 
ultimately to enable future exploration of Mars and other destinations. 
To accomplish this, NASA initially plans to (1) complete its work on 
the International Space Station by 2010, fulfilling its commitment to 
15 international partner countries; (2) begin developing a new manned 
exploration vehicle to replace the space shuttle; and (3) return to the 
moon as early as 2015 and no later than 2020 in preparation for future, 
more ambitious missions. 

[2] NASA Policy Directive (NPD) 7120.4C describes the management system 
by which NASA formulates, approves, implements, and evaluates all 
programs and projects. NASA Procedural Requirement (NPR) 7120.5C 
establishes the management system requirements for implementing NPD 
7120.4C. NPR 7120.5C governs the formulation, approval, implementation, 
and evaluation of all agency programs and projects. For purposes of 
this report, we refer to NPR 7120.5C as NASA's program and project 
management policy. 

[3] NASA consists of NASA headquarters, nine centers, the Jet 
Propulsion Laboratory (operated under contract to NASA by the 
California Institute of Technology), and several ancillary 
installations and offices in the United States and abroad. The 
implementation of NASA programs and aeronautical and space/earth 
science research occurs primarily at the centers. 

[4] GAO, NASA: Better Mechanisms Needed for Sharing Lessons Learned, 
GAO-02-195 (Washington, D.C.: Jan. 30, 2002). 

[5] The approach was intended to help NASA reduce costs, become more 
efficient, and increase scientific results by conducting more and 
smaller missions in less time. 

[6] NPR 7120.5C contains specific requirements for both programs and 
projects. The policy defines a program as a strategic investment by a 
Mission Directorate or Mission Support Office that has defined goals, 
objectives, architecture, funding level, and a management structure 
that supports one or more projects. A project is defined as a specific 
investment identified in a Program Plan having defined goals, 
objectives, requirements, life cycle cost, a beginning, and an end. For 
purposes of this report, we have focused our analysis of the policy as 
it relates to the traditional development approach to flight systems 
and ground support projects. 

[7] According to the NASA Administrator, evolutionary acquisition 
processes will not be practiced at NASA. Therefore, NASA officials 
indicated that evolutionary acquisition for flight systems and ground 
support projects will likely be deleted in the next version of NPR 
7120.5. 

[8] One oversight authority is called a Project Management Committee 
(PMC). The PMC is defined as one of the hierarchy of forums, composed 
of senior management, that assesses project planning and 
implementation, and provides oversight and direction as appropriate. 
These are established at the agency, mission directorate, center, and 
lower levels. The Independent Program Assessment Office (IPAO) is the 
NASA organization responsible for scheduling, organizing, and 
conducting the independent reviews at the Preliminary Non-Advocate 
Review (Pre-NAR) and the Non-Advocate Review (NAR) for programs and 
projects reporting to the agency PMC. The Systems Management Office 
(SMO) is the center organization responsible for independent review and 
assessment of projects at the Pre-NAR and the NAR, whose findings are 
reported to the center PMC. 

[9] As defined by NPR 7120.5C, a CADRe is a formal document to 
understand the cost and cost risk of space flight projects. The policy 
defines EVM as a tool for measuring and assessing project performance 
through the integration of technical scope with schedule and cost 
objectives during the execution of the project. As defined in NPR 
7120.5C, all projects over $20 million must apply EVM. 

[10] National Aeronautics and Space Administration, NASA Systems 
Engineering Handbook, SP-6105, (Washington, D.C.: June 1995). 

[11] Our best practice reviews are identified in the "Related GAO 
Products" section at the end of this report. 

[12] The key management documents for flight system and ground support 
projects are the Project Formulation Authorization Document (FAD), or 
equivalent, and the Project Plan. The FAD authorizes a Project Manager 
to initiate the formulation phase of a new project. Because a new 
project represents a major commitment of resources, the FAD requires 
approval of the project decision authority before the project enters 
Phase A of the life cycle. The Project Plan is an agreement between the 
Mission Directorate Associate Administrator, the Center Director, 
Program Manager, and Project Manager, as applicable. It defines, at a 
high level, the scope of the project, the implementation approach, the 
environment within which the project operates, and the commitments of 
the project. The Project Plan also establishes the cost, schedule, and 
technical baselines for the implementation phase and is used by the 
GPMC in the review process to determine if the project is fulfilling 
its agreements. A preliminary Project Plan is due at the Pre-NAR, and 
an updated or final version is due at the NAR. 

[13] The decision authority for Category I projects is the Deputy 
Administrator; for Category II projects, the Mission Directorate 
Associate Administrator; and for Category III projects, the Center 
Director (of the executing center). 

[14] A positive recommendation may be unconditional, or conditional on 
the Project Manager completing assigned action items. A negative 
recommendation could result in the decision authority either directing 
the Project Manager to address the deficiencies, or in the 
authorization of a termination review. 

[15] GAO's 2005 assessment of 54 systems within the Department of 
Defense showed that development costs for programs that started 
development with mature technology increased by an average of 9 percent 
over the first full estimate, whereas the development costs for the 
programs that started development with immature technologies increased 
an average of 41 percent over the first full estimate. Likewise, 
program acquisition unit costs for the programs with mature technology 
increased by less than 1 percent, whereas the programs that started 
development with immature technologies experienced an average program 
acquisition unit cost increase of nearly 21 percent over the first full 
estimate. Programs with mature technology experienced an average 
schedule delay of 7 months--a 9 percent increase--whereas the schedule 
for the programs that started development with immature technology 
increased an average of 13 months--a 13 percent increase. GAO, Defense 
Acquisitions: Assessments of Selected Major Weapon Programs, GAO-05-301 
(Washington, D.C.: Mar. 31, 2005). 

[16] In NASA this is "Phase D--Fabrication, Assembly and Test." 

[17] Projects may also be required to complete specific technical and 
management reviews per individual center policy. 

[18] According to NASA officials, some NASA centers require projects to 
provide status updates to the center PMC after the NAR, for example 
through monthly or quarterly status reports. 

[19] GAO, Better Management of Technology Development Can Improve 
Weapon System Outcomes, GAO/NSIAD-99-162 (Washington, D.C., July 30, 
1999) and GAO, Best Practices: Capturing Design and Manufacturing 
Knowledge Early Improves Acquisition Outcomes, GAO-02-701 (Washington, 
D.C., July 15, 2002). 

[20] NASA's Systems Engineering Handbook provides systems engineers and 
project managers with a generic description of NASA systems engineering 
and a common language and perspective of the systems engineering 
process. See National Aeronautics and Space Administration, NASA 
Systems Engineering Handbook, SP-6105. (Washington, D.C.: June 1995). 

GAO's Mission: 

The Government Accountability Office, the investigative arm of 
Congress, exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site (www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to e-mail alerts" under the "Order 
GAO Products" heading. 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548: 

To order by Phone: 

Voice: (202) 512-6000: 

TDD: (202) 512-2537: 

Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov 

Automated answering system: (800) 424-5454 or (202) 512-7470: 

Public Affairs: 

Jeff Nelligan, managing director, 

NelliganJ@gao.gov 

(202) 512-4800 

U.S. Government Accountability Office, 

441 G Street NW, Room 7149 

Washington, D.C. 20548: