This is the accessible text file for GAO report number GAO-15-100 
entitled 'James Webb Space Telescope: Project Facing Increased 
Schedule Risk with Significant Work Remaining' which was released on 
December 15, 2014. 

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as 
part of a longer term project to improve GAO products' accessibility. 
Every attempt has been made to maintain the structural and data 
integrity of the original printed product. Accessibility features, 
such as text descriptions of tables, consecutively numbered footnotes 
placed at the end of the file, and the text of agency comment letters, 
are provided but may not exactly duplicate the presentation or format 
of the printed version. The portable document format (PDF) file is an 
exact electronic replica of the printed version. We welcome your 
feedback. Please E-mail your comments regarding the contents or 
accessibility features of this document to Webmaster@gao.gov. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Report to Congressional Committees: 

December 2014: 

James Webb Space Telescope: 

Project Facing Increased Schedule Risk with Significant Work Remaining: 

GAO-15-100: 

GAO Highlights: 

Highlights of GAO-15-100, a report to congressional committees. 

Why GAO Did This Study: 

JWST is one of NASA's most complex and expensive projects, at an
anticipated cost of $8.8 billion. With significant integration and 
testing planned until the launch date, the JWST project will need to 
address many challenges before NASA can conduct the science the 
telescope is intended to produce. GAO has made a number of prior 
recommendations to NASA, including in December 2012 that the project 
perform an updated joint cost and schedule risk analysis to improve 
cost estimates. NASA initially concurred with this recommendation,
but it later indicated that the tracking of information it already had 
in place was sufficient and ultimately decided not to conduct another 
joint cost and schedule risk analysis. 

GAO was mandated to assess the program annually and report on its
progress. This is the third such report. This report assesses, among 
other issues, the extent to which (1) technical challenges are 
impacting the JWST project's ability to stay on schedule and budget, 
and (2) budget and cost estimates reflect current information about 
project risks. To conduct this work, GAO reviewed monthly and 
quarterly JWST reports, interviewed NASA and contractor officials, 
reviewed relevant policies, and conducted independent analysis of
NASA and contractor data. 

What GAO Found: 

With just under 4 years until its planned launch in October 2018, the 
James Webb Space Telescope (JWST) project reports it remains on 
schedule and budget. Technical challenges with JWST elements and major 
subsystems, however, have diminished the project's overall schedule 
reserve and increased risk. During the past year, delays have occurred 
on every element and major subsystem schedule—-especially with the 
cryocooler—-leaving all at risk of negatively impacting the overall 
project schedule reserve if further delays occur. 

Figure: Schedule Reserve Changes on the James Webb Space Telescope's 
Elements and Major Subsystems from 2013 to 2014: 

[Refer to PDF for image: illustration] 

Elements and major subsystems of the James Webb Space Telescope: 

Current critical path: 11 months. 

Spacecraft: 
Current schedule reserve (2014): 14 months; 
Schedule reserve (2013): 11.5 months. 

Sunshield: 
Current schedule reserve (2014): 14 months; 
Schedule reserve (2013): 11.25 months. 

Cryocooler[A]: 
Current schedule reserve (2014): 16 months; 
Schedule reserve (2013): 11 months. 

OTE: 
Current schedule reserve (2014): 14 months; 
Schedule reserve (2013): 11.5 months. 

ISIM: 
Current schedule reserve (2014): 14.5 months; 
Schedule reserve (2013): 11.5 months. 

Source: GAO analysis of NASA data. GAO-15-100. 

OTE: Optical Telescope Element; 
ISIM: Integrated Science Instrument Module. 

[A] The cryocooler chills an infrared light detector on one of JWST's 
four scientific instruments. 

[End of figure] 

The project reports its overall schedule reserve is above its plan and 
standards. However, JWST is one of the most complex projects in the 
National Aeronautics and Space Administration's (NASA) history and has 
begun integrating and testing only two of the five elements and major 
subsystems. As such, maintaining as much schedule reserve as possible 
to navigate through almost 4 more years of integration and testing 
that remains, where prior work has shown problems are commonly found 
and schedules tend to slip, is critical. While the project has been
able to reorganize work when necessary to mitigate schedule slips, 
this flexibility will diminish going forward. JWST is also facing 
limited short-term cost reserves to mitigate additional project 
schedule threats. 

The JWST project and prime contractor's cost risk analyses used to 
validate the JWST budget are outdated and do not account for many new 
risks identified since 2011. GAO best practices for cost estimating 
call for regularly updating cost risk analyses to validate that 
reserves are sufficient to account for new risks. NASA officials said 
they conduct sufficient analysis to monitor the health of the budget. 
These efforts, however, do not incorporate potential impacts of risks
identified since 2011 into estimates. While the project has 
subsequently agreed to conduct a cost risk analysis of the contract, 
it is important that they follow best practices, for example, by 
regularly updating that analysis. Doing so would provide the project 
with reliable information to gauge whether the contractor is at
risk of future cost overruns. 

What GAO Recommends: 

Among other actions, NASA should follow best practices when updating its
cost risk analysis to ensure reliability. In commenting on a draft of 
this report, NASA partially concurred with this recommendation. 

View [hyperlink, http://www.gao.gov/products/GAO-15-100]. For more 
information, contact Cristina Chaplain at (202) 512-4841 or 
chaplainc@gao.gov. 

[End of section] 

Contents: 

Letter: 

Background: 

Project Risk Heightened Due to Widespread Use of Schedule Reserve to 
Address Technical Issues: 

Project Lacks Current Cost Risk Analyses, Creating Uncertainty About 
Adequacy of Its Budget: 

Contractors Have Been Responsive to Incentives, but Final Evaluation 
May Not Reflect Total Contract Performance: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Elements and Major Subsystems of the James Webb Space 
Telescope (JWST) Observatory: 

Appendix III: Organizational Chart for the James Webb Space Telescope 
Program: 

Appendix IV: Other Technical Issues for the James Webb Space Telescope: 

Appendix V: Comments from the National Aeronautics and Space 
Administration: 

Appendix VI: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: NASA's Process for Evaluation of Award Fees: 

Table 2: James Webb Space Telescope Award Fee Plans Comparison with 
NASA Final Evaluation Guidance: 

Figures: 

Figure 1: James Webb Space Telescope: 

Figure 2: Integration and Test (I&T) Schedule for the James Webb Space 
Telescope Elements: 

Figure 3: Components of the Mid-Infrared Instrument Cryocooler: 

Figure 4: James Webb Space Telescope Schedule Reserve by Element and 
Major Subsystem in 2013 and 2014: 

Figure 5: Status of James Webb Space Telescope Milestones in Fiscal 
Years 2011-2014: 

Figure 6: Cryocooler Component Schedule Delays Since April 2013 
Schedule Replan: 

Figure 7: James Webb Space Telescope: Interactive Information: 

Figure 8: Cryocooler Compressor Vibration: 

Figure 9: Excess Mid-Infrared Stray Light: 

Abbreviations: 

EVM: earned value management: 

FAR: Federal Acquisition Regulation: 

FGS: Fine Guidance Sensor: 

Goddard: Goddard Space Flight Center: 

I&T: integration and test: 

ICRP: Independent Comprehensive Review Panel: 

ISIM: Integrated Science Instrument Module: 

JCL: Joint Cost and Schedule Confidence Level: 

JPL: Jet Propulsion Laboratory: 

JWST: James Webb Space Telescope: 

K: Kelvin: 

MIRI: Mid-Infrared Instrument: 

NASA: National Aeronautics and Space Administration: 

NIRCam: Near-Infrared Camera: 

NIRISS: Near-Infrared Imager and Slitless Spectrograph: 

NIRSpec: Near-Infrared Spectrograph: 

OTE: Optical Telescope Element: 

OTIS: Optical Telescope Element and Integrated Science Instrument 
Module: 

STScI: Space Telescope Science Institute: 

[End of section] 

United States Government Accountability Office: 
GAO:
441 G St. N.W. 
Washington, DC 20548: 

December 15, 2014: 

Congressional Committees: 

The successful launch and operation of the James Webb Space Telescope 
(JWST) is one of the National Aeronautics and Space Administration's 
(NASA) and the scientific community's top priorities to continue to 
advance human knowledge of the cosmos. It is intended to revolutionize 
understanding of star and planet formation, advance the search for the 
origins of the universe, and further the search for earth-like 
planets. Projects of this scale, complexity, and technological 
sophistication confront a myriad of unforeseen challenges in their 
ambitious attempts to expand the limits of scientific accomplishment. 
With significant integration and testing planned over the course of 
the next 4 years, the JWST project will need to address many 
challenges before the telescope's planned launch in October 2018. The 
success of JWST hinges on the ability of NASA and its contractors to 
adjust and respond to these challenges in a timely and cost-effective 
manner. With future NASA missions and exploration depending on the 
success of JWST, its ultimate outcome will have effects for many years 
into the future. 

The on-time and on-budget delivery of JWST is a high congressional 
priority. In November 2011, the conferees of the Joint Explanatory 
Statement of the Committee of Conference for the Consolidated and 
Further Continuing Appropriation Act, 2012, directed GAO to assess the 
JWST program annually and to report to the Committees on 
Appropriations on key issues relating to program and risk management, 
achievement of cost and schedule goals, and program technical status. 
[Footnote 1] This report is our third in response to that mandate. For 
this report, we assessed the extent to which (1) technical challenges 
are impacting the JWST project's ability to stay on schedule and 
budget, (2) budget and cost estimates reflect current information 
about project risks, and (3) the JWST project uses award fee contracts 
to motivate and assess contractor performance. 

Our approach included an examination of the cost, schedule, and 
technical performance of the project since our last report in January 
2014, which focused on the cost and schedule baselines that were 
established for the JWST program in 2011. We reviewed the project's 
risk database, monthly status reviews, and technical and other 
documentation; analyzed the progress made against planned milestones; 
and held interviews with program, project, and contractor officials on 
elements and major subsystems. To assess the extent to which budget 
and cost estimates reflect current information about project risks, we 
compared the prime contractor's past and present risk documentation; 
analyzed the project's cost reserve data; and interviewed project, 
program, and contractor officials to obtain information on their cost 
estimating and cost reserve management practices to compare against 
GAO best practices.[Footnote 2] To assess the extent to which the JWST 
project uses award fee contracts to motivate and assess contractor 
performance, we reviewed the project's contract documentation, 
contract files, and performance evaluation plans for JWST's two award 
fee contracts, as well as federal and NASA acquisition regulations and 
guidance. See appendix I for a detailed description of our scope and 
methodology. 

We conducted this performance audit from February 2014 to December 
2014 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

Background: 

JWST is a large, deployable, infrared-optimized space telescope 
intended to be the scientific successor to the Hubble Space Telescope. 
JWST is designed for a 5-year mission to find the first stars and 
trace the evolution of galaxies from their beginnings to their current 
formation. It is intended to operate in an orbit approximately 1.5 
million kilometers--or 1 million miles--from the Earth. With its 6.5-
meter primary mirror, JWST will be able to operate at 100 times the 
sensitivity of the Hubble Space Telescope. The JWST's science 
instruments require extremely cold temperatures, so a tennis-court-
sized sunshield will be used to protect the mirrors and instruments 
from the sun's heat and allow them to observe very faint infrared 
sources. The Hubble Space Telescope operates primarily in the visible 
and ultraviolet regions of the electromagnetic spectrum.[Footnote 3] 
The observatory segment of JWST includes several elements (Optical 
Telescope Element (OTE), Integrated Science Instrument Module (ISIM), 
and spacecraft) and major subsystems (sunshield, cryocooler).[Footnote 
4] 

These elements and major subsystems are being developed through a 
mixture of NASA, contractor, and international partner efforts. See 
interactive graphic, figure 1. For more information on the JWST's 
organizational structure, see appendix III. 

Figure 1: James Webb Space Telescope: 

[Refer to PDF for image: Interactive Graphic] 

Rollover the white dots to see description. See appendix II for the 
non-interactive, printer-friendly version. 

Sources: GAO (analysis); NASA (data and images). GAO-15-100. 

[End of figure] 

Given JWST's complexity, it requires a multiyear integration and 
testing schedule that includes five separate periods over the course 
of almost 7 years to build the observatory in preparation for launch. 
See figure 2 for the planned integration and test flow for JWST. 

Figure 2: Integration and Test (I&T) Schedule for the James Webb Space 
Telescope Elements: 

[Refer to PDF for image: timeline] 

July 2011--December 2015: 
ISIM I&T: integrates and tests the 4 science instruments; 
Schedule reserve: early 2016. 

September 2014[[January 2016: 
OTE I&T: integrates and tests the optics and support structure; 
Schedule reserve: early 2016. 

February 2016--July 2017: 
Spacecraft I&T: integrates the spacecraft and the sunshield; 
Schedule reserve: mid-2017. 

April 2016--July 2017: 
OTIS I&T: integrates the ISIM and the OTE; 
Schedule reserve: mid-2017. 

September 2017--May 2018: 
Observatory I&T: integrates spacecraft and OTIS; 
Schedule reserve: late-2018. 

October 2018: 
Launch date. 

ISIM: Integrated Science Instrument Module; 
I&T: Integration and Test; 
OTE: Optical Telescope Element; 
OTIS: OTE + ISIM Integrated. 

Source: GAO analysis of NASA data. GAO-15-100. 

Note: There are multiple lower level I&T efforts that flow in to the 
ISIM, OTE, and Spacecraft I&T flow that are not depicted on figure 2. 
For example, the sunshield is a major subsystem that has its own I&T 
effort that ends with delivery for Spacecraft I&T. 

[End of figure] 

For the majority of the work remaining, the JWST project will rely on 
three contractors: Northrop Grumman, Exelis, and the Space Telescope 
Science Institute (STScI). Northrop Grumman plays the largest role, 
developing the sunshield, the OTE, the spacecraft, and a cooling 
subsystem for the Mid-Infrared Instrument (MIRI). Northrop Grumman 
performs most of this work under a prime contract with NASA, but its 
work on the MIRI cooler is performed under a separate subcontract with 
the Jet Propulsion Laboratory (JPL). Exelis is manufacturing the test 
equipment, equipping the test chamber, and assisting in the testing of 
the optics of JWST. Finally, STScI will collect and evaluate research 
proposals from the scientific community and will receive and store the 
scientific data collected by those observations, both of which are 
services that they currently provide for the Hubble Space Telescope. 
Additionally, for JWST, STScI will develop and operate the ground 
system that manages and controls the telescope's observations on 
behalf of NASA. 

MIRI--one of JWST's four instruments in the ISIM--requires a 
dedicated, interdependent two-stage cooler subsystem designed to bring 
the infrared light detector within MIRI to the required temperature of 
6.7 Kelvin (K), just above absolute zero. This major subsystem is 
referred to as a cryocooler. The cryocooler moves helium gas through 
10 meters (approximately 33 feet) of refrigerant lines from the 
spacecraft located on the sun-facing surface of the JWST observatory 
to the colder, shaded side where the ISIM is located. According to 
NASA officials, a cooler of this configuration has never been 
developed or flown in space before. See figure 3 for a depiction of 
the cryocooler on JWST. 

Figure 3: Components of the Mid-Infrared Instrument Cryocooler: 

[Refer to PDF for image: illustration] 

Components depicted: 

Optical Telescope Element (OTE): 
Spacecraft (~300K); 
Sun Facing Surface; 
Cryocooler electronics; 
Sunshield; 
40K Integrated Science Instrument Module (ISIM); 
6 Kelvin (K) Mid-infrared instrument (MIRI): 
6K coldhead; 
18K coldhead. 

Source: GAO presentation of NASA data. GAO-15-100. 

[End of figure] 

Project officials stated that the MIRI cryocooler is particularly 
complex and challenging because of the relatively great distance 
between the cooling components and the need to overcome multiple 
sources of unwanted heat through the regions of JWST before the 
subsystem can cool MIRI's detector. Specifically, the cooling 
components span temperatures ranging from approximately 300K (about 80 
degrees Fahrenheit, or room temperature) where the spacecraft is 
located on the sun-facing surface to approximately 40K (about -388 
degrees Fahrenheit) within the ISIM. 

Cost and Schedule Reserves for NASA Projects: 

Complex development efforts like JWST must plan to address a myriad of 
risks and unforeseen technical challenges. To do this, projects 
reserve extra time in their schedules--which is referred to as 
schedule reserve--and extra money in their budgets-which is referred 
to as cost reserve. Schedule reserve is extra time in the project's 
overall schedule that is allocated to specific activities, elements, 
and major subsystems in the event there are delays or to address 
unforeseen risks. Each JWST element and major subsystem has been 
allocated schedule reserve. When an element or major subsystem 
exhausts schedule reserve it may begin to affect schedule reserves on 
other elements or major subsystems whose progress is dependent on 
prior work being finished for its activities to proceed. The element 
or major subsystem with the least amount of schedule reserve 
determines the critical path for the project. Any delay to an activity 
that is on the critical path will reduce schedule reserve for the 
whole project, and could ultimately impact the overall project 
schedule. 

Cost reserves are additional funds that can be used to address 
unanticipated issues for any element or major subsystem. Cost reserves 
are used to mitigate issues during the development of a project. For 
example, cost reserves can be used to buy additional materials to 
replace a component or, if a project needs to preserve schedule 
reserve, reserves can be used to accelerate work by adding extra 
shifts to expedite manufacturing and save time. NASA's Goddard Space 
Flight Center (Goddard)--the NASA center with responsibility for 
managing JWST--has issued requirements that at project confirmation 
establish both the level of cost and funded schedule reserves that 
projects must hold. After this point, a specified amount of schedule 
reserve continues to be required throughout the remainder of 
development.[Footnote 5] 

History of Cost Growth, Low Project Reserves, and Schedule Delays: 

Prior to 2011, early technical and management challenges, contractor 
performance issues, low level cost reserves, and poorly phased funding 
levels caused JWST to delay work, which contributed to significant 
cost and schedule overruns. After years of requirements changes, cost 
growth, low reserve funding, deferred work, and launch delays, the 
Chair of the Senate Subcommittee on Commerce, Justice, Science, and 
Related Agencies requested an independent review of JWST. NASA 
commissioned the Independent Comprehensive Review Panel, which issued 
its report in October 2010, and concluded that JWST was executing well 
from a technical standpoint, but that the baseline funding did not 
reflect the most probable cost with adequate reserves in each year of 
project execution, resulting in an unexecutable project.[Footnote 6] 
Following this review, the JWST program underwent a replan in 
September 2011 and Congress in November 2011 placed an $8 billion cap 
on the formulation and development costs for the project.[Footnote 7] 
On the basis of the replan, NASA rebaselined JWST with a life-cycle 
cost estimate of $8.835 billion that included additional money for 
operations and a planned launch in October 2018. The revised life-
cycle cost estimate included a total of 13 months of funded schedule 
reserve. In the President's fiscal year 2013 budget request, NASA 
reported a 66 percent joint cost and schedule confidence level for 
these cost and schedule baselines. A joint cost and schedule 
confidence level, or JCL, is the process NASA uses to assign a 
percentage to the probable success of meeting cost and schedule 
estimates and is part of the project's estimating process.[Footnote 8] 

Previous GAO Reviews of JWST Project: 

In January 2014, GAO found that the JWST project was generally 
executing to its 2011 revised cost and schedule baselines, but that 
the most recent work was taking longer than planned.[Footnote 9] We 
noted that roughly half of the project's cost reserves for fiscal 
years 2014 and 2015 had been committed before those years had even 
begun, which would constrain JWST's cost reserve spending in the near 
term. We reported that while the cryocooler subcontractor was making 
technical progress, cryocooler development remained an ongoing 
challenge as there were still unresolved technical issues related to 
its performance, and its schedule was both optimistic and unreliable. 
We also found that three of the element and major subsystem schedules 
the project was using to manage were unreliable and did not meet best 
practices, which called into question the reliability of the 
integrated master schedule that is built from those schedules. In 
2012, we reported that the accuracy of the project's cost estimate was 
lessened because the summary schedule used for the joint cost and 
schedule confidence level was not detailed enough to determine how 
risks were applied to critical project activities.[Footnote 10] We 
recommended that NASA conduct an updated joint cost and schedule risk 
analysis to address the issues we identified and use more detailed 
cost information to adjust its cost estimates. While NASA concurred 
with our recommendation, officials subsequently stated that they did 
not plan to conduct an updated joint cost and schedule confidence 
level analysis, and that the project's monthly analyses were 
sufficient for the project's needs. Thus, we expressed concern in our 
January 2014 report that NASA's continued lack of up-to-date cost 
estimates that incorporate new risks which had emerged since 2011 
would inhibit NASA's ability to effectively monitor its contractors' 
progress. We proposed that Congress consider requiring NASA to conduct 
an updated joint cost and schedule confidence level analysis to 
address this concern. 

Project Risk Heightened Due to Widespread Use of Schedule Reserve to 
Address Technical Issues: 

The JWST project has 11 months of schedule reserve remaining--more 
than required by Goddard standards. In the case of JWST, however, the 
most significant risks lie ahead in the remaining 4 years, as the 
project must complete five integration and test periods, three of 
which have not yet started. Essentially, the risk is defined by 
bringing all of the complex JWST elements and major subsystems 
together, testing them, and resolving problems when there is limited 
schedule left for resolution. All of JWST's five elements and major 
subsystems have just weeks of reserve left before their schedules 
become pacing items on the project's critical path, potentially 
reducing the reserve further. More milestones established annually for 
the various elements and major subsystems have been delayed or 
deferred during fiscal year 2014 than in the previous 3 years 
following the replan. For example, the MIRI cryocooler deferred seven 
milestones until fiscal year 2015 as a result of manufacturing and 
development delays. Schedule risks are further heightened as the 
project entered fiscal year 2015 with approximately 40 percent of its 
cost reserves already committed, leaving fewer dollars available to 
mitigate other threats to the project schedule. 

Flexibility to Address Future Challenges Constrained by Schedule 
Reserve Loss Prior to Most Integration and Test Efforts: 

The JWST schedule reserve was diminished this year following the need 
to address several significant technical challenges across the 
project's various elements and major subsystems, and as a result of 
the October 2013 government shutdown. Specifically, in the past 14 
months, the overall project schedule reserve declined from 14 months 
to 11 months.[Footnote 11] With less than 4 years until the planned 
launch in October 2018, the project's overall schedule reserve is 
above the Goddard standard and JWST plan--which was set above the 
Goddard standard and included more reserve than required--for schedule 
reserve at this point in the project. The scale of JWST's integration 
and test effort, however, is more complex than most NASA or Goddard 
projects. For example, JWST has five major integration and test 
periods to build the telescope, test the optics, integrate the various 
elements and major subsystems, and ensure they work properly together. 
While JWST's total integration and test cycle runs almost 7 years, the 
next longest integration and test cycle for a current project managed 
by Goddard Space Flight Center is over 3 years and the average length 
of integration and testing for all other current Goddard projects, 
excluding JWST, is just over 2 years. The JWST project has used over 
20 percent of the schedule reserve it held in the past 14 months with 
almost 4 years remaining for its integration and test effort, where 
GAO's prior work has shown and NASA has concurred problems are 
commonly found and schedules tend to slip. 

All individual element and major subsystem schedules have lost 
schedule reserve since last year. While some use of schedule reserve 
is expected, this widespread use has resulted in every JWST element 
and major subsystem schedule being within weeks of becoming the 
critical path of the project, before most have entered their 
integration and testing periods. The proximity of all of the element 
and major subsystem schedules to the critical path means that an issue 
on any of the schedules may reduce the overall project schedule 
reserve further, which could then put the overall project schedule at 
risk. As a result of this, the project has less flexibility to choose 
which issues to mitigate. As an illustration, the critical path has 
switched numerous times to several different schedules as a result of 
using schedule reserve in the past year. See the different amounts of 
schedule reserve remaining on all elements and major subsystems, their 
proximity to the critical path and the total schedule reserve for the 
critical path in figure 4. 

Figure 4: James Webb Space Telescope Schedule Reserve by Element and 
Major Subsystem in 2013 and 2014: 

[Refer to PDF for image: vertical bar graph] 

Elements and major subsystems of the James Webb Space Telescope: 

Current critical path: 11 months. 

Spacecraft: 
Current schedule reserve (2014): 14 months; 
Schedule reserve (2013): 11.5 months. 

Sunshield: 
Current schedule reserve (2014): 14 months; 
Schedule reserve (2013): 11.25 months. 

Cryocooler: 
Current schedule reserve (2014): 16 months; 
Schedule reserve (2013): 11 months. 

OTE: 
Current schedule reserve (2014): 14 months; 
Schedule reserve (2013): 11.5 months. 

ISIM: 
Current schedule reserve (2014): 14.5 months; 
Schedule reserve (2013): 11.5 months. 

Source: GAO analysis of NASA data. GAO-15-100. 

OTE: Optical Telescope Element; 
ISIM: Integrated Science Instrument Module. 

[End of figure] 

Delays were also evident in key milestone slips where more than half 
of the project-identified key milestones for fiscal year 2014 were 
completed late or deferred to fiscal year 2015--the most in any year 
since the replan. Seven of the 11 deferred milestones are related to 
the development of the cryocooler. The other 4 are due to delays in 
reviews or assembly for components of the sunshield and spacecraft and 
late completion of some integration activities for the chamber to test 
the optics of JWST in 2016. See figure 5 for a list of total 
milestones and their status. 

Figure 5: Status of James Webb Space Telescope Milestones in Fiscal 
Years 2011-2014: 

[Refer to PDF for image: vertical bar graph] 

Fiscal year: 2011; 
Milestones completed: 18; 
Milestones completed late: 3; 
Milestones deferred to future year: 0; 
Total milestones: 21. 

Fiscal year: 2012; 
Milestones completed: 32; 
Milestones completed late: 2; 
Milestones deferred to future year: 3; 
Total milestones: 37. 

Fiscal year: 2013; 
Milestones completed: 33; 
Milestones completed late: 5; 
Milestones deferred to future year: 3; 
Total milestones: 41. 

Fiscal year: 2014[A]; 
Milestones completed: 15; 
Milestones completed late: 8; 
Milestones deferred to future year: 11; 
Total milestones: 36. 

Source: GAO presentation of NASA data. GAO-15-100. 

[A] Only 34 of the 36 milestones are accounted for above as NASA 
counted 2 of them as late, but not deferred. One milestone was 
completed after the close of fiscal year 2014, but it was not included 
with deferred milestones because the delay was due to the government 
shutdown in 2013 that forced a significant test to be started later. 
The other milestone involved two pieces of focal plane hardware that 
needed to be developed--both of which are complete--but one of the two 
has not yet arrived at Goddard Space Flight Center to close out the 
milestone. 

[End of figure] 

The project has mitigated a number of issues that could have further 
impacted the status of schedule reserve by reorganizing work to obtain 
efficiencies and maintain schedule or stem further losses. For 
example, a component of the cryocooler had some planned acceptance 
testing reorganized along with a reduction in the length of time of 
the testing to accommodate delays. However, with further progression 
into subsequent integration and test periods, flexibility will be 
diminished because work during integration and testing tends to be 
more serial. This is particularly the case with JWST given its 
complexity. 

To some extent, projects have the ability to use cost reserves to 
maintain schedule. For example, if work is taking longer than 
expected, a project may be able to fund a second shift of work to 
speed it up and maintain schedule. In the short term, however, this 
option may be limited for JWST project officials because approximately 
40 percent of the project's fiscal year 2015 cost reserves were 
already committed prior to the start of the fiscal year. As such, one 
of the project's top issues for fiscal year 2015 is its cost reserve 
posture, which the project reported is less than desired and will 
require close monitoring. The project's cost reserve status for fiscal 
year 2015 is slightly better than in fiscal year 2014 that we reported 
on in January 2014, where the project successfully managed low 
reserves without delaying work. This issue, however, will likely 
require continued monitoring until fiscal year 2016 when more program 
level reserves will be available to the project. With schedule losses 
having already occurred on most elements and major subsystems across 
the project prior to most of the project's integration and testing 
efforts, the fiscal year 2016 funding may arrive too late to prevent 
further schedule delays and the risk of such delays impacting the 
launch date is heightened. 

Schedule Delays Driven by Significant Design and Manufacturing 
Challenges with Several Key Elements and Major Subsystems: 

Cryocooler: 

The project continues to face major technical challenges building the 
cryocooler that have significantly delayed delivery of key components, 
have made it the driver of the project's overall schedule or the 
project's critical path, and required the use of a disproportionate 
amount of project cost reserves. Since the 2011 replan, the cryocooler 
development has experienced over 150 percent cost growth. 
Consequently, it was one of the largest users of project cost reserves 
in fiscal year 2014 and is contributing to the project's limited cost 
reserve status for fiscal year 2015. 

All three cryocooler components (the cold head assembly, the 
compressor assembly, and the electronics assembly) fell significantly 
behind the delivery dates established during the cryocooler's April 
2013 schedule replan. As we reported in January 2014, the schedule 
that was adopted during the April 2013 cryocooler replan did not take 
into account significant risks identified by the cryocooler 
subcontractor and was overly optimistic--a sentiment that was shared 
by the cryocooler subcontractor program manager.[Footnote 12] 
Moreover, the cryocooler design was not mature at the time of the 2013 
replan. The cryocooler subcontractor program manager's low confidence 
in the replanned schedule was validated in 2014, when design and 
manufacturing problems contributed to further delays. In late 2013 and 
early 2014, implementation of a design change to resolve a 
manufacturability issue resulted in a 4-month delay to the cryocooler 
compressor assembly schedule. Additionally, as cryocooler 
manufacturing efforts accelerated, the cryocooler subcontractor 
program manager indicated that further delays occurred as a result of 
design immaturity, manufacturing, and process issues. See figure 6 
below for the delays since the April 2013 replan to subcontractor 
Northrop Grumman's delivery of the various cryocooler components to 
JPL or Goddard Space Flight Center. 

Figure 6: Cryocooler Component Schedule Delays Since April 2013 
Schedule Replan: 

[Refer to PDF for image: timeline] 

Cryocooler cold head assembly: 
April 2013: 12/2/13; 
November 2014: Delivered to Goddard Space Flight Center 11/10/14; 
delay of 11.5 months. 

Cryocooler compressor assembly: 
April 2013: 1/27/14; 
November 2014: 4/19/15[A]; delay of 14.5 months. 

Cryocooler compressor electronics assembly: 
April 2013: 1/6/14; 
November 2014: Delivered to Jet Propulsion Laboratory 10/6/14; delay 
of 9 months. 

Source: GAO analysis of NASA/Jet Propulsion Laboratory/Northrop 
Grumman data. GAO-15-100. 

[A] Current projected delivery date. 

Note: Dates above reflect delays to scheduled delivery dates from the 
April 2013 schedule replan. 

[End of figure] 

As a result of NASA concerns with the sliding compressor assembly 
delivery date, the schedule for that component was again replanned in 
July 2014. However, the planned delivery date for the compressor 
assembly has already begun slipping. According to the cryocooler 
subcontractor program manager, the delivery schedule remains overly 
optimistic and meeting it would have required perfect execution of the 
remaining work. For example, according to project, JPL, and 
subcontractor officials, the July replan's February 2015 subcontractor 
delivery date for the cryocooler compressor assembly was achievable 
only under a best case scenario where the subcontractor's 
manufacturing issues did not persist. However, since July, issues such 
as a manufacturing error and manufacturing process oversight have 
delayed delivery until at least April 2015. For example, one delay 
occurred when a subcontractor employee conducted unauthorized work on 
one of the cryocooler compressors that led to a concern of possible 
compressor contamination. Ultimately, the subcontractor determined 
that no contamination occurred and the compressor was still usable--
but not before a 3 week halt in work on the specific compressor for 
the investigation into the incident. JPL officials noted that the 
schedule impact would only be 8 days as they would be rearranging 
hardware flow and adding some holiday work to recover most of the time 
lost. Subsequently, a separate manufacturing error on a compressor 
assembly component resulted in 3 weeks of additional delay. In 
addition to these manufacturing issues, according to the subcontractor 
program manager, the subcontractor continues to encounter problems 
resulting from design immaturity of certain parts of the compressor 
assembly as they pursue the manufacturing effort. 

Given the cryocooler subcontractor's history of prior schedule delays 
and continued performance challenges, the project recognizes the 
cryocooler's late delivery to JWST's integration and testing schedules 
as a top issue to the project and is planning mitigation strategies in 
anticipation of further schedule slips--which would reduce the 
project's overall schedule reserve further. JPL's analysis of 
subcontractor performance trends on the compressor assembly schedule 
projects that the subcontractor is very unlikely to meet the current 
April 2015 delivery date and the compressor assembly is projected to 
be as late as November 2015, if past performance trends continue. The 
cryocooler used more schedule reserve than any JWST element or major 
subsystem in the past year, expending 5 months. Currently, the 
cryocooler schedule contains approximately 11 months of schedule 
reserve, of which 6 follow spacecraft integration and testing. 
[Footnote 13] Consequently, without modifications to the current 
integration and testing schedule, a subcontractor delivery of the 
compressor assembly to JPL after August 2015 would begin depleting the 
schedule reserve that follows spacecraft integration and testing. 
Through the reordering and compression of JPL's test schedule 
following delivery of the compressor assembly, the project and JPL 
have accommodated cryocooler schedule slips to date. In anticipation 
of further delays to the cryocooler compressor assembly, the project 
and Northrop Grumman are exploring impact mitigation strategies in the 
event of JPL's late delivery of the compressor assembly to spacecraft 
integration and testing, which is to begin in February 2016. 

The cryocooler subcontractor has, however, resolved other significant 
challenges related to cryocooler design and valve leakage that, as we 
have previously reported, have been the basis of prior delays. 
[Footnote 14] For example, in 2014, a verification model for one of 
the cryocooler's two compressor types was successfully tested, which 
confirmed that the design would achieve the temperatures required for 
the initial cooling stage--a key technical issue we reported on in 
2014. Additionally, the project selected a valve for the cryocooler's 
cold head assembly, the component of the cryocooler that allows it to 
cool MIRI to the 6 degree Kelvin temperature required for the 
instrument's function.[Footnote 15] 

Sunshield: 

Northrop Grumman experienced a significant manufacturing problem 
during the construction of the large composite panel that forms part 
of the sunshield's primary support structure. This problem resulted in 
a loss of overall project schedule reserve and put the sunshield on 
the project's critical path temporarily before it was overtaken by the 
cryocooler. Specifically, after the initial panel was manufactured, 
Northrop Grumman determined that it did not meet strength 
requirements. Because of the weakness identified in the panel, 
Northrop Grumman and NASA took some time to determine a root cause--
moisture was absorbed into the panel during the manufacturing process--
and implemented corrective actions for remanufacture of the panel. 
This total effort resulted in a loss of 1.75 months of total project 
schedule reserve. JWST officials expressed confidence in Northrop 
Grumman's proposed remanufacture plan, which involves the 
incorporation of a moisture barrier to the tool upon which the panel 
is made to prevent moisture transfer. According to contractor 
officials, the replacement panel, which was manufactured using the 
modified process, was completed in October 2014 and a second, smaller 
panel is to be completed by January 2015. As early as 2010, the 
project and Northrop Grumman understood that the manufacturing process 
for the sunshield's two composite panels would be complex and took 
risk mitigation steps, such as expedited manufacturing preparations, 
to address potential schedule impacts of this risk. Project officials 
maintain that these mitigation steps prevented the panel remanufacture 
from using the full 8 months of sunshield schedule reserve. 

The contractor has made progress, however, on the manufacture of other 
components of the sunshield, such as the sunshield subsystem's 
membrane layers and other structural hardware. Northrop Grumman 
successfully demonstrated the sunshield's membrane deployment concept 
by conducting a full deployment test of a full scale engineering model 
of the membrane system in July 2014. Currently, one of the five flight 
membrane layers has been manufactured and work has begun on an 
additional two. Further, Northrop Grumman has completed manufacturing 
of the main beams of the sunshield's primary support structure and 
tubes that assist in deploying the mid-boom structure. 

ISIM and OTE Preparing for OTIS: 

The ISIM schedule has lost schedule reserve since our January 2014 
report, and is a half month away from becoming the critical path of 
the project. The ISIM has used 3 months of schedule reserve out of the 
7.5 months it held in 2013, due in part to longer than expected 
integration and testing for NIRCam and NIRSpec and to additional 
anticipated testing time for the second and third cryovacuum tests. 
The first of three ISIM cryovacuum tests--in which a test chamber is 
used to simulate the near absolute zero temperatures in space--was 
completed in November 2013, with two of the observatory's four science 
instruments integrated. This first test was considered a risk 
reduction effort that provided insight into how to efficiently conduct 
the remaining two cryovacuum tests and how to analyze test results. 
The disruption of this first test from the government shutdown did 
result in some deferred testing, which increased the planned duration 
for the subsequent cryovacuum test. The second cryovacuum test, which 
included all four science instruments, began in June 2014 and finished 
in October 2014. According to project officials, the second ISIM 
cryovacuum test progressed as expected, with no technical disruptions. 
However, the schedule to prepare ISIM for the approximately 3.5-month-
long third cryovacuum test--scheduled to commence in late July 2015--
is considered very ambitious by NASA officials. Between the second and 
third cryovacuum tests, additional testing must be carried out, as 
well as the retrofitting of certain components, such as the near 
infrared detectors on two of the instruments. 

The OTE has also used schedule reserve during preparations for the 
element's integration with the ISIM. The OTE now has 4.5 months of 
schedule reserve remaining of the 7 months it held in 2013, putting it 
in the same position as ISIM in terms of currently held schedule 
reserve. Contractors successfully completed structural integrity 
testing of the OTE's primary mirror backplane support structure that 
will hold the observatory's science instruments and its 18-segment 
primary mirror. Additionally, for the 2015 readiness assessment of the 
chamber and test equipment for the subsequent integrated OTE and ISIM-
-known as OTIS--cryovacuum testing, the project is preparing a testing 
structure that incorporates a two-segment primary mirror and a 
secondary mirror. For this effort, the project took delivery of an 
engineering model of the OTE backplane's center section at Goddard 
Space Flight Center in July 2014 and completed the process of mounting 
the two primary mirror segments. 

Spacecraft: 

The spacecraft element also used 2.5 months of its schedule reserve 
over the past year, putting it a half month from the critical path. 
Specifically, the project reported that manufacturing challenges with 
the spacecraft structure required the use of schedule reserve to 
accommodate the delays. Despite these manufacturing challenges, the 
spacecraft completed significant milestones this year. Specifically, 
in January 2014, NASA assessed the spacecraft element as having 
successfully completed its critical design review, a major milestone 
indicating that the spacecraft's design maturity was deemed 
appropriate to support proceeding with full-scale fabrication, 
assembly, integration, and test. While the review required actions to 
be taken, such as the effort to further mature certain technologies, 
most of these have been addressed, though work is ongoing on a few key 
spacecraft components. Among the technologies requiring further 
maturation is a release mechanism we reported on in January 2014, 
which was causing excessive shock when engaged. This mechanism has 
been modified and shock is now within an acceptable range, though 
testing is ongoing on the modified design. 

The project continues to address other technical risks and challenges 
as it continues development and testing of the various JWST elements 
and major subsystems. For more information on selected risks and 
challenges, see appendix IV. 

Project Lacks Current Cost Risk Analyses, Creating Uncertainty About 
Adequacy of Its Budget: 

The JWST project and prime contractor cost risk analyses have not been 
updated since the project's replan in 2011 and do not account for 
newer risks. GAO best practices call for programs to regularly update 
cost risk analyses to account for new risks. We planned to conduct an 
independent updated cost risk analysis, but the prime contractor did 
not allow us the access necessary to execute our planned methodology. 
The project did subsequently agree to conduct its own cost risk 
analysis of the prime contractor's remaining work. If properly 
conducted, its analysis will provide the project with the reliable 
information necessary to gauge whether the contractor's budget is on 
target or at risk of future cost overruns. 

Project and Prime Contractor Have Not Updated Risk Analyses to Account 
for New Risks: 

Neither NASA's joint cost and schedule confidence level (JCL) analysis 
nor Northrop Grumman's cost risk analysis to support the prime 
contract has been updated since 2011. Additionally, both NASA and 
Northrop Grumman reported that they do not intend to update their risk 
analyses, despite the fact that these analyses are 3 years old and 
almost 4 years remain until launch. As we have reported previously, 
GAO's best practice criteria call for risk analyses to be updated 
regularly over the life of a project in order to ensure an estimate's 
accuracy over time and account for actual costs and new project risks. 
[Footnote 16] Without regular updates to the cost risk analysis, JWST 
decision makers and stakeholders for the project cannot be certain 
that its existing cost estimates accurately portray the project's 
financial status. Updating these analyses is particularly important in 
light of prior project optimism and newly discovered risks that 
threaten the project's cost and schedule reserves. 

Although our previous work found that the JWST project's 2011 JCL was 
not fully reliable, the JWST project continues to rely on its original 
cost estimate, which was confirmed by the JCL, to provide the basis of 
its budget estimates and requests. Based on this estimate, the JWST 
project is currently reporting that the project remains on schedule to 
launch in October 2018 and within the $8 billion developmental cost 
cap designated by Congress. The JWST project manager stated that the 
project conducts a number of programmatic and analytical activities to 
monitor the health of its cost and schedule that he believes provides 
him with accurate information on the project's status. In addition, 
the project conducted two schedule risk analyses in response to a 
recommendation we made in 2014, but analyses have not been completed 
for all schedules.[Footnote 17] The project stated one of its many 
methods of measuring cost and schedule health is analysis of earned 
value management (EVM) data.[Footnote 18] However, EVM analysis 
provides a cost and schedule projection based on past performance that 
does not reflect the potential impact of future risks. The 2011 JCL 
was the last risk analysis performed to model impacts of future risks 
on the project's cost and schedule. In 2012 and 2014 we made both a 
recommendation to NASA and provided a matter to Congress for 
consideration related to the need to update the JWST JCL to address 
weaknesses we found and to incorporate new risk information. However, 
a new JCL has not been conducted to date and NASA has indicated it 
does not intend to update the JCL for JWST. 

As a result of the significant changes that have occurred over the 
last 3 years since the 2011 replan and lacking reliable cost 
estimates, we determined that an updated cost risk analysis was 
warranted to provide Congress with insight on JWST's remaining work. 
The changes we observed include risks that have been encountered both 
by JWST and Northrop Grumman that were not included in their 2011 cost 
risk analyses. For example, the highest risk for the JWST project in 
September 2014 was the delivery date of the cryocooler to mission 
integration and testing events, which was entered into the project's 
risk database in January 2013. By October 2014, this risk had become 
one of the JWST project's top issues and is threatening cost and 
schedule reserves. Additionally, 67 percent of the risks present in 
April 2014 on Northrop Grumman's list of project risks were not 
present in September 2011. The JWST project's budget system provides a 
means of quantifying future threats to the project's budget by 
estimating a threat's potential cost and likelihood of occurrence. 
However, our analysis of the system's data since the replan found that 
almost 70 percent of reserve spending has not been anticipated as 
threats by the budget system. Without an analytical framework that 
comprehensively captures future risks into potential project costs, 
the project's estimates do not represent a comprehensive portrayal of 
the financial status of the project. As a consequence, the project and 
Congress do not have all the data necessary to be confident in the 
project's projected budgetary needs. 

To address the lack of updated information, GAO planned to conduct an 
updated cost risk analysis on the Northrop Grumman prime contract, 
given that it represents the most significant contractor work 
remaining on the project. The methodology for this analysis, which has 
been cited as an important tool by NASA's cost estimating community, 
would have provided an updated level of confidence associated with the 
project meeting its cost estimate and allowed decision makers to 
determine if additional cost reserves were warranted to mitigate 
future risks. However, we were unable to execute our methodology 
because Northrop Grumman declined to allow GAO to conduct an 
independent, unbiased analysis. GAO best practices for cost estimating 
require anonymous interviews with technical experts on a project to 
discuss the project's risks. We were not able to create this 
environment because while Northrop Grumman officials were open to 
allowing us to conduct the analysis, they did not allow us to 
interview employees without a manager present. Anonymity is critical 
to the methodology in order to ensure the interviews are unbiased and 
provide a comprehensive portrayal of project risks. 

The JWST project and NASA disagreed with the approach of an 
independent cost risk analysis conducted by an outside organization at 
this point in the project. Additionally, neither believes that an 
organization external to NASA can fully comprehend the project's 
risks, and any such analysis would be overly conservative due to the 
complexities of the risks. As a result, they asserted that the results 
of any outside analysis would not be representative of the real risk 
posture of the project. GAO's best practices call for cost estimates 
to be compared to independent cost estimates in addition to being 
regularly updated. Without an independent and updated analysis, both 
the committee members' and NASA's oversight and management of JWST 
will be constrained since the impact of newer risks have not been 
reflected in key tools, including the cost estimate. 

In response to our concerns, NASA indicated it would conduct its own 
cost risk analysis for the committee members of the Northrop Grumman 
contract. The project already had plans to use additional EVM 
analysis, which the project does plan to use as a management tool, 
that incorporates risks to provide a health check on the project's 
cost reserves. However, a NASA project official stated that data from 
the cost risk analysis was for the purpose of providing cost 
information to committee members, but the JWST project does not intend 
to use it to manage the project. Additionally, the project has 
indicated that it will consider updating the cost risk analysis in the 
future, but presently has not committed to regularly updating the 
analysis. To maintain quality cost estimates over the life of a 
project, best practices state that cost risk analyses be updated 
regularly to incorporate new risks and be used in conjunction with EVM 
analysis to validate cost estimates. If the project does not follow 
best practices in conducting its cost risk analysis or use it to 
manage the project, the resulting information may be unreliable and 
not substantively provide insight into JWST's potential costs to allow 
either Congress or project officials to take any warranted action. 

Contractors Have Been Responsive to Incentives, but Final Evaluation 
May Not Reflect Total Contract Performance: 

The JWST project has used NASA's award fee structure to implement 
incentives that align with conferee's priorities for cost and schedule 
for JWST and contractors have been responsive to these incentives, for 
example by making changes to key practices to improve communication. 
When completed, JWST's prime contract will have spanned almost two 
decades, during which there have been and could be future significant 
variances in contractor performance and key management changes among 
program officials. While NASA's award fee guidance and its procurement 
regulations require the development of explicit evaluation factors to 
evaluate performance, evaluation criteria are not specified for the 
final evaluation of total contract performance in the project's 
performance evaluation plans.[Footnote 19] Without clear criteria in 
award fee plans, it is unclear how JWST officials will assess 
contractor performance over the life of the contracts, as is required 
per NASA regulations--particularly for the prime contract, which will 
include a significant cost overrun, three launch delays, and almost 20 
years of varying performance. 

NASA's Cost-Plus-Award-Fee Process to Assess Contractor Performance: 

Award fee contracts provide contractors the opportunity to obtain 
monetary incentives for performance in designated areas, which can be 
weighted differently at NASA's discretion. For JWST's award fee 
contracts, these areas include cost, schedule, technical, and business 
management and are established in the contracts' performance 
evaluation plans. Award fees are used when key elements of performance 
cannot be defined objectively, and, as such, require the project 
officials' judgment to assess contractor performance. Table 1 provides 
a general outline of the performance evaluation process. 

Table 1: NASA's Process for Evaluation of Award Fees: 

1. Performance evaluation board may establish areas of emphasis for 
the award fee period and provide them to the contractor. 

2. Award fee evaluation period ends (which usually should be at least 
six months) and performance monitors (NASA technical experts) assess 
the contractor's performance against the areas of emphasis and 
criteria in the performance evaluation plan. Contractor should also be 
provided the opportunity to provide a self-assessment. 

3. Performance evaluation board recommends to the fee determining 
official a performance rating for the contractor based on the 
performance monitors' assessments, the contractor's self-assessment, 
and its assessment of contractor performance. 

4. Fee determining official determines the contractor's performance 
rating for the period, based on information provided by the 
performance evaluation board and the official's assessment of 
contractor performance. 

5. Contractor is paid a percentage of the available award fee based on 
their performance rating for the period. Interim payments cannot 
exceed 80 percent of the available fee. 

Source: GAO analysis of policy and guidance. GAO-15-100. 

[End of table] 

Contractors Responsive to NASA's Use of Cost and Schedule Incentives: 

The JWST project has used NASA's award fee structure to implement 
incentives that align with conferee's priorities for cost and schedule 
for JWST. The JWST project has modified the fee structure for one of 
its two cost-plus-award-fee contracts, the Northrop Grumman prime 
contract, to enhance cost and schedule incentives. In December 2013, 
the JWST project completed contract negotiations with Northrop Grumman 
to incorporate the launch delay to October 2018. During negotiations, 
NASA and Northrop Grumman agreed to replace a $56 million on-orbit 
incentive fee with award fees to increase the available award fee for 
the entire contract to almost a quarter of a billion dollars. While 
award fees for the Northrop Grumman contract are intended to reward 
performance for cost, schedule, and meeting technical requirements, 
the on-orbit incentive would have been based on exceeding technical 
requirements. Both JWST and Northrop Grumman officials stated 
restructuring the award fee incentives gave NASA more flexibility to 
incentivize the contractor to prioritize cost and schedule performance 
over exceeding technical requirements. In addition, a $6 million 
schedule incentive fee was added to the contract that would be paid to 
the contractor for achieving three key deliveries ahead of schedule. 
[Footnote 20] According to NASA officials, these modifications were 
intended to help ensure the Northrop Grumman contract contained proper 
cost and schedule incentives. 

For JWST's two award fee contracts--Northrop Grumman and Exelis--NASA 
award fee letters of award fee periods from February 2013 to March 
2014 indicate that contractors have been responsive to interim award 
fee period criteria provided by NASA. Also, contractor officials from 
both award fee contracts confirmed that they pay close attention to 
this guidance in prioritizing their work. For example, Northrop 
Grumman officials reported that they had made specific changes to 
improve communications in direct response to this guidance which was 
validated by award fee letters from NASA. 

History of Varied Performance Is Not Included in Criteria for Final 
Evaluation of Contractor Performance: 

The JWST project does not have sufficient award fee guidance for two 
of its largest contracts, as required by the NASA Federal Acquisition 
Regulation Supplement. Our finding is consistent with the NASA Office 
of Inspector General findings that cost-plus-award-fee contracts did 
not include explicit criteria related to the final 
evaluation.[Footnote 21] JWST's criteria for assessing Northrop 
Grumman's prime contract do not specify that in the final evaluation 
performance metrics for cost and schedule should reflect performance 
prior to the replan. If performance prior to the replan is not 
considered, the contractor could receive award fee that does not 
reflect its performance over the life of the contract. To see how 
JWST's cost-plus-award-fee contracts compare to NASA policy and 
guidance for both contracts, see table 2. 

Table 2: James Webb Space Telescope Award Fee Plans Comparison with 
NASA Final Evaluation Guidance: 

NASA policy and guidance: Final evaluation will assess performance 
over the life of the contract (NASA Federal Acquisition Regulation 
(FAR) Supplement) and take into consideration all significant events, 
including launch delays (NASA Award Fee Contracting Guide); 
Northrop Grumman performance evaluation plan: 
* Does not specify that launch delays should be included as criterion 
for the final evaluation; 
* Does not specify that in the final evaluation performance metrics 
for cost and schedule should reflect contractor's performance prior to 
the 2011 replan; 
Exelis performance evaluation plan: 
* Three of four performance areas include specific criteria for 
interim evaluations and do not include additional criteria for the 
final evaluation. 

NASA policy and guidance: Explicit evaluation criteria must be 
established for all award fee periods, which includes the final 
evaluation (NASA FAR Supplement); 
Northrop Grumman performance evaluation plan: 
* Does not state whether criteria apply to the final evaluation; 
Exelis performance evaluation plan: 
* Three of four performance areas do not include specific criteria for 
the final evaluation. 

Source: GAO analysis of NASA policy and contractor evaluation plans. 
GAO-15-100. 

[End of table] 

On contracts that result in the delivery of an end-item product such 
as JWST, NASA makes interim payments to contractors after each 
evaluation period has been completed. While the contractor receives 
the money from the government in the short-term, the contractor has 
not officially earned it until the end of the contract, when a final 
evaluation occurs. The final evaluation supersedes all interim 
evaluation scores, which could result in the contractor earning more 
or less fee than that earned on the basis of the aggregate of the 
interim scores.[Footnote 22] 

According to NASA procurement officials, guidance and criteria for the 
final evaluations should be found in the contract's performance 
evaluation plan. However, the performance evaluation plan for Northrop 
Grumman does not specify that in the final evaluation performance 
metrics for cost and schedule should include contractor performance 
prior to the 2011 replan. Having criteria to evaluate performance 
prior to 2011 is particularly important because that time period 
includes a significant cost overrun and launch delay, turnover in JWST 
officials managing the contract and making award fee decisions, and 
lower contractor performance ratings. Additionally, NASA attributed 
responsibility to Northrop Grumman for a portion of the cost and 
schedule overrun that contributed to the latest launch delay of over 4 
years and significant related cost growth. This is particularly 
important because while the period of performance since the replan 
represents only 58 percent of the contract's cost, it represents 81 
percent of the contract's award fees due to the contract's conversion 
of the on-orbit incentive into award fee in 2011. This weighting 
magnifies the importance of the final evaluation and lessens the 
relative importance of performance prior to the replan. 

Exelis's cost-plus-award-fee contract also does not contain specific 
final evaluation criteria. While the contract began almost 8 years 
later than the Northrop Grumman contract and is substantially smaller 
in dollar value, the performance period will incorporate at least one 
cost overrun and some varying performance. Without sufficient criteria 
the contract's final evaluation may also be conducted without taking 
performance over the life of the contract into account. 

Conclusions: 

As the JWST project enters fiscal year 2015 with under 4 years until 
launch, project officials have made progress building significant 
pieces of hardware, but continue to face numerous technical challenges 
that have eroded schedule reserve for the first time since the 2011 
replan. While some use of schedule reserve is expected, JWST is a 
schedule-driven program and preserving the reserve is critical. The 
project will have fewer opportunities for the types of workarounds it 
has been able to incorporate thus far because initiating work is often 
dependent on the successful and timely completion of the prior work. 
Further, the project has limited funds available in the short-term to 
use to preserve schedule. Effectively managing and making targeted use 
of existing cost reserves to preserve schedule reserve until fiscal 
year 2016, when more reserves are available, is critical to 
maintaining the planned launch date in October 2018. 

Three years after the 2011 replan, project officials and the prime 
contractor have not recently assessed risks for the potential impact 
to cost to ensure they are measuring progress against a valid 
baseline. While the project reports that JWST remains on schedule and 
within its developmental cost cap, this is based on outdated analysis 
and risks that were present in 2011. Without incorporating the myriad 
of new and sometimes significant risks into any assessment of cost 
risk, the project's cost risk posture cannot be independently 
validated. While NASA has chosen to conduct a cost risk analysis of 
Northrop Grumman's remaining work itself, its quality hinges on 
whether NASA chooses to follow best practices. Doing so would better 
ensure that the results of the analysis would be reliable and would 
provide an accurate portrayal of status that can then be used to 
ensure better informed decisions are being made about project funding 
forward, at least for Northrop Grumman's remaining work. Furthermore, 
because JWST is one of NASA's most expensive and complex science 
projects and any issues can have significant impacts to other science 
projects, going above and beyond what is required per policy or 
guidance is one small step that could help demonstrate NASA's 
commitment to the success of JWST and to improving acquisition 
management outcomes. 

Effectively using the award fee evaluation process to thoroughly and 
fairly assess contractor performance is critical given the size and 
long duration of the JWST contracts, in particular its prime contract 
with Northrop Grumman. Without clear guidance for both NASA officials 
and contractors on how to conduct the final award fee evaluation and 
account for performance over the life of the project, the final 
determination of all award fees for two of JWST's largest contracts 
may not accurately reflect the contractor's performance. Fairly 
assessing that performance over the life of the contract is especially 
critical for JWST given the variance in performance that Northrop 
Grumman has shown from contract award in 2002 to the present. Not 
doing so could result in Northrop Grumman receiving a higher 
percentage of award fee than interim scores would warrant. 

Recommendations for Executive Action: 

We recommend that the NASA Administrator take the following two 
actions: 

* In order to provide additional information and analyses to 
effectively manage the program and account for new risks identified 
after the 2011 replan, direct JWST project officials to follow best 
practices while conducting a cost risk analysis on the prime contract 
for the work remaining and ensure the analysis is updated as 
significant risks emerge. 

* In order to ensure JWST's award fee contracts' final evaluations 
thoroughly and fairly evaluate contractor performance over the life of 
the contract and to provide clarity to the process that will be used 
for the final evaluation, direct JWST project officials, in 
conjunction with the performance evaluation board for JWST and the 
Goddard Space Flight Center fee determining official, to modify 
performance evaluation plans for its award fee contracts to ensure 
they: 

(a)specify evaluation criteria that reflects total contract 
performance in advance of the final evaluation, and: 

(b)clearly describe the process the performance evaluation board and 
fee determining official will use to evaluate contractor performance 
in the final evaluation. 

Agency Comments and Our Evaluation: 

NASA provided written comments on a draft of this report. These 
comments are reprinted in appendix V. In responding to a draft of this 
report, NASA partially concurred with one recommendation and concurred 
with another. 

NASA partially concurred with our recommendation directing the JWST 
project to follow best practices while conducting a cost risk analysis 
on the prime contract for the work remaining and ensure it is updated 
as significant risks emerge. In response to this recommendation, NASA 
stated that the JWST program and project use a range of tools to 
assess all major contractors' performance and that the approach the 
project has in place is consistent with best practices. NASA indicated 
that it will continue to use the tools currently in place to manage 
JWST. Furthermore, NASA noted the JWST project initiated a cost risk 
analysis of the prime contract that is incorporating best practices 
and will update it when required by NASA policy. 

NASA policy did not require cost risk analyses when the JWST project 
began, therefore the project is also not required to update such an 
analysis. Policy notwithstanding, NASA and GAO's best practices do 
support conducting a cost risk analysis and updating it as product 
development progresses. Specifically, NASA's cost estimating handbook 
states that while the product is being designed, developed, and 
tested, there are changes which can impact the estimate and the risk 
assessment. It is critical to capture these changes to maintain a 
realistic program estimate now and in the future. During this phase, 
programmatic data may have just as much of an impact on the estimate 
and risk assessment as technical data. We acknowledge in the report 
the JWST project's use of EVM data, but reiterate that it does not 
reflect the potential impact of future risks. As our report notes, the 
JWST project has only started two of the five integration and test 
periods and risks have changed significantly since a cost risk 
analysis was last conducted on the prime contractor's work. As the 
telescope is one of NASA's most complex and expensive projects, both 
JWST and NASA officials have a tremendous stake in the success of the 
JWST project. As a result, an important best practice NASA can follow 
is to conduct a cost risk analysis in the future so that realistic 
estimates are being communicated to Congress, especially as new risks 
emerge in future testing. We encourage JWST officials to rely not on 
policy direction but on good management practices to spur them to 
update the cost risk analysis they are conducting on the prime 
contractor's work as significant risks continue to emerge. 

NASA concurred with our recommendation directing the JWST project, in 
conjunction with the performance evaluation board for JWST and the 
Goddard Space Flight fee determining official, to modify its 
performance evaluation plans for its award fees to ensure they (a) 
specify evaluation criteria that reflects total contract performance 
in advance of the final evaluation, and (b) clearly describe the 
process the performance evaluation board and fee determining official 
will use to evaluate contractor performance in the final evaluation. 
NASA indicated it will revise the performance evaluations plans to 
clarify that criteria in the existing plans applies to both interim 
periods and the final award fee determination and will revise the 
plans to specify the process that will be used to conduct the final 
evaluation and final determination. Once complete, these actions 
should address our recommendation to clarify the criteria and the 
process that will be used so a thorough and fair final evaluation may 
occur that incorporates information from performance over the life of 
each contract. 

NASA also provided technical comments, which have been addressed in 
the report, as appropriate. 

We are sending copies of the report to NASA's Administrator and 
interested congressional committees. In addition, the report will be 
available at no charge on GAO's website at [hyperlink, 
http://www.gao.gov]. 

Should you or your staff have any questions on matters discussed in 
this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. 
Contact points for our Offices of Congressional Relations and Public 
Affairs may be found on the last page of this report. GAO staff who 
made major contributions to this report are listed in appendix VI. 

Signed by: 

Cristina T. Chaplain: 
Director: 
Acquisition and Sourcing Management: 

List of Committees: 

The Honorable Barbara A. Mikulski: 
Chairwoman: 
The Honorable Richard C. Shelby: 
Ranking Member: 
Subcommittee on Commerce, Justice, Science, and Related Agencies: 
Committee on Appropriations: 
United States Senate: 

The Honorable Frank R. Wolf: 
Chairman: 
The Honorable Chaka Fattah: 
Ranking Member: 
Subcommittee on Commerce, Justice, Science, and Related Agencies: 
Committee on Appropriations: 
House of Representatives: 

The Honorable Lamar Smith: 
Chairman: 
The Honorable Eddie Bernice Johnson: 
Ranking Member: 
Committee on Science, Space, and Technology: 
House of Representatives: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

Our objectives were to assess the extent to which (1) technical 
challenges are impacting the James Webb Space Telescope (JWST) 
project's ability to stay on schedule and budget, (2) budget and cost 
estimates reflect current information about project risks, and (3) the 
project uses award fee contracts to motivate and assess contractor 
performance. 

To assess the extent to which technical challenges are impacting the 
JWST project's ability to stay on schedule and budget, we reviewed 
project and contractor schedule and cost documentation, analyzed the 
progress made against planned milestones established during the 
project's replan in 2011, and held interviews with program, project, 
and contractor officials. We reviewed project monthly status reviews, 
quarterly provided flight program reviews, information from the 
project's risk database, and other technical documentation to gain 
insights on the project's progress since our last report in January 
2014. In addition, we reviewed briefings and schedule documentation 
provided by project and contractor officials. These documents included 
information on the project's technological challenges and risks, 
mitigation plans, timelines for addressing these risks and challenges, 
and available schedule reserve. We analyzed the project's reported 
schedule reserve data across multiple elements and major subsystems. 
We also interviewed project officials to clarify information and to 
obtain additional information on element and major subsystem level 
risks and technological challenges. Further, we attended flight 
program reviews at the National Aeronautics and Space Administration 
(NASA) headquarters on a quarterly basis where the current status of 
the program was briefed to NASA headquarters officials outside of the 
project. Finally, we interviewed officials from Exelis, the Jet 
Propulsion Laboratory, and different divisions of Northrop Grumman 
Aerospace Systems concerning risks and challenges on the elements and 
major subsystems, instruments, or components they were developing. 

To assess the extent to which budget and cost estimates reflect 
current information about project risks, we reviewed project, 
contractor, and subcontractor documentation and held interviews with 
officials from all of these different organizations. We reviewed 
project monthly status reviews, quarterly provided flight program 
reviews, and contractor information on the potential cost to address 
identified risks. To identify whether JWST's current cost management 
practices were sufficient, we reviewed its earned value management 
(EVM) and budget systems and compared them to GAO best 
practices.[Footnote 23]To identify new risks that had emerged since 
the 2011 replan, we conducted analysis to compare the prime 
contractor's current risk documentation to the risk documentation at 
the time of the 2011 replan. Further, we analyzed the project's cost 
reserve data since the 2011 replan to identify where the project has 
expended cost reserves and predicted threats, and conducted interviews 
with contractors to discuss their cost reserve management practices. 
We also examined and analyzed EVM data from several contractors to 
identify trends in cost and schedule performance, entering both high-
level and subsystem-level monthly contractor EVM data into a GAO-
developed spreadsheet, which includes checks to ensure the EVM data 
provided was reliable. We discussed our assessments of the project's 
data and analysis with JWST program and project officials from the 
Goddard Space Flight Center. 

To assess the extent to which the project uses award fee contracts to 
motivate and assess contractor performance, we reviewed the project's 
documentation, conducted contract file reviews, examined performance 
evaluation plans for two contractors, and reviewed the NASA Office of 
Inspector General's report on NASA's use of award fees.[Footnote 24] 
We also examined the relevant portions related to award fees of the 
Federal Acquisition Regulation (FAR), NASA FAR Supplement, and NASA 
Award Fee Contracting Guide. We conducted interviews with project 
officials, NASA Office of Procurement officials, the JWST fee 
determining official, and contractor officials to obtain information 
on the project's use of contract incentives and implementation of 
award fee evaluations. We evaluated the adequacy of JWST's performance 
evaluation criteria and processes by comparing the processes and 
criteria in the project's performance evaluation plans and other 
documentation to NASA's policies, regulations, and guidance. We 
interviewed project and NASA headquarters officials to clarify 
information and obtain additional information on NASA's typical award 
fee processes and plans for evaluating JWST's contractors. 

Our work was performed primarily at NASA headquarters in Washington, 
D.C., Goddard Space Flight Center in Greenbelt, Maryland, and Johnson 
Space Center in Houston, Texas. We also visited the Jet Propulsion 
Laboratory in Pasadena, California; Northrop Grumman Aerospace Systems 
in Redondo Beach, California; the Defense Contract Management Agency 
in Redondo Beach, California; and the Space Telescope Science 
Institute in Baltimore, Maryland. 

We conducted this performance audit from February 2014 to December 
2014 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit 
to obtain sufficient, appropriate evidence to provide a reasonable 
basis for our findings and conclusions based on our audit objectives. 
We believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

[End of section] 

Appendix II: Elements and Major Subsystems of the James Webb Space 
Telescope (JWST) Observatory: 

Figure 7: James Webb Space Telescope: Interactive Information: 

[Refer to PDF for image: 5 illustrations and accompanying information] 

Integrated Science Instrument Module; 
Acronym: ISIM; 
Contractor/Center: Goddard Space Flight Center; 
Description: Combines the 4 instruments. 

Mid Infrared Instrument; 
Acronym: MIRI; 
Contractor/Center: Jet Propulsion Lab and European Consortium; 
Description: Science instrument. 

Near Infrared Spectrograph: 
Acronym: NIRSpec; 
Contractor/Center: European Space Agency; 
Description: Science instrument. 

Fine Guidance Sensor/Near-Infrared Imager and Slitless Spectrograph: 
Acronym: FGS/NIRISS; 
Contractor/Center: Canadian Space Agency; 
Description: Telescope guider and Science instrument. 

Near Infrared Camera: 
Acronym: NIRCam; 
Contractor/Center: University of Arizona;
Description: Science instrument and Wave Front Sensor. 

Spacecraft: 
Contractor/Center: Northrop Grumman Aerospace Systems; 
Description: Contains the power, communications, and avionics needed 
to operate the observatory. Contains the cryocooler needed to achieve 
MIRI operational temperatures approximating 7 Kelvin. 

Optical Telescope Element: 
Acronym: OTE; 
Contractor/Center: Northrop Grumman Aerospace Systems; 
Description: 18 primary mirror segments, secondary mirror, tertiary 
mirror, backplane support structure. 

Optical Telescope & Integrated Science Instrument Module: 
Acronym: OTIS (OTE+ISIM); 
Contractor/Center: Goddard Space Flight Center; 
Description: Hardware configuration created when OTE and ISIM are 
integrated. 

Sunshield: 
Contractor/Center: Northrop Grumman Aerospace Systems; 
Description: Tennis court sized series of 5 thin membranes, provides 
passive cooling to achieve operational temperatures approximating 45 
Kelvin for the OTE and ISIM. 

Sources: GAO (analysis); NASA (data and images). GAO-15-100. 

[End of figure] 

[End of section] 

Appendix III: Organizational Chart for the James Webb Space Telescope 
Program: 

[Refer to PDF for image: organizational chart] 

Top level: 
NASA HQ Associate Administrator. 

Second level: 
NASA HQ JWST Program Director: 
* NASA HQ Science Mission Directorate (Execution oversight). 

Third level: 
NASA HQ JWST Program Manager. 

Fourth level: 
Goddard JWST Project Manager: 
* Goddard Systems Engineer (Matrix organization); 
* Goddard Center Director (Execution oversight); 
* Goddard JWST Project Scientist (Programmatic). 

Fifth level: 
Goddard In-house project contributions: 
* ISIM; 
* OTIS; 
* Johnson Space Center Chamber A; 
* Integration & Test. 

Major contractor contributions: 
University of Arizona: 
* NIRCam; 
* Detectors; 
Northrop Grumman: 
* OTE; 
* Spacecraft; 
* Sunshield; 
* Integration & Test; 
Jet Propulsion Lab: 
* MIRI; 
* Cryocooler[A]; 
Space Telescope Science Institute: 
* Ground Systems; 
Exelis: 
* OTIS Integration & Test. 

International partner contributions: 
European Space Agency: 
* NIRSpec; 
* Launch Vehicle; 
Canadian Space Agency: 
* FGS/NIRISS; 
European Consortium: 
* MIRI. 

Source: GAO analysis of NASA information. GAO-15-100. 

FGS/NIRISS: Fine Guidance Sensor/Near-Infrared Imager and Slitless 
Spectrograph; 
ISIM: Integrated Science Instrument Module; 
MIRI: Mid-Infrared Instrument; 
NIRCam: Near-Infrared Camera; 
NIRSpec: Near-Infrared Spectograph; 
OTE: Optical Telescope Element; 
OTIS: OTE + ISIM Integrated. 

[A] The Jet Propulsion Laboratory is the contractor for the 
development of the cryocooler, but has subcontracted most of the work 
to a different division of Northrop Grumman than the one that is 
responsible for OTE, spacecraft, and sunshield development. 

[End of figure] 

[End of section] 

Appendix IV: Other Technical Issues for the James Webb Space Telescope: 

Figure 8: Cryocooler Compressor Vibration: 

[Refer to PDF for image: illustration and associated data] 

Components depicted: 

Optical Telescope Element (OTE): 
Spacecraft (~300K); 
Sun Facing Surface; 
Cryocooler electronics; 
Sunshield; 
40K Integrated Science Instrument Module (ISIM); 
6 Kelvin (K) Mid-infrared instrument (MIRI): 
6K coldhead; 
18K coldhead. 

Issue: Compressor Vibration: 

Background: The James Webb Space Telescope (JWST) project's modeling 
suggests that cryocooler generated vibration on the observatory 
exceeds permissible levels. The project's predictive modeling for this 
vibration contains considerable uncertainty, as various relevant tests 
that will later inform the model have yet to be carried out. According 
to project officials, the current model is very conservative and may 
therefore indicate greater levels of vibration than will actually 
occur. 

Risk: Such vibration could potentially degrade the quality of optical 
images taken by the observatory. 

Current Status: The project has established a working group for this 
problem, exploring several mitigation strategies: adjusting the on-
orbit cryocooler speed to mitigate adverse vibrations, increasing the 
allocation for allowable cryocooler-induced vibration, and developing 
higher fidelity models for predicting vibration. Information for 
validating the model will become available at different junctures of 
the integration and testing process. 

Source: GAO presentation of NASA data. GAO-15-100. 

[End of figure] 

Figure 9: Excess Mid-Infrared Stray Light: 

[Refer to PDF for image: illustration and associated data] 

Components depicted: 

Thermal emission from optics; 
Thermal emission from sunshield. 

Issue: Excess Mid-Infrared (MIR) Stray Light: 

Background: Heat emitted from JWST's observatory optics and surfaces, 
such as the sunshield, radiates at various wavelengths, among which is 
mid-infrared light. The project's modeling predicts that mid-infrared 
light emissions emanating from the sunshield will exceed project-
established permissible levels. Such mid-infrared light is at a 
wavelength that is detectable by the Mid-Infrared Instrument (MIRI), 
but not by the other three JWST science instruments. 

Risk: Mid-infrared stray light beyond permissible levels could 
adversely impact scientific observations from MIRI, as the stray light 
creates a visible “noise” impacting the optical images taken at these 
wavelengths. 

Current Status: The project believes that a measured relaxation of the 
stray light requirements to permit the currently predicted level of 
stray mid-infrared light at the wavelength of concern might not result 
in a loss of any project science objectives, although the project is 
still developing analysis to validate this contention. Such a 
relaxation of the stray light requirements would likely necessitate an 
increase in MIRI's image-taking exposure time for certain 
observations. Recent testing of the sunshield engineering model is 
providing the project with a greater ability to predict the likely 
stray light to be encountered. 

Source: NASA. GAO-15-100. 

[End of figure] 

[End of section] 

Appendix V: Comments from the National Aeronautics and Space 
Administration: 

National Aeronautics and Space Administration: 
Office of the Administrator: 
Washington, DC 20546-0001: 

December 1, 2014: 

Ms. Cristina Chaplain: 
Director: 
Acquisition and Sourcing Management: 
United States Government Accountability Office: 
Washington, DC 20548: 

Dear Ms. Chaplain: 

The National Aeronautics and Space Administration (NASA) appreciates the
opportunity to review and comment on the Government Accountability 
Office (GAO) draft report entitled, "James Webb Space Telescope: 
Project Facing Increased Schedule Risk with Significant Work 
Remaining" (GAO-15-100). 

During the 2011 James Webb Space Telescope (JWST) replan, the JWST 
project established 13 months of funded schedule to achieve the 2018 
Launch Readiness Date (LRD), which was above the guidelines used for 
NASA projects. The reserve was distributed throughout the span of the 
project to allow for resolution of issues during more stringent lower 
level testing. For example, sophisticated cryogenic vacuum tests are
conducted at lower and simpler levels of assembly and complexity, 
before integrated flight article testing occurs. Today, JWST has II 
months of funded schedule reserve. The JWST project will continue the 
approach of incremental testing throughout the remainder of the 
project allowing the project to continually evaluate performance and 
make course corrections as needed rather than wait until higher level 
of testing is completed. Though no project can prevent all possible 
delays encountered during integration and test, the practices and 
procedures put in place by JWST have, over the past three years, 
proven to be effective for JWST, NASA's most complex cryogenic mission. 

In the draft report, GAO addresses the following two recommendations 
to the NASA Administrator: 

Recommendation 1: In order to provide additional information and 
analyses to effectively manage the program and account for new risks 
identified after the 2011 replan, direct the JWST project to follow 
best practices while conducting a cost risk analysis on the prime 
contract for the work remaining and ensure it is updated as 
significant risks emerge. 

Management's Response: NASA partially concurs with this recommendation.
The NASA JWST program and project use a range of tools to assess not 
only Northrop Grumman Aerospace Systems (NGAS) performance, but all 
major JWST contractor performance. These analyses include monthly 
Earned Value Measurement data evaluation, Estimate-at-Complete 
analyses which incorporate the current risk posture, independent 
analysis of those data, and monthly schedule analysis and health 
metric assessment. NASA believes this approach is consistent with best 
practices and will continue to use those tools to effectively manage the
JWST program. The NASA JWST project initiated a cost-risk analysis of 
the NGAS 'prime contract' portion of JWST, and will complete this 
analysis using best practices. NASA will update this cost-risk 
analysis when required by NASA policy. 

Estimated Completion Date: February 28, 2015. 

Recommendation 2: In order to ensure JWSTs award fee contracts' final
evaluations thoroughly and fairly evaluate contractor performance over 
the life of the contract and to provide clarity to the process that 
will be used for the final evaluation, direct the JWST project, in 
conjunction with the Performance Evaluation Board for JWST and the 
Goddard Space Flight Center Fee Determining Official, to modify its 
performance evaluation plans for its award fee contracts to ensure 
they: 

a) specify evaluation criteria that reflects total contract 
performance in advance of the final evaluation, and, 

b) clearly describe the process the performance evaluation board and fee
determining official will use to evaluate contractor performance in 
the final evaluation. 

Management's Response: NASA concurs with this recommendation and will
revise the performance evaluation plans for the NASA-NGAS and NASA-
Exelis contracts to clarify that the criteria contained in the 
existing performance evaluation plans applies to both the interim 
evaluations and to total contract performance for the final award fee 
determination. NASA will also revise the plans to specify the process 
that will be used to conduct the final evaluation and award fee 
determination. 

Estimated Completion Date: March 30, 2015. 

Again, thank you for the opportunity to comment on this draft report. 
If you have any questions or require additional information, please 
contact Ray Taylor at 202-358-0766. 

Sincerely, 

Signed by: 

Robert M. Lightfoot: 
Associate Administrator: 

[End of section] 

Appendix VI: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Cristina Chaplain, (202) 512-4841 or chaplainc@gao.gov. 

Staff Acknowledgments: 

In addition to the contact named above, Shelby S. Oakley, Assistant 
Director; Karen Richey, Assistant Director; Jason Lee, Assistant 
Director; Patrick Breiding; Laura Greifner; Samuel P. Harris; Jonathan 
Mulcare; Kenneth E. Patton; Timothy N. Shaw; Sylvia Schatz; Jay 
Tallon; and Ozzy Trevino made key contributions to this report. 

[End of section] 

Footnotes: 

[1] H.R. Rep. No. 112-284, at 254 (2011) (Conf. Rep.). 

[2] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: Mar. 2, 
2009). 

[3] The electromagnetic spectrum consists of the wavelengths of all 
visible and invisible light. The infrared part of the spectrum, also 
known as radiant heat, has wavelengths that go from about 0.75 microns 
to a few hundred microns. The Hubble is designed to operate primarily 
in the ultraviolet and visible wavelengths of the spectrum from 0.1 to 
0.8 microns. Humans cannot see in the ultraviolet region. 

[4] The JWST project is divided into three major segments: the launch 
segment, the ground segment, and the observatory segment. The hardware 
configuration created when the Optical Telescope Element and the 
Integrated Science Instrument Module are integrated, referred to as 
OTIS, is not considered an element by NASA, but we categorize it as 
such for ease of discussion. The ground segment is not pictured in the 
interactive graphic. 

[5] Goddard Space Flight Center, Goddard Procedural Requirements 
7120.7 (May 4, 2008). 

[6] James Webb Space Telescope (JWST) Independent Comprehensive Review 
Panel (ICRP): Final Report (Oct. 29, 2010). 

[7] During formulation, the basic project concept is defined, as well 
as the technologies, time, funding, and other resource requirements. 

[8] The JCL is a quantitative probability analysis that requires the 
project to combine its cost, schedule, and risks into a complete 
quantitative picture to help assess whether the project will be 
successfully completed within cost and on schedule. NASA introduced 
the analysis in 2009, and it is among the agency's initiatives to 
reduce acquisition management risk. The move to probabilistic 
estimating marks a major departure from NASA's prior practice of 
establishing a point estimate and adding a percentage on top of that 
point estimate to provide for contingencies. NASA's procedural 
requirements state that Mission Directorates should plan and budget 
programs and projects based on a 70 percent JCL, or at a different 
level as approved by the Decision Authority of the Agency Program 
Management Council, and any JCL approved at less than 70 percent must 
be justified and documented. NASA Procedural Requirements (NPR) 
7120.5E, NASA Space Flight Program and Project Management 
Requirements, paragraph 2.4.4 (Aug. 14, 2012). 

[9] GAO, James Webb Space Telescope: Project Meeting Commitments but 
Current Technical, Cost, and Schedule Challenges Could Affect 
Continued Progress, [hyperlink, http://www.gao.gov/products/GAO-14-72] 
(Washington, D.C.: Jan. 8, 2014). 

[10] GAO, James Webb Space Telescope: Actions Needed to Improve Cost 
Estimate and Oversight of Test and Integration, [hyperlink, 
http://www.gao.gov/products/GAO-13-4] (Washington, D.C.: Dec. 3, 2012). 

[11] The 2011 baseline plan had 13 months of schedule reserve. 
However, by accelerating some work the project was able to increase 
the schedule reserve to 14 months in June 2012. 

[12] [hyperlink, http://www.gao.gov/products/GAO-14-72]. 

[13] The cryocooler schedule reserve number has not been updated to 
reflect any changes since November 2014. 

[14] [hyperlink, http://www.gao.gov/products/GAO-13-4]; [hyperlink, 
http://www.gao.gov/products/GAO-14-72]. 

[15] We reported in January 2014 that the subcontractor had not fully 
resolved a design issue with valves in the cold head assembly. In July 
2014, the project concluded an investigation into a 2013 valve leak 
occurrence in testing and determined that the valves are nonetheless 
suitable for flight, although the project modified the operational 
sequences for the valves to reduce leakage risk. 

[16] [hyperlink, http://www.gao.gov/products/GAO-13-4]; [hyperlink, 
http://www.gao.gov/products/GAO-14-72]; [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP]. 

[17] [hyperlink, http://www.gao.gov/products/GAO-14-72]. 

[18] Earned value management is a project management tool that 
integrates the technical scope of work with schedule and cost elements 
and compares the value of work accomplished in a given period with the 
value of the work expected in that period. When used properly, earned 
value management can provide objective assessments of project 
progress, produce early warning signs of impending schedule delays and 
cost overruns, and provide unbiased estimates of anticipated costs at 
completion. As a best practice, the work breakdown structure should 
match the schedule, cost estimate, and earned value management system 
at a high level so that it clearly reflects the work to be done. 

[19] NASA Federal Acquisition Regulation Supplement (NFS) § 1816.405-
273(b) and NFS contract clause 1852.216-77(a). 

[20] Through negotiations, NASA and the contractor agreed on three key 
events to include in the schedule milestone fee that are critical to 
JWST's success: (1) delivery of OTE to NASA prior to April 28, 2016; 
(2) completion of Spacecraft and Sunshield I&T prior to September 28, 
2017; and (3) Observatory completion prior to September 4, 2018. 

[21] NASA, Office of Inspector General, NASA's Use of Award Fee 
Contracts, IG-14-003 (Washington, D.C.: Nov. 19, 2013). 

[22] NASA may include other types of incentive fees on award fee 
contracts, such as JWST's schedule incentive fee. 

[23] GAO, GAO Cost Estimating and Assessment Guide: Best Practices for 
Developing and Managing Capital Program Costs, [hyperlink, 
http://www.gao.gov/products/GAO-09-3SP] (Washington, D.C.: Mar. 2, 
2009). 

[24] NASA, Office of Inspector General, NASA's Use of Award Fee 
Contracts, IG-14-003 (Washington, D.C.: Nov. 19, 2013). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation, and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the 
performance and accountability of the federal government for the 
American people. GAO examines the use of public funds; evaluates 
federal programs and policies; and provides analyses, recommendations, 
and other assistance to help Congress make informed oversight, policy, 
and funding decisions. GAO's commitment to good government is 
reflected in its core values of accountability, integrity, and 
reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's website [hyperlink, http://www.gao.gov]. Each 
weekday afternoon, GAO posts on its website newly released reports, 
testimony, and correspondence. To have GAO e-mail you a list of newly 
posted products, go to [hyperlink, http://www.gao.gov] and select 
"E-mail Updates." 

Order by Phone: 

The price of each GAO publication reflects GAO's actual cost of 
production and distribution and depends on the number of pages in the 
publication and whether the publication is printed in color or black 
and white. Pricing and ordering information is posted on GAO's 
website, [hyperlink, http://www.gao.gov/ordering.htm]. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 
TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 
MasterCard, Visa, check, or money order. Call for additional 
information. 

Connect with GAO: 

Connect with GAO on facebook, flickr, twitter, and YouTube.
Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts.
Visit GAO on the web at [hyperlink, http://www.gao.gov]. 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 
Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; 
E-mail: fraudnet@gao.gov; 
Automated answering system: (800) 424-5454 or (202) 512-7470. 

Congressional Relations: 

Katherine Siggerud, Managing Director, siggerudk@gao.gov: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, DC 20548. 

Public Affairs: 

Chuck Young, Managing Director, youngc1@gao.gov: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, DC 20548. 

[End of document]