Electronic Health Records: Additional DOD Actions Could Improve Cost and Schedule Estimating for New System
Fast Facts
The Department of Defense provides health care to millions of people. It uses multiple health record systems to manage patient health information—such as separate systems for inpatient and outpatient information.
DOD began to replace these separate systems with a comprehensive electronic health record system in 2017. However, DOD's cost and schedule estimates for fully deploying this system are not reliable. For instance, DOD didn't include enough details about its methodology or the data used for the cost estimate.
We recommended that DOD develop more reliable cost and schedule estimates for deploying this new system.
Highlights
What GAO Found
The Department of Defense (DOD) began deploying Military Health System (MHS) GENESIS—a commercial electronic health record system intended to integrate inpatient and outpatient medical and dental information in 2017. MHS GENESIS's contract award totals $5.5 billion. DOD's plans call for implementing MHS GENESIS in 24 waves or phases. The first wave was completed in October 2017 with the last wave expected to be deployed by December 2023 and additional activities planned through 2025.
DOD has not fully met the characteristics and associated best practices for developing MHS GENESIS cost and schedule estimates, as shown below.
GAO Assessment of DOD Cost and Schedule Estimates against Best Practice Characteristics
|
Cost estimate characteristic |
Assessment of cost estimate |
Schedule estimate characteristic |
Assessment of schedule estimate |
|
Comprehensive |
Substantially met |
Comprehensive |
Substantially met |
|
Well-documented |
Partially met |
Well-constructed |
Partially met |
|
Accurate |
Substantially met |
Credible |
Partially met |
|
Credible |
Minimally met |
Controlled |
Substantially met |
Legend: substantially met = DOD provided evidence that satisfies a large portion of the criterion; partially met = DOD provided evidence that satisfies about one-half of the criterion; minimally met = DOD provided evidence that satisfies a small portion of the criterion
Source: GAO assessment of the Department of Defense (DOD) Military Health System GENESIS program documentation. | GAO-22-104521
MHS GENESIS's cost estimate was unreliable because it did not substantially meet all four characteristics of a reliable cost estimate, as described in GAO's cost guide. Specifically, DOD minimally met the “credible” characteristic associated with reliable cost estimates, in part, because it did not provide evidence that a sensitivity analysis, a risk and uncertainty analysis, or an independent cost estimate were conducted. Reliable cost estimates are critical for successfully delivering IT programs.
In addition, the subproject schedules did not meet all four characteristics of a high quality, reliable schedule. Because an integrated master schedule consolidates subproject schedules, errors and reliability issues in subproject schedules will be reflected in higher-level schedules. Therefore, the larger master schedule is unreliable. Specifically, the subproject schedules partially met the “well-constructed” characteristic associated with reliable schedule estimates, in part because their critical paths could not be validated or they exhibited total float values that could allow activities and milestones to slip months or years before delaying key program activities. A reliable schedule can assist with the systematic execution of a program and the means by which to gauge progress, identify and address potential problems, and promote accountability.
Because the MHS GENESIS program cost and schedule estimates were not reliable, DOD increases the risk that management will not have the information necessary for effective decision-making. Following cost and schedule best practices can help minimize the risk of cost overruns and schedule delays, and would better position DOD for successful program implementation.
Why GAO Did This Study
DOD operates one of the nation's largest health care systems providing health care to about 9.6 million beneficiaries. It relies on multiple legacy electronic health record systems to create, maintain, and manage patient health information. DOD determined that these systems, implemented over the past three decades, require modernization and replacement with a comprehensive, real-time electronic health record, MHS GENESIS.
Congress included a provision in the Joint Explanatory Statement accompanying the Consolidated Appropriations Act, 2021 for GAO to review DOD's deployment of MHS GENESIS. This report determines the extent to which DOD's MHS GENESIS's cost estimate and program schedule are consistent with best practices. GAO reviewed documentation supporting the program's October 2020 cost estimate against best practices. In addition, GAO reviewed the program's February 2021 integrated master schedule, specifically assessing three subproject schedules. Further, GAO interviewed DOD officials within the program office to understand their practices for developing and maintaining the cost estimate and program schedule.
Recommendations
GAO is making two recommendations to DOD that it develop reliable cost and schedule estimates for the MHS GENESIS program that are consistent with GAO-identified best practices. DOD concurred with GAO's recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Defense | 1. The Secretary of Defense should direct the Program Executive Officer of Defense Health Management Systems to ensure that the program office develops a reliable cost estimate using best practices described in GAO's Cost Estimating and Assessment Guide, in particular, by addressing those cost practices that were partially or minimally met. (Recommendation 1) |
In its comments on our draft report, DOD agreed with our recommendation and outlined steps it would take in response. In July 2023, DOD reported that the program will start transitioning to new cost estimating software in September 2023. As a result of this transition, DOD stated that they will not have a new cost estimate that adheres to the best practices described in our Cost Estimating and Assessment Guide until September 2024. We will continue to be in contact with DOD to gain additional information on the actions they are taking and their progress toward completion.
|
| Department of Defense | 2. The Secretary of Defense should direct the Program Executive Officer of Defense Health Management Systems to ensure that the program office develops a reliable schedule using best practices described in GAO's Schedule Assessment Guide, in particular, by addressing those schedule practices that were partially met. (Recommendation 2) |
In July 2023, the Department of Defense (DOD) provided documentation describing the steps it had taken to implement our recommendation by developing a more reliable schedule using best practices described in GAO's Schedule Assessment Guide. Specifically, the department addressed the well-constructed and credible schedule characteristics that we reported to be only partially met. Although we were unable to verify all elements of a reliable schedule, we recognize the significant improvements DOD made to the MHS GENESIS schedule in response to this recommendation. For example, regarding the well-constructed characteristic, activities were logically sequenced and linked, meaning they are listed in the order they are to be carried out and joined with logic. Additionally, regarding the credible characteristic, the schedule was vertically and horizontally traceable, meaning products and outcomes were linked and data are consistent across different levels of the schedule. Based on these improvements, we consider the scheduling practices to be sufficiently met to close the recommendation.
|