GAO-03-241, Federal Student Aid: Progress in Integrating Pell Grant and Direct Loan Systems and Processes, but Critical Work Remains


This is the accessible text file for GAO report number GAO-03-241 
entitled 'Federal Student Aid: Progress in Integrating Pell Grant and 
Direct Loan Systems and Processes, but Critical Work Remains' which was 
released on December 31, 2002.



This text file was formatted by the U.S. General Accounting Office 

(GAO) to be accessible to users with visual impairments, as part of a 

longer term project to improve GAO products’ accessibility. Every 

attempt has been made to maintain the structural and data integrity of 

the original printed product. Accessibility features, such as text 

descriptions of tables, consecutively numbered footnotes placed at the 

end of the file, and the text of agency comment letters, are provided 

but may not exactly duplicate the presentation or format of the printed 

version. The portable document format (PDF) file is an exact electronic 

replica of the printed version. We welcome your feedback. Please E-mail 

your comments regarding the contents or accessibility features of this 

document to Webmaster@gao.gov.



Report to the Secretary of Education:



United States General Accounting Office:



GAO:



December 2002:



Federal Student Aid:



Progress in Integrating Pell Grant and Direct Loan Systems 

and Processes, but Critical Work Remains:



GAO-03-241:



GAO Highlights:



Highlights of GAO-03-241, a report to the Secretary of Education:



Why GAO Did This Study:



To address system problems and other long-standing management 

weaknesses, in 1998, the Congress created a discrete unit within the

 Department of Education, the Office of Federal Student Aid (FSA). This 

office subsequently adopted a new approach to systems integration using 

middleware (a type of software that can allow an application to access 

data residing in different databases) and Extensible Markup Language 

(XML)—a flexible, nonproprietary set of standards that is intended to 

make it easier to identify, integrate, and process information widely 

dispersed among systems and organizations. 



FSA’s first use of this approach is the Common Origination and 

Disbursement (COD) process for the Direct Loan, Pell Grant, and campus-

based programs. GAO initiated a follow-up review to assess FSA’s 

progress in implementing this process.



What GAO Found:



FSA has made progress in implementing the COD process. Specifically, it 

has implemented:



* a new information technology infrastructure that uses middleware to 

enable data exchange among disparate systems;



* the initial version of the basic COD system, which replaces two 

existing systems and is being used by schools participating in the Pell 

Grant and Direct Loan programs;



* middleware into existing systems to support the COD process; and



* a common record based on XML that schools can use to submit student 

financial data for the Pell Grant and Direct Loan programs.



However, the implementation of the COD process is behind schedule, and 

its ultimate success hinges on FSA’s completing critical work, 

including addressing serious postimplementation operational problems, 

and having thousands of postsecondary schools implement the common 

record. Further, there are important elements to managing any 

information technology investment that FSA has not yet completed:



* Determining whether expected benefits are being achieved. As 

illustrated below, FSA has only some of the metrics, baseline data, and 

tracking processes necessary to determine whether it is achieving all 

expected benefits. 



* Tracking lessons learned. FSA has relied on an ad hoc approach for 

gathering and disseminating lessons learned related to schools’ 

implementation of the common record. To address this issue, FSA plans 
to 

include lessons learned as part of an update to its school testing 
guide. 

However, this does not replace the need for an ongoing mechanism to 

capture and disseminate lessons learned, without which schools may 

encounter problems that could have been avoided or mitigated.  



Highlights Figure:



[See PDF for image]



[End of image]



What GAO Recommends:



Among GAO’s recommendations is that the Secretary of Education direct 
the 

Chief Operating Officer, FSA, to establish a process to capture and 

disseminate lessons learned to schools.



In commenting on a draft of this report, FSA provided updated 
information 

and technical comments. GAO modified the report to address this new 

information, as appropriate.



www.gao.gov/cgi-bin/getrpt?GAO-03-241.



To view the full report, including the scope and methodology, click on 
the 

link above.For more information, contact David A. Powner at (202) 512-
9286 

or pownerd@gao.gov.



Contents:



Letter:



Results in Brief:



Background:



Progress in Implementing the COD Process, but Critical Work Remains, 

and Benefits and Lessons Learned Are Not Being Tracked:



FSA Is Not Completely Tracking Actual COD Benefits or Lessons Learned 

Related to Schools’ Implementation of the Common Record:



Conclusions:



Recommendations:



Agency Comments and Our Evaluation:



Appendix I: Systems and Technologies Supporting the Common Origination 

and Disbursement Process, as of 

November 2002:



Appendix II: Comments from the Department of Education’s 

Office of Federal Student Aid:



GAO Comments:



Tables:



Table 1: Status of FSA Tracking of Actual Benefits of the COD System:



Table 2: Examples of Lessons Learned Related to School Migration to the 

Common Record:



Abbreviations:



COD: Common Origination and Disbursement:



EAI: Enterprise Application Integration:



FSA: Office of Federal Student Aid:



IT: information technology:



XML: Extensible Markup Language:



Washington, DC 20548:



December 31, 2002:



The Honorable Roderick R. Paige

The Secretary of Education:



Dear Mr. Secretary:



Over the past decade, the Department of Education has spent millions of 

dollars to modernize and integrate[Footnote 1] its disparate financial 

aid systems in an effort to improve the administration of its programs 

and provide more information and greater service to its customers--

students, parents, schools, and lenders. However, as we reported last 

year, the department’s efforts had achieved limited success.[Footnote 

2] In January 1993, for example, the department awarded a 5-year, $39 

million contract for the development and maintenance of the National 

Student Loan Data System, which was to provide information on systems 

across programmatic boundaries, yet it often lacks the most recent 

information from other financial aid systems.



To address such systems problems and other long-standing management 

weaknesses, in 1998, as you know, the Congress created the federal 

government’s first performance-based organization,[Footnote 3] 

currently called the Office of Federal Student Aid (FSA).[Footnote 4] 

Under the performance-based organization concept, FSA is a discrete 

organizational unit within the Department of Education, led by its own 

Chief Operating Officer. FSA focuses on managing the operation of the 

student financial assistance programs, while the Department of 

Education focuses on policy-making functions.



FSA subsequently adopted a systems integration approach that uses 

middleware[Footnote 5] and Extensible Markup Language (XML)[Footnote 6] 

technologies. In November 2001, we reported that in selecting 

middleware, FSA adopted a viable, industry-accepted mechanism for 

addressing its long-standing systems integration problems.[Footnote 7] 

We also reported that FSA’s first use of middleware and XML as part of 

a process for delivering Direct Loan and Pell Grant aid to students was 

expected in early 2002. This process, called the Common Origination and 

Disbursement (COD) process, was expected to use a new system, also 

called COD, and be supported by a new information technology (IT) 

infrastructure and various existing systems.



In the summer of 2002, we initiated a follow-up review to our November 

2001 report. Our objective was to assess FSA’s progress in implementing 

the COD process. In doing this work, we reviewed applicable FSA and 

Accenture (the prime contractor) documentation, including FSA’s 

modernization blueprint, the COD business case, the COD system’s 

requirements, the COD and Enterprise Application Integration 

implementation timelines, Accenture project status briefings, and test 

guides and results. In addition, we reviewed reports by an independent 

verification and validation contractor. We also interviewed applicable 

officials from FSA IT and program offices and Accenture.



We performed our work at FSA headquarters in Washington, D.C., between 

August and November 2002, in accordance with generally accepted 

government auditing standards.



Results in Brief:



FSA has made progress in implementing the COD process. [Footnote 8] 

Specifically, it has implemented (1) a new IT infrastructure that uses 

middleware to enable data exchange between disparate systems, (2) the 

initial version of the basic COD system, (3) middleware into existing 

systems to support the COD process, and (4) a common record based on 

XML that schools can use to submit student financial data for both the 

Pell Grant and Direct Loan programs. Nevertheless, the implementation 

of the COD process is behind schedule, and its ultimate success hinges 

on FSA’s completing critical work, including the implementation of the 

basic COD system requirements, addressing serious postimplementation 

operational problems, and having thousands of postsecondary schools 

implement the common record.



Important elements of managing an IT investment are determining whether 

expected benefits are being achieved and tracking lessons learned. 

However, at this time, FSA is not fully tracking whether it is 

achieving certain expected benefits, such as increased customer 

satisfaction. In this instance, FSA COD officials stated that they did 

not have approval from the Department of Education to perform the 

survey that they had planned to use to validate this expected benefit, 

but they are trying to identify alternative metrics. Without such data, 

FSA lacks vital information on whether it is achieving all of its 

investment goals. Regarding lessons learned, FSA relies on an ad hoc 

approach to gathering and disseminating such information, which it 

believes is an adequate approach. However, such an ad hoc process may 

not ensure that all schools obtain these critical data. As a result, 

schools may encounter problems that could have been avoided or 

mitigated had they known of other schools’ experiences.



We are making recommendations to improve the management of FSA’s COD 

program, including recommending that you direct FSA’s Chief Operating 

Officer to develop (1) metrics and baseline data that are related to 

increased customer satisfaction and increased financial integrity and 

(2) a tracking process to assess the extent to which the expected 

results are being achieved.



In commenting on a draft of this report, FSA’s Chief Operating Officer 

provided updated information and technical comments, but did not 

comment on our recommendations. Among the comments provided, the Chief 

Operating Officer (1) stated that the draft report did not adequately 

portray the level of COD progress that had been made, (2) agreed that 

FSA was not tracking all expected benefits at this time and provided 

new information and supporting documentation on its recent efforts in 

this area, and (3) stated that FSA’s “informal” process for 

communicating lessons had worked well but planned to provide written 

lessons learned as part of a planned update of its school testing 

guide. We updated our report to reflect new FSA processes and 

information, as appropriate. However, we believe that we have 

accurately portrayed FSA’s progress, particularly in light of its need 

to address operational problems and facilitate the implementation of 

the common record at thousands of postsecondary schools. In addition, 

we continue to believe that FSA’s ad hoc processes for capturing and 

disseminating lessons learned do not provide assurance that it has 

captured and disseminated lessons learned related to schools’ 

implementation of the common record and could overlook important 

improvements that could be made.



Background:



The Department of Education’s FSA manages and administers student 

financial assistance programs authorized under title IV of the Higher 

Education Act of 1965, as amended. These postsecondary programs include 

the William D. Ford Federal Direct Loan Program (often called the 

Direct Loan program), the Federal Family Education Loan Program (often 

called the Guaranteed Loan program), the Federal Pell Grant Program, 

and campus-based programs.[Footnote 9] Annually, these programs 

together provide about $50 billion in student aid to approximately 8 

million students and their families.



During the past three decades, the Department of Education has created 

many disparate information systems to support these various student 

financial aid programs. In many cases, these systems--run on multiple 

operating platforms using different network protocols[Footnote 10] and 

maintained and operated by a host of different contractors--were unable 

to easily exchange the timely, accurate, and useful data needed to 

ensure the proper management and oversight of the various student 

financial aid programs. For example, as we reported in 1997, neither 

the National Student Loan Data System nor other systems were designed 

for efficient access to reliable student financial aid information, 

since many systems were incompatible and lacked data standards and 

common identifiers.[Footnote 11] In addition, because FSA used three 

separate systems to originate and/or disburse title IV funds, access to 

student and school data was fragmented and unreliable. As a result, FSA 

found it increasingly difficult to quickly access data to support day-

to-day operational and management decisions, and schools could not 

easily access data to obtain a clear picture of the title IV student 

aid that had been disbursed.



In September 1999, FSA issued its initial modernization 

blueprint,[Footnote 12] which was subsequently updated in July 2000, to 

transform the title IV student financial aid systems using technology. 

COD is one of four school service business processes[Footnote 13] in 

FSA’s blueprint and is intended to implement a simplified process for 

the operation of the Direct Loan and Pell Grant programs. According to 

FSA’s modernization blueprint, the common origination and disbursement 

process is composed of seven steps involving students, the Department 

of Education, and schools: (1) obtain applicant data, (2) determine 

eligibility, (3) determine award, (4) notify the Department of 

Education of the intent to disburse, (5) obtain funds from Education, 

(6) disburse funds to student, and (7) close out. A common process to 

support origination and disbursement is considered critical to FSA’s 

goal of achieving an enterprisewide solution that provides real-time 

data to students, schools, and financial partners via Web portals.



To implement COD, FSA is using middleware and XML technologies. 

Specifically, middleware is being used to integrate FSA systems that 

support the COD process.[Footnote 14] Traditionally, systems 

integration would require building separate point-to-point interfaces 

between every two applications. Although this approach can be 

effective, it creates several problems, such as (1) every connection 

between two applications requires custom programming; (2) a lot of 

connections have to be developed when there are multiple data sources; 

and (3) whenever the logic or data in one application changes, the 

accompanying interface often also needs to be altered. Middleware 

represents an alternative means to the traditional approach, and it can 

provide a quicker and more robust solution to systems integration. In 

essence, middleware separates the business application from the 

technical details of interapplication communications. Thus, middleware 

can simplify and reduce the number of interfaces for multiple systems 

because it can handle differences in data formats and record layouts.



As part of the COD process, XML is being used to consolidate multiple 

legacy record formats previously used by schools to submit data on the 

Pell Grant and Direct Loan programs.[Footnote 15] By using an XML-based 

common record, schools can transmit one file with all of the student’s 

data instead of submitting separate legacy records with redundant 

student and school information.



Appendix I provides a high-level depiction of the systems and 

technologies supporting the COD process as of November 2002. As 

depicted, the COD system can translate or convert legacy records by 

using middleware. In addition, middleware has been built into several 

existing systems so that they can establish connectivity and exchange 

data with the COD system through a common IT infrastructure. This IT 

infrastructure, called the Enterprise Application Integration (EAI) 

bus, is also implemented using middleware to route data between systems 

in a correct format. In addition, as part of the COD process, some 

schools have begun submitting Pell Grant and Direct Loan data using the 

XML-based common record.



FSA hired Accenture as its “modernization partner” to help carry out 

its modernization blueprint, including the implementation of the COD 

process. Accenture is the prime contractor[Footnote 16] providing 

leadership of critical planning activities that are essential to the 

success of FSA’s modernization. Regarding the COD system part of FSA’s 

modernization, FSA also hired an independent verification and 

validation contractor to review the initial release of this system, 

which was completed earlier this year.



Progress in Implementing the COD Process, but Critical Work Remains, 

and Benefits and Lessons Learned Are Not Being Tracked:



FSA has made progress in implementing the new COD process. In 

particular, it has begun implementing (1) its middleware solution in 

its IT infrastructure and various existing systems, (2) the COD system, 

and (3) an XML-based common record. However, FSA’s implementation of 
COD 

is behind schedule, and critical work remains to be completed. For 

example, the basic COD system was to be completed by mid-October 2002; 

however, only about three-quarters of the COD basic system requirements 

had been implemented as of October 23, 2002. In addition, FSA is not 

tracking whether it is achieving certain benefits because it is still 

in the process of defining applicable metrics to measure progress. 

Without such tracking processes, FSA lacks critical information about 

whether it is achieving expected benefits. Finally, FSA lacks assurance 

that it has captured and disseminated important lessons learned related 

to schools’ implementation of the common record because it believes 

that its current ad hoc process is adequate. Accordingly, the thousands 

of schools that have not yet implemented the common record may not 

benefit from the experience of those that have.



FSA Has Begun Implementing COD Capabilities, but Critical Work Remains:



FSA has made progress in implementing COD. The following are 

significant elements of the COD process that have been implemented:



* Deployment of the EAI bus. As a prerequisite to implementing COD, in 

late October 2001, FSA deployed its middleware solution in an EAI 

“bus”--an IT infrastructure that uses middleware to access data from 

disparate systems, transform the data formats as necessary, and route 

the data to the appropriate requesting systems, thus enabling data 

exchange among disparate systems. The EAI bus provides the set of 

technical capabilities necessary to integrate FSA’s disparate systems.



* Initial implementation of basic COD system (release 1.x).[Footnote 

17] On April 29, 2002, FSA went live with version 1.0 of the basic COD 

system. As of mid-November 2002, FSA had released an additional five 

sub-versions of the COD system (e.g., version 1.1). The COD system 

replaces the Direct Loan Origination System and the Recipient Financial 

Management System, and it currently processes files for all schools 

participating in the Pell Grant and Direct Loan programs.[Footnote 18] 

According to FSA, in the first 6 months of its operation, the COD 

system processed just under 16 million transactions, representing Pell 

Grant and Direct Loan awards totaling almost $10 billion to over 5 

million recipients.



* Implementation of middleware in selected systems. As of mid-November 

2002, FSA had built middleware into seven systems so that these systems 

can interact with the COD system through the EAI bus. These systems 

include (1) the Central Processing System, which determines students’ 

eligibility and award levels, and (2) the National Student Loan Data 

System, which contains loan-and grant-level information and is used by 

schools to screen student aid applicants to identify ineligible 

borrowers.



* Development and implementation of the common record. Using XML, FSA 

developed and began implementing a common record that schools can use 

to submit student financial aid data to the COD system. The common 

record, designed with assistance from members of the National Council 

of Higher Education Loan Programs and the Postsecondary Electronic 

Standards Council, consolidates multiple legacy file formats previously 

used by the Pell Grant and Direct Loan programs.[Footnote 19]

:



Although FSA has made progress in implementing the COD process, 

critical work remains to be completed. First, FSA is behind schedule in 

implementing the basic COD system. Although FSA had planned to complete 

the basic COD system by mid-October 2002, only about three-

quarters[Footnote 20] of the COD basic system requirements had been 

implemented as of October 23, 2002.[Footnote 21] For example, as of 

early November 2002, one of the basic business functions that remains 

to be implemented is to enable FSA to make automated adjustments in 

batches to school current funding levels. FSA now estimates that most 

of the remaining functionality will be completed by the end of 

September 2003. According to FSA IT and program officials, the 

implementation of the basic COD functionality was delayed to allow 

adequate time for testing to ensure the quality of the system.[Footnote 

22]



Second, as of November 19, 2002, Accenture reported several operational 

problems that needed to be addressed. For example, in some cases, the 

COD system was incorrectly processing school batch data that contained 

multiple change records for an individual student. According to COD and 

contractor officials, the causes of operational problems included 

unclear requirements and software design defects. An independent 

verification and validation contractor also found problems with the 

requirements and design aspects of release 1.0. The COD Contracting 

Officer’s Representative characterized these operational problems as 

very serious and stated that they could impede operations and the 

delivery of future COD releases. This same official noted that FSA and 

Accenture are currently undertaking efforts to address these problems. 

For example, FSA has established production teams composed of agency 

and contractor staff to address problems in specific areas. In 

addition, FSA has established a continuous improvement process to more 

rigorously manage its relationship with Accenture.



Third, fewer postsecondary schools than planned have implemented the 

common record. FSA had estimated that 50 schools (out of about 5,500) 

would implement the common record in fiscal year 2002. However, as of 

November 26, 2002, only 22 schools[Footnote 23] had implemented and 

tested the common record with FSA. FSA COD officials attributed the 

fewer-than-expected number of schools using the common record to 

schools and vendors not being ready to implement it. FSA expects that 

the number of schools using the common record will be considerably 

higher during the next award year (2003-2004) because, by April 2003, 

it plans to implement and test the common record with EDExpress, a 

software application FSA distributes free of charge to about 3,000 

schools for use in submitting data. In addition, FSA expects that all 

schools will be using the common record format by March 2004, in time 

for the 2004-2005 award year.



FSA Is Not Completely Tracking Actual COD Benefits or Lessons Learned 

Related to Schools’ Implementation of the Common Record:



In its COD business case, FSA outlined five expected benefits: 

(1) reduced cost, (2) increased customer satisfaction, (3) increased 

employee satisfaction, (4) increased financial integrity, and (5) the 

integration and modernization of legacy systems. An important aspect of 

implementing an IT investment cited by the Office of Management and 

Budget[Footnote 24] and our IT investment management guide[Footnote 25] 

is evaluating the results of the investment by determining whether such 

expected benefits are being achieved. However, as illustrated in table 

1, at this time FSA has only some of the data necessary to determine 

whether it is achieving all expected benefits. In particular, for the 

increased customer satisfaction and financial integrity benefits, FSA 

(1) has not fully defined the performance metrics to be used, (2) does 

not have all baseline data, and/or (3) is not fully tracking whether 

the benefits are being achieved. In these cases, FSA COD officials 

stated that they were in the process of developing relevant metrics, 

which would be tracked to measure the project’s performance against 

expected benefits. However, until FSA develops these data and begins 

tracking actual benefits and comparing them with expected benefits, it 

will lack vital data with which to demonstrate actual investment 

results.



Table 1: Status of FSA Tracking of Actual Benefits of the COD System:



Expected benefit: Reduced cost; Metrics defined?: Yes--The COD business 

case identifies decreased costs for systems operations, customer 

service, and the virtual data center.; Baseline data available?: Yes; 

Benefits tracked?: Yes[A].



Expected benefit: Increased customer

satisfaction; Metrics defined?: Partially--The COD business case 

identifies the use of the American Customer Satisfaction Index scores 

to demonstrate clarity of instructions, ease of submitting data, and 

accuracy of records. However, FSA COD officials stated that at this 

time, they did not have approval from the Department of Education to 

perform this survey; therefore, they are trying to identify alternative 

metrics that could be used to measure customer satisfaction 

improvements due to COD.; Baseline data available?: Partially--

According to FSA, baseline data related to the American Customer 

Satisfaction Survey exist, but other planned metric(s) are not yet 

defined.; Benefits tracked?: No.



Expected benefit: Increased employee 

satisfaction; Metrics defined?: Not applicable--The COD business case 

identifies a metric for this expected benefit, 

but in commenting on our draft report, FSA stated that it has decided 

to discontinue the 

use of employee satisfaction as a key performance measure and thus will 

not be tracking this as an expected benefit of the 

COD system.; Baseline data available?: Not applicable; Benefits 

tracked?: Not applicable.



Expected benefit: Increased financial 

integrity; Metrics defined?: Partially--FSA COD officials stated that 

they have identified increased school compliance with its requirements 

that schools substantiate the amount of funds withdrawn within certain 

time frames as a metric. They also stated that they are currently 

developing other performance metrics for increased financial integrity 

but have not set a target 

date for completing them.; Baseline data available?: Partially--

Baseline data related to the identified metric exist,[B] but other 

planned metric(s) are not yet defined.; Benefits tracked?: Partially--

Tracking the identified metric only..



Expected benefit: Integration and modernization

of legacy systems; Metrics defined?: Yes--The COD business case 

identifies a variety of quantitative and qualitative 

measures, including the retirement of two legacy applications and the 

simplification of testing and ongoing development.; Baseline data 

available?: Yes; Benefits tracked?: Yes.



[A] In providing comments on a draft of this report, FSA’s Chief 

Operating Officer stated that after the completion of our review, FSA 

had begun tracking this expected benefit against the metrics defined in 

the business case. FSA also provided supporting documentation that 

previously was not available. We did not validate these new data.



[B] In providing comments on a draft of this report, FSA’s Chief 

Operating Officer stated that estimated baseline data for the 

identified metric are available, but FSA is working to develop more 

formal baseline data for this performance metric.



Source: GAO analysis on the basis of FSA documentation.



[End of table]



FSA IT officials also stated that they plan to have a contractor 

conduct a postimplementation review of the COD basic system in fiscal 

year 2003, which is expected to look at the achievement of expected 

benefits. While this is an important initiative that could provide FSA 

with valuable information, it does not take the place of a continuing 

and systematic process of tracking actual benefits.



According to our IT investment management guide, another critical 

activity is establishing a process for developing and capturing lessons 

learned in a written product or knowledge base and disseminating them 

to decision-makers.[Footnote 26] Lessons-learned mechanisms serve to 

communicate acquired knowledge more effectively and to ensure that 

beneficial information is factored into planning, work processes, and 

activities. Lessons learned can be based on positive experiences that 

save money or on negative experiences that result in undesirable 

outcomes.



FSA has recognized the importance of generating lessons learned in 

certain areas. For example, it has implemented a process for developing 

lessons related to managing the relationship between the agency and its 

prime contractor. However, FSA lacks such a process for capturing or 

disseminating lessons related to school migration to the common record. 

FSA COD officials stated that lessons learned pertaining to school 

migration to the common record are addressed through periodic 

discussions during biweekly conference calls with schools undergoing 

testing with FSA and during portions of various FSA-sponsored 

conferences. FSA COD officials stated that they believed this process 

for capturing and disseminating lessons learned was adequate.



However, by relying on such an ad hoc process, FSA lacks assurance that 

it has captured and disseminated all key lessons learned related to 

schools’ implementation of the common record and could overlook 

important improvements that could be made. In addition, schools that do 

not attend the conferences may not receive and benefit from the lessons 

identified in the initial phase of implementation. As a result, schools 

may encounter problems that could have been avoided or mitigated had 

they known of other schools’ experiences. This could hamper FSA’s 

ability to facilitate the transition of schools to the new common 

record and thus the agency’s ability to fully implement the new COD 

process and achieve the expected benefits.



In commenting on a draft of this report, FSA stated that it plans to 

provide lessons learned as part of a planned update to its school 

testing guide. While this is a positive step, it does not replace the 

need for mechanisms to continuously capture and disseminate acquired 

knowledge as schools implement the common record.



Table 2 includes examples of lessons learned provided by FSA at our 

request that were drawn from schools’ initial implementation of the 

common record for the 2002-2003 award year. Such information would be 

important for the thousands of schools that have not yet implemented 

the common record so that they can avoid problems during the common 

record implementation and testing processes.



Table 2: Examples of Lessons Learned Related to School Migration to the 

Common Record:



Lesson learned: Several schools and vendors did not have the technical 

staff to support the switch to XML.; Effect: FSA had fewer schools 

participating in testing than initially expected..



Lesson learned: Many schools had only a technical contact or a business 

contact, but not both.; Effect: Instructions FSA provided to schools 

may not be translated properly between the business and technical 

contacts..



Lesson learned: Schools did not sufficiently test their applications 

before testing them with COD.; Effect: Schools and vendors required 

more support from FSA than anticipated..



Lesson learned: Service-level expectations were not well-defined before 

the start of testing and were not made available to all 

parties involved.; Effect: Schools had unrealistic expectations of the 

turnaround time required to process their records..



Lesson learned: Schools should use an XML tool to validate school XML-

based applications before testing with the COD system.; Effect: Schools 

that did not participate in such validations had more errors and a 

greater chance of their records being rejected upon submission to the 

COD system..



Source: FSA.



[End of table]



Conclusions:



FSA has taken important steps toward achieving full implementation of 

the new COD process. However, critical actions, such as completing the 

basic functionality of the COD system and the implementation of the 

common record at thousands of affected schools, must still be 

undertaken. In addition, FSA has not yet fully established the metrics 

and processes to track actual benefits related to all of its expected 

benefits or the lessons that have been generated by the few schools 

that have implemented the common record thus far. By not tracking 

actual benefits, FSA lacks information that is critical to determining 

whether it is meeting all of its goals. Further, not capturing and 

disseminating information to schools regarding lessons learned could 

make achieving these goals more difficult.



Recommendations:



To determine the extent to which the new COD process is achieving 

expected results related to customer satisfaction and financial 

integrity, we recommend that you direct FSA’s Chief Operating Officer 

to expeditiously develop metrics and baseline data to measure these 

benefits and develop a tracking process to assess the extent to which 

the expected results are being achieved.



To ensure that the schools that have not yet implemented the common 

record benefit from the experiences of those that have, we recommend 

that you direct FSA’s Chief Operating Officer to establish a process 

for capturing lessons learned in a written product or knowledge base 

and for disseminating them to these schools.



Agency Comments and Our Evaluation:



In providing written comments on our draft report, FSA’s Chief 

Operating Officer provided technical comments and updated information, 

but did not comment on our recommendations. Specifically,



* The Chief Operating Officer did not believe the report adequately 

portrayed the level of COD progress that had been made. In particular, 

she took issue with our using the completion of 75 percent of COD 

requirements as an indication of progress. Although the Chief Operating 

Officer did not disagree with the accuracy of this figure, she stated 

that FSA’s informal analysis indicated that between 85 to 90 percent of 

COD functions had been implemented, which she believed was a better 

gauge of progress. We believe that we have accurately portrayed FSA’s 

progress in implementing the COD process. First, since FSA’s analysis 

was “informal,” and FSA’s supporting documentation had limited detail 

that we could not validate, we do not agree that this should be the 

primary basis for an analysis of COD’s progress. Second, we included 

both the percentage of COD’s requirements that had been implemented and 

FSA’s estimate in our report. Nevertheless, we modified our report to 

include additional data provided by FSA regarding the number of 

transactions processed by the COD system to further indicate progress.

:



* The Chief Operating Officer agreed that the tracking of all of the 

expected benefits is not in place at this time, but stated that work is 

under way in this area. FSA also provided updated information and 

supporting documentation related to the tracking of some of the 

expected benefits. We made changes to the report reflecting this new 

information, as appropriate.

:



* The Chief Operating Officer agreed that it is important and 

beneficial to communicate lessons learned, but stated that FSA’s 

informal method for communicating lessons related to school migration 

to the common record worked well in the first year of COD 

implementation. FSA also noted that it plans to include lessons learned 

in a planned update to its school testing guide. We modified the report 

to reflect this initiative, but we do not agree that FSA’s informal 

method or its plan to include lessons learned in its testing guide is 

adequate because these approaches do not provide a continuous process 

for actively capturing and disseminating lessons learned. As a result, 

some important lessons may be overlooked, and all schools may not be 

aware of potential problems associated with implementing the common 

record.



FSA’s written comments, along with our responses, are reproduced in 

appendix II.



We are sending copies of this report to the appropriate congressional 

committees, the Secretary of Education, the Chief Operating Officer of 

Education’s Office of Federal Student Aid, and the Director of the 

Office of Management and Budget. This report will also be available at 

no charge on the GAO Web site at http://www.gao.gov.



If you have any questions on matters discussed in this report, please 

contact me at (202) 512-9286 or Linda J. Lambert, Assistant Director, 

at (202) 512-9556. We can also be reached by E-mail at pownerd@gao.gov 

and lambertl@gao.gov, respectively. Other individuals making key 

contributions to this report included Jason B. Bakelar and Anh Q. Le.



Sincerely yours,



Signed by David A. Powner:



David A. Powner

Director (Acting), Information Technology

 Management Issues:



[End of section]



Appendix I Systems and Technologies Supporting the Common Origination 

and Disbursement Process, as of November 2002:



[see PDF for figure]



[A] This is a temporary interface. DLOS is targeted to be retired in 

fiscal year 2003.



[B] MQSeries is IBM’s proprietary message format.



[C] This is an information technology infrastructure that enables data 

exchange among disparate systems.



[End of image]



[End of section]



Appendix II Comments from the Department of Education’s Office of 

Federal Student Aid:



UNITED STATES DEPARTMENT OF EDUCATION:



Federal Student Aid Chief Operating Officer:



December 18, 2002:



Mr. David A. Powner Director (Acting), 

Information Technology 

Management Issues

United States General Accounting Office 

Washington, DC 20548:



Dear Mr. Powner:



I am writing in response to your request for comment on the draft GAO 

report to Secretary Paige on the implementation of the Common 

Origination and Disbursement (COD) system by Federal Student Aid. The 

draft report is entitled, “Federal Student Aid: System Integration 

Progressing, But Critical Work Remains.” Our understanding was that the 

purpose of this audit was refocused to specifically examine FSA’s 

progress in implementing the Common Origination and Disbursement System 

(COD), and did not include an examination of FSA’s system integration 

plan nor progress against that plan. Thus, we suggest that GAO change 

the title to make it clear that the audit was limited to an examination 

of the implementation of COD. In addition, we ask you to consider the 

information provided below as you finalize this report.



Highlights, Results in Brief and Background:



In the “Highlights” section, GAO makes reference to “serious post-

implementation operational problems”. However, no such problems are 

identified in the report. Further, in the “Results in Brief’ section on 

page 2 of the report, while GAO states that FSA has made progress in 

implementing the COD process, and describes four very significant 

accomplishments, the information that follows is out of context, and 

suggests that progress made has been minimal. Progress of significance 

that should be reflected in the report is that in the first six months 

of COD operations, the system has processed just under 16 million 

transactions. Those transactions report Pell Grant and Direct Loan 

awards totaling almost $10 billion to over 5 million recipients. While 

we certainly acknowledge that the system has had problems, the fact is 

that we have processed millions of records for billions of dollars, and 

have given the right money to the right people.



In addition, the report on page 3 states that “...the COD process is 

behind schedule, and its ultimate success hinges on FSA’s completing 

critical work, including having thousands of post-secondary schools 

implement the common record.” This statement does not take into 

consideration that all post-secondary educational institutions 

participating in the Pell Grant and/or Direct Loan Programs 

(approximately 5,500) are current users of the COD system,

whether common record users or not. We believe that the GAO report 

should note that middleware functionality enables FSA to process all 

Pell Grant and Direct Loan record formats so that our schools can 

implement the XML formatted common record as their schedule permits. 

The common record, while new and innovative is only one of the 

important parts of the COD and it has always been our published 

intention that schools migrate to using the common record over a three-

year period. Since the inception of the Commonline file format, which 

was first introduced many years ago, it has been standard student aid 

industry practice to have a planned phase-in of new file formats and 

processes. This is widely viewed as risk mitigation, and in fact, the 

industry supports that mitigation by supporting several versions of 

file formats while its customers adapt to the new process. GAO seems to 

be implying that phasing in the implementation of the common record was 

actually an INCREASE in risk, while FSA feels it was prudent as a way 

to DECREASE risk.



The report on page 3 also discusses in brief the findings of the review 

that we shall address in the appropriate more detailed section of the 

draft report. This discussion, as well as the Highlights section, 

should be revised to reflect changes as a result of the comments that 

follow.



Progress in Implementing COD Process, But Critical Work Remains and 

Benefits and Lessons Learned Not Being Tracked:



On page 6 and again on page 8 of the report, GAO states that as of 

October 22 only about three quarters of the COD basic system 

requirements had been implemented. This is based on an arithmetic 

calculation using the number of requirements implemented versus the 

total number of requirements. We believe it is important to note the 

relative functional value of the implemented and non-implemented 

requirements, because in fact, if only 75 percent of the critical COD 

functionality were in place it is doubtful that so many Pell Grants and 

Direct Loans could have been processed through the current date.



As previously discussed with GAO staff, FSA COD IT development managers 

performed an informal analysis of the percentage of COD functionality 

that was implemented by the end of September 2002, as well as provided 

an analysis of the requirements in the initial 1.0 COD software release 

against the contract’s 1.0 acceptance criteria. In both cases, the 

development managers provided an estimate that between 85-90 percent of 

the functionality was implemented and that the critical functionality 

necessary to process records had been implemented.We have replicated 

virtually all of the existing functionality of the two previous systems 

into an integrated common system, as well as provided a number of 

enhancements that please our customers and improve financial integrity. 

It seems to FSA that our goal of integrating existing systems into a 

common system has been met, although we recognize that there is support 

functionality that still needs to be completed.



FSA Not Completely Tracking Actual COD Benefits or Lessons Learned 

Related to Schools’_ Implementation of the Common Record:



On pages 3, 6, and in detail beginning on page 9 of the report, the GAO 

states that FSA is not tracking whether it is achieving certain 

expected benefits (as discussed in the FSA COD system business case). 

We agree that all of the tracking of the expected benefits is not in 

place at this time but work on all is underway. The following is an 

update to each of the items listed in Table 1 on page 11.



Reduced Cost. The Contracting Officer’s Representative for the COD 

system has been tracking monthly COD costs for the 6 months of COD 

operations and has compared those costs to the baseline legacy system 

costs. However, as noted in the report, those costs were not broken 

down into the component business case costs of Operations, Customer 

Service, and virtual data center costs at the time of the GAO review 

team data collection. As part of our response, we are providing an 

excel spreadsheet (Attachment 1) containing such a comparison for the 

five months of COD operations in FY 02 with the understanding that this 

is an on-going tracking of this expected benefit.



Increased Customer Satisfaction. Although FSA has for several years 

conducted a valid and reliable survey of its customers, the Department 

intends to modify the approach to, among other things, eliminate 

duplicate surveys of the Department’s customers. During this 

transition, no survey was conducted in FY2002, so we have no data for 

this indicator. However, we certainly intend to measure customer 

satisfaction, and as you noted, we are in the process of identifying 

alternative metrics that could be used to measure customer satisfaction 

improvements due to COD and we agree that it may be necessary to alter 

the COD contract to do so.



Increase Employee Satisfaction. FSA has decided to discontinue employee 

satisfaction as a key performance measure for the organization and 

therefore, will not be using this activity as an expected benefit of 

the COD system.



Increased Financial Integrity. FSA is tracking the performance of 

participating institutions against the key financial requirement of 

reporting actual disbursements of Pell Grants and Direct Loans within 

30 days of drawing the funds down. GAO was provided with the procedures 

being employed by the COD Customer Service Center to monitor and act on 

the resulting performance data related to this key financial integrity 

measure. We are attaching a copy of the most recent weekly monitoring 

report (Attachment 2). This report demonstrates for the week of 12/02/

02-12/06/02, a total of 421 of 5,500 (approximately 7.6%) participating 

schools are outside the 30-day requirement. We believe that this result 

is very noteworthy given that we have deliberately not aggressively 

pursued the schools compliance with this requirement this early into 

the implementation of COD. This 7.6% is compared to an estimate of 15-

20% of schools being non-compliant with that requirement for any given 

period under the legacy systems. FSA is working to develop a more 

formal baseline set of numbers for this performance measure.



Lessons Learned Related to School Testing. On page 12 in the report, 

GAO took issue with the manner in which lessons learned during school 

testing were being communicated to the 22 full participant schools. FSA 

clearly understands the importance and benefits that can be derived by 

communication of lessons learned. FSA employed a more informal method 

of communicating lessons learned because of the limited number of full 

participant schools in this inaugural year of COD implementation. This 

process worked well for this year, but FSA is providing a more formal

 method of disseminating this type of information to full COD 

participant schools that will test in the upcoming 03-04 COD year. GAO 

was provided with a set of slides used at the most recent FSA sponsored 

Electronic Access Conference held this month. One of the slides was 

labeled School Testing Lessons Learned (Attachment 3). In addition, the 

03-04 School COD Testing Guide under development will have a section on 

lessons learned from the schools and financial aid software vendors 
that 

tested with COD in the 02-03 COD year. We expect that document to be 

available in January 2003.



Thank you for this opportunity to comment on the draft report on the 

implementation of COD.



Sincerely,



Signed by Theresa S. Shaw:



Theresa S. Shaw:



Note: GAO comments supplementing those in the report text appear at the 

end of this appendix.



See comment 3.



See comment 2.



See comment 1.



See comment 5.



See comment 4.



See comment 6.



See comment 8.



See comment 7.



The following are GAO’s comments on the Department of Education’s 

Office of Federal Student Aid’s letter dated December 18, 2002.



GAO Comments:



1. We revised the draft report title to clarify that our follow-up 

review was focused on assessing FSA’s progress in implementing the COD 

process.



2. Information related to operational problems was contained in the 

draft report. We asked the COD Contracting Officer’s Representative to 

characterize these problems, and he stated that they were very serious. 

In addition, we confirmed the seriousness of these problems at the 

conclusion of our review with FSA IT and program officials.



3. We modified this report to include data on the number of 

transactions processed. We also modified our report to clarify that all 

schools participating in the Pell Grant and/or Direct Loan programs 

currently use the COD system.



4. We do not agree that the report implies that FSA’s use of a phased-

in approach in implementing the common record increases risks. Instead, 

this report notes that the implementation is not yet complete.



5. We believe that we have accurately portrayed FSA’s progress in 

implementing the COD process. First, since FSA’s analysis was 

“informal,” and FSA’s supporting documentation had limited detail that 

we could not validate, we do not agree that the COD Development 

Manager’s functionality estimate should be the primary basis for an 

analysis of COD’s progress. Second, we included both the percentage of 

COD’s requirements that had been implemented and FSA’s estimate in our 

report.



6. We modified our report to reflect this updated information as 

appropriate.



7. We do not agree that FSA’s informal process for capturing and 

disseminating lessons learned was adequate because (1) it may lead to 

important lessons being overlooked and (2) all schools may not be aware 

of potential problems associated with implementing the common record.



8. We modified this report to reflect that FSA plans to include lessons 

learned in a planned update to its school testing guide. While this is 

a positive step, it does not replace the need for mechanisms to 

continuously capture and disseminate acquired knowledge as schools 

implement the common record.



FOOTNOTES



[1] Information integration is defined by the National Institute of 

Standards and Technology as the establishment of the appropriate 

computer hardware/software, methodology, and organizational 

environment to provide a unified and shared information management 

capability for a complex business enterprise. 



[2] U.S. General Accounting Office, Student Financial Aid: Use of 

Middleware for Systems Integration Holds Promise, GAO-02-7 (Washington, 

D.C.: Nov. 30, 2001).



[3] The Higher Education Amendments of 1998, which amended the Higher 

Education Act of 1965, states that the responsibilities of the 

performance-based organization include integrating the information 

systems supporting federal student financial assistance programs; 

implementing an open, common, and integrated system for delivery of 

student financial assistance under title IV; and developing and 

maintaining a student financial assistance system that contains 

complete, accurate, and timely data to ensure program integrity.



[4] Financial aid programs are administered by an office previously 

known as the Office of Student Financial Assistance, which was changed 

to the Office of Federal Student Aid on March 6, 2002. 



[5] Middleware is a type of software that can allow an application to 

access data residing in different databases. In addition, middleware 

can enable dissimilar systems to communicate and work together as if 

they all resided on a single platform.



[6] XML is a flexible, nonproprietary set of standards that offers the 

promise of making it significantly easier to identify, integrate, and 

process information that is widely dispersed among systems and 

organizations.



[7] GAO-02-7.



[8] Accenture was responsible for providing the project management, 

software, testing, and operation of the technology elements of the COD 

process. However, as used here, we ascribe the action to FSA because it 

made the decision to accept and deploy the contractor’s work. 



[9] Campus-based programs, which include the Federal Work-Study 

Program, the Federal Perkins Loan Program, and the Federal Supplemental 

Educational Opportunity Grant Program, are administered jointly by FSA 

and postsecondary educational institutions.



[10] For example, the FSA systems environment includes operating 

platforms, such as IBM OS/390 mainframe, Sun Solaris on Sparc, and 

Windows NT, and network protocols, such as the Transmission Control 

Protocol/Internet Protocol and Systems Network Architecture.



[11] U.S. General Accounting Office, Student Financial Aid Information: 

Systems Architecture Needed to Improve Programs’ Efficiency, GAO/

AIMD-97-122 (Washington, D.C.: July 29, 1997).



[12] FSA’s modernization blueprint describes the agency’s business 

requirements, business and technical architecture, and sequencing plan.



[13] The other three processes are program eligibility, program 

support, and financial transactions.



[14] For additional information on middleware and how it works, see 

GAO-02-7 and North Carolina Information Resource Management Commission, 

North Carolina Statewide Technical Architecture, which can be found at 

http://irm.state.nc.us/techarch/chaps/pdffiles/pdflist.htm.



[15] For additional information on XML and how it works, see U.S. 

General Accounting Office, Electronic Government: Challenges to 

Effective Adoption of the Extensible Markup Language, GAO-02-327 

(Washington, D.C.: Apr. 5, 2002).



[16] Accenture also uses subcontractors to perform some of the 

requirements related to this contract.



[17] “X” represents the sub-version number of the COD system. As of 

mid-November 2002, FSA planned to deploy a total of seven sub-versions 

of release 1.x.



[18] FSA plans to keep these two systems functioning until it (1) 

completes the reconciliation of prior years’ processing for direct 

loans, which is expected to occur by the end of this fiscal year, and 

(2) migrates the Recipient Financial Management System data to COD, 

which is expected by the summer of 2003.



[19] The common record also includes data blocks for campus-based 

programs to simplify school reporting. At this time, schools cannot 

report these data to FSA because the campus-based program part of the 

COD system is not yet available. According to a COD official, the 

campus-based data blocks in the common record are expected to be 

enabled in the summer of 2003. 



[20] This percentage does not include functions initially planned to be 

included in the COD system but subsequently cancelled.



[21] In addition, on September 4, 2002, the COD Development Manager 

estimated that Accenture had met about 88 percent of the COD production 

acceptance criteria. 



[22] Once release 1.x is completed, FSA plans to implement other 

functionality, including campus-based program reporting.



[23] In addition, as of mid-October 2002, five vendors had implemented 

and tested the common record with FSA. Although FSA does not know how 

many schools are serviced by these vendors because the vendors consider 

this information proprietary, some of the 22 schools that have 

implemented the common record use these vendors.



[24] Office of Management and Budget Circular A-130, Management of 

Federal Information Resources (Nov. 30, 2000).



[25] U.S. General Accounting Office, Information Technology Investment 

Management: A Framework for Assessing and Improving Process Maturity, 

GAO/AIMD-10-1.23, Exposure Draft (Washington, D.C.: May 2000).



[26] GAO/AIMD-10-1.23.



GAO’s Mission:



The General Accounting Office, the investigative arm of Congress, 

exists to support Congress in meeting its constitutional 

responsibilities and to help improve the performance and accountability 

of the federal government for the American people. GAO examines the use 

of public funds; evaluates federal programs and policies; and provides 

analyses, recommendations, and other assistance to help Congress make 

informed oversight, policy, and funding decisions. GAO’s commitment to 

good government is reflected in its core values of accountability, 

integrity, and reliability.



Obtaining Copies of GAO Reports and Testimony:



The fastest and easiest way to obtain copies of GAO documents at no 

cost is through the Internet. GAO’s Web site ( www.gao.gov ) contains 

abstracts and full-text files of current reports and testimony and an 

expanding archive of older products. The Web site features a search 

engine to help you locate documents using key words and phrases. You 

can print these documents in their entirety, including charts and other 

graphics.



Each day, GAO issues a list of newly released reports, testimony, and 

correspondence. GAO posts this list, known as “Today’s Reports,” on its 

Web site daily. The list contains links to the full-text document 

files. To have GAO e-mail this list to you every afternoon, go to 

www.gao.gov and select “Subscribe to daily E-mail alert for newly 

released products” under the GAO Reports heading.



Order by Mail or Phone:



The first copy of each printed report is free. Additional copies are $2 

each. A check or money order should be made out to the Superintendent 

of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. 

Orders should be sent to:



U.S. General Accounting Office



441 G Street NW,



Room LM Washington,



D.C. 20548:



To order by Phone: 	



	Voice: (202) 512-6000:



	TDD: (202) 512-2537:



	Fax: (202) 512-6061:



To Report Fraud, Waste, and Abuse in Federal Programs:



Contact:



Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov



Automated answering system: (800) 424-5454 or (202) 512-7470:



Public Affairs:



Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.



General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.



20548: