This is the accessible text file for GAO report number GAO-12-519 entitled 'Federal Telework: Program Measurement Continues to Confront Data Reliability Issues' which was released on April 19, 2012. This text file was formatted by the U.S. Government Accountability Office (GAO) to be accessible to users with visual impairments, as part of a longer term project to improve GAO products' accessibility. Every attempt has been made to maintain the structural and data integrity of the original printed product. Accessibility features, such as text descriptions of tables, consecutively numbered footnotes placed at the end of the file, and the text of agency comment letters, are provided but may not exactly duplicate the presentation or format of the printed version. The portable document format (PDF) file is an exact electronic replica of the printed version. We welcome your feedback. Please E-mail your comments regarding the contents or accessibility features of this document to Webmaster@gao.gov. This is a work of the U.S. government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. Because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. United States Government Accountability Office: GAO: Report to the Subcommittee on Oversight of Government Management, the Federal Workforce, and the District of Columbia, Committee on Homeland Security and Governmental Affairs, U.S. Senate: April 2012: Federal Telework: Program Measurement Continues to Confront Data Reliability Issues: GAO-12-519: GAO Highlights: Highlights of GAO-12-519, a report to the Subcommittee on Oversight of Government Management, the Federal Workforce, and the District of Columbia, Committee on Homeland Security and Governmental Affairs, U.S. Senate. Why GAO Did This Study: The Telework Enhancement Act of 2010 requires OPM to report to Congress on the degree of telework participation at executive agencies. To meet this requirement, OPM collects information on agency telework programs through an annual survey, which it refers to as the data call. However, concerns exist about the reliability of these data. GAO was asked to assess OPM’s: (1) actions in response to the requirements of the act and (2) handling of identified data reliability issues in the 2011 telework data call. To address these objectives GAO reviewed its previous reports addressing telework data reliability, and used the Office of Management and Budget’s guidance for federal surveys to review OPM’s (1) plans to collect telework participation data from agencies and (2) development of a data collection instrument. GAO interviewed key OPM officials about its implementation of the 2011 data call. What GAO Found: To prepare for its reporting obligations under the Telework Enhancement Act of 2010, the Office of Personnel Management (OPM) assembled the Interagency Telework Measurement Group, consisting of officials from several federal agencies, to assist in revising the telework data call—the survey OPM has used since 2002 to collect telework data from executive agencies. This group standardized key terms such as telework, employee, and eligibility to promote a common reporting methodology among the agencies. The revised telework data call also included changes to the time period for which OPM requested agencies report telework data, and included more extensive training for respondents. Because of changes made to the data call to allow OPM to meet requirements of the act and assist agencies in responding to the data call, OPM officials believe they will be able to provide to Congress an improved report on telework in June 2012. However, these changes also mean that OPM officials will not be able to use participation and frequency data from the 2011 data call to compare to data from previous years and across agencies. OPM officials have noted that this could limit OPM’s ability to report agency progress in its first report to Congress. The ability to compare with previous years is affected by: * agencies use of methods of varying reliability to collect telework data, and some agencies made changes to their data collection systems for the 2011 data call. Executive agencies provide telework participation and frequency data by relying on estimates, counting telework agreements, or using automated time and attendance records. * modifications to the data call instrument, including changes to terminology and the time period during which telework data was requested. OPM officials said they expect these changes will improve the consistency of data. But if OPM reports progress based on data collected using changing terminology and from different time periods, the agency may reach erroneous conclusions. Participants at the two data call training sessions may not have received the same reporting instructions, and uncertainty about whether all agency respondents attended training, created a risk that some respondents may be unaware of important terms and instructions. While some of the information provided at the two training sessions was similar, each session contained some new information, usually in response to questions raised at a previous session. Future data call improvement efforts could result in a trade-off between the desire for maintenance of a consistent data series over time for comparison with previous data calls and a need to improve data collection. According to OPM, agencies will begin piloting automated telework data collection during 2012 and 2013. OPM expects that this method of data collection will provide it more reliable data than other methods. However, these efforts to standardize methods for tracking telework data may result in changes to agencies’ methods of data collection. The 2011 data call, notwithstanding its limitations, will be useful to help OPM identify and understand major changes in reported participation data that could occur during a transition to automated data collection. What GAO Recommends: GAO recommends that OPM (1) clearly report reliability limitations with the 2011 telework data call in its June 2012 report to Congress and (2) continue efforts to improve data collection and gather information to allow for the appropriate qualification of year-to-year comparisons and inform users about the effects of data collection changes going forward. OPM partially concurred with the first recommendation. However, GAO believes it should report limitations in its annual report. OPM fully concurred with the second. OPM provided a number of technical comments which GAO incorporated as appropriate. View [hyperlink, http://www.gao.gov/products/GAO-12-519]. For more information, contact Yvonne D. Jones at (202) 512-6806 or jonesy@gao.gov. Contents: Letter: Background: OPM Revised the 2011 Telework Data Call to Prepare for Reporting under the Telework Enhancement Act of 2010: Various Methods of Data Collection and Ongoing Changes to the Data Call Make It Difficult to Assure Data Reliability and to Make Year-to- Year Comparisons: Conclusions: Recommendations for Executive Action: Agency Comments and Our Evaluation: Appendix I: Objectives, Scope, and Methodology: Appendix II: Revisions to OPM's Telework Data Call to Collect Information to Prepare for Reporting under the Telework Enhancement Act of 2010: Appendix III: Comments from the Office of Personnel Management: Appendix IV: GAO Contact and Staff Acknowledgments: Related GAO Products: Table: Table 1: Principles Used to Assess the Data Call: Figure: Figure 1: Differences in Time Period of Data Requested and Reported between 2010 and 2011 Telework Data Calls: Abbreviations: EHRI: Employee Human Resources Integration: FAQ: frequently asked questions: ITMG: Interagency Telework Measurement Group: OMB: Office of Management and Budget: OPM: Office of Personnel Management: [End of section] United States Government Accountability Office: Washington, DC 20548: April 19, 2012: The Honorable Daniel K. Akaka: Chairman: The Honorable Ron Johnson: Ranking Member: Subcommittee on Oversight of Government Management, the Federal Workforce, and the District of Columbia: Committee on Homeland Security and Governmental Affairs: United States Senate: Congress has encouraged federal agencies to expand their use of telework by requiring that executive agencies establish telework policies, incorporate those policies into continuity of operations plans, and provide interactive telework training to employees and managers. The Office of Personnel Management (OPM) has reported telework can enhance the productivity and flexibility of federal employees; help recruit and retain a qualified workforce; and reduce real estate, overhead, energy, and environmental costs. The Telework Enhancement Act of 2010 established a statutory framework for implementing a comprehensive federal telework program.[Footnote 1] The act requires executive agencies to establish and to implement a policy under which eligible employees may be authorized to telework. The act also requires OPM to report annually to Congress on the telework programs of each agency, including the level of participation in telework in each agency, each agency's goal for increasing employee participation in telework, agency progress during the reporting period toward meeting telework goals, and reasons for significant increases or decreases from previous year telework participation. In 2010, OPM set a high-priority performance goal of increasing by 50 percent the number of eligible federal workers who telework by fiscal year 2011. To measure agency progress toward this and agency telework goals, OPM will use data it collects through an annual survey referred to as the data call. However, concerns have existed about the reliability of these data. According to OPM, in order to help agencies overcome barriers related to telework and raise federal telework participation rates, the agency needs reliable, systematically collected agency data. Because of the new telework reporting requirements in the act and the need for reliable data to assess the agencies' progress in implementing their individual telework programs, you asked us to assess OPM's: * actions to respond to the reporting requirements of the Telework Enhancement Act of 2010, and: * handling of identified data reliability issues in the 2011 telework data call. To answer these questions, we reviewed relevant reports and guidance published by OPM that describe the status of telework programs across executive agencies, and previous telework data calls and their instructions. We also reviewed our previous reports on telework and the reliability of OPM's telework data. We identified the relevant generally accepted survey standards as promulgated by the Office of Management and Budget and compared OPM's design and implementation of the 2011 telework data call against these standards. We interviewed the OPM officials responsible for the planning, design, implementation, and analysis of the 2011 telework data call. This included discussions on the role of the Interagency Telework Measurement Group (ITMG), the process for developing the definitions and key terms used in the data call, the training and assistance provided to executive agency officials responsible for completing the data call, and agency plans to address outstanding data reliability issues associated with the data call.[Footnote 2] We were also able to simulate completion of the web-based data call that allowed us to respond to the data call in the same manner as the other agency respondents. This simulation gave us the opportunity to determine the ease of use of the online data call instrument, and whether there were any potential design or functionality issues that could affect the validity and reliability of the data. We conducted this performance audit from June 2011 through April 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Background: The Telework Enhancement Act of 2010, enacted in December 2010, requires each executive agency to designate a telework managing officer, develop training programs, establish a telework policy, and submit an annual report to the Chair and Vice Chair of the Chief Human Capital Officers Council on the agency's efforts to promote telework. Under the act, OPM is to play a leading role in helping executive agencies implement the new telework provisions, which include setting telework goals and establishing qualitative and quantitative measures. The law requires OPM to provide policy and guidance for telework in several areas, including pay and leave, agency closure, performance management, official worksite information, recruitment and retention, and accommodations for employees with disabilities. In an annual report to Congress, OPM is to assess each agency's progress toward goals for participation and other goals relating to telework, such as emergency readiness. The first of these reports under the act is due to Congress in June 2012. Since 2002, OPM has used a telework survey--the data call--to annually collect information from the executive agencies in order to provide Congress with a report on the status of telework across these agencies.[Footnote 3] OPM conducts this data call to determine the extent to which agency employees are teleworking and to gauge agency progress in various aspects of their telework programs, such as participation, policy, eligibility, cost savings, and technology, as well as to provide examples of barriers agencies face to implementing telework programs. However, throughout the past decade, OPM has been concerned about the reliability of the telework data it receives from executive agencies because, although data reported from agencies have improved, OPM continues to consider it an estimate of telework participation and frequency.[Footnote 4] In its 2003 and 2007 telework reports to Congress, OPM raised concerns about the ability of agencies to track employee participation in their telework programs.[Footnote 5] In its 2008 report, OPM identified weaknesses in the methodology most agencies used to collect and report telework participation data and OPM stated that inconsistencies within data systems and inaccuracies triggered by hand-counting telework agreements could affect data reliability.[Footnote 6] OPM cautions that existing measures of telework participation are a barrier to measuring any increase in telework as the measures vary widely in validity and reliability and limit the capability of any federal body to track the actual level and frequency of telework participation.[Footnote 7] At the request of Congress, we have previously reported on telework programs across the federal government and have made recommendations related to the reliability of agency-reported data. In a 2005 report, we reviewed the telework data for five federal agencies and found they had reported the total number of employees who were eligible to telework, but had included individuals who were, in fact, excluded from participation based on various criteria such as employee performance, thereby raising concerns about the reliability of the telework data reported by these agencies. In addition, none of the agencies could report the actual number of employees who teleworked or how often they did so because none had fully implemented the capability to track this through their time and attendance systems. [Footnote 8] Our 2007 testimony reiterated our concern that agencies were measuring employee participation in telework based on their potential to telework rather than their actual usage.[Footnote 9] More recently, we reported that since the 2004 data call, OPM asked agencies if they had integrated telework into agency emergency and continuity of operations plans, but agencies had no guidance as to what constitutes incorporating telework into continuity and emergency planning. This lack of a definition or description raised concerns about the reliability of reports on this matter.[Footnote 10] OPM Revised the 2011 Telework Data Call to Prepare for Reporting under the Telework Enhancement Act of 2010: In response to the Telework Enhancement Act of 2010, OPM revised the 2011 data call and provided instructions to executive agency respondents that incorporated common definitions and standards to use in providing OPM with their agency data. The revisions and additions to the 2011 data call were developed in consultation with an Interagency Telework Measurement Group (ITMG), which OPM formed in January 2011. (See appendix II for a comparison of definitional and instructional changes from the 2010 and 2011 telework data calls.) The OPM official designated with leading the ITMG said 10 agency officials from 7 agencies were selected for the group because of their knowledge of federal telework programs. According to the OPM official, the ITMG provided expertise in telework program implementation, policy, and methodology development, work/life balance programs, and expertise in research methods, such as surveys. The ITMG met on biweekly from January 2011 until July 2011 and resumed biweekly meetings in September 2011 with the goal of addressing three primary topics: * Definitions of key terms, such as telework, eligibility, and employee, to use in the 2011 data call. ITMG interpreted some requirements of the act and developed additional instructions to encourage common reporting methodology across the agencies. For example, according to OPM officials, ITMG clarified the definition of eligibility in light of agencies' concerns that the act did not specifically define the categories of employees that should be eligible to participate in their agencies' telework programs, and therefore notified about their eligibility to telework. The group instructed respondents to ensure they excluded military and contract personnel as employees when reporting their telework data. The group also clarified that respondents should include full-time, part-time and intermittent employees when responding to questions about telework participation and frequency. * Revision and/or addition of data call questions. For example, OPM officials stated that in collaboration with the ITMG, they clarified the definition of telework to specifically state that telework includes what is generally referred to as remote work, but excludes mobile work and work done on official travel.[Footnote 11] OPM officials added a new question to capture the number of mobile workers. This addressed a reliability problem from the previous data call when some agencies included mobile work in reporting telework. * Revision and development of data collection instruments, in addition to the data call, to collect telework information. For example, ITMG worked to revise telework-related items in the Federal Employee Viewpoint Survey, an OPM data collection instrument that gauges employees' perceptions of their agency. In the 2011 survey three out of 84 questions focused on telework. In the past, OPM has found this survey to provide complementary employee views on telework. The ITMG also assisted in developing focus groups of telework managing officers and telework coordinators to identify issues, challenges and strategies associated with implementing telework programs at the agency level, such as successful telework implementation strategies, as well as barriers to telework. According to an OPM official, these revisions also included questions that may enable them to better understand the differences between telework programs across executive agencies, including differences in training on telework, use of technology, and how agencies responded to the requirements of the act. This official said such information will help inform the development of telework programs. OPM also made changes to the 2011 data call time period for which employee telework participation and frequency was to be reported. This change was made to allow agencies time to develop telework policies in accordance with the act and to allow OPM time to meet its reporting obligation under the act (see figure 1). In previous data calls, OPM asked agencies for telework data during the calendar year (12 months), if available. OPM reduced the time period for the 2011 data call to 4 weeks, as it decided this was the best methodology to meet its reporting requirements under the act. Agencies were to select a 4 week period during September and October on which to report. In addition to the change in time period for requested telework data, the 2011 data call asked for more detail on employee telework participation, frequency of employee telework participation, and additional information on telework policy and program implementation, and telework goals as required by the act. Figure 1: Differences in Time Period of Data Requested and Reported between 2010 and 2011 Telework Data Calls: [Refer to PDF for image: illustration] 2010 telework data call: Time period for which OPM requested telework data from agencies: January 2009 - December 2009. Agencies report telework data through the data call: February 2010 - March 2010. OPM reports federal government telework data to Congress: February 2011. OPM compressed the period for which it requested agencies to report telework data from the 2010 to 2011 data call. 2011 telework data call: Time period for which OPM requested telework data from agencies: September - October 2011. Agencies report telework data through the data call: October - December 2011. OPM reports federal government telework data to Congress: June 2011. Source: GAO analysis of OPM information. Note: From April 2010 through January 2011, OPM verified, analyzed, and drafted its 2010 report to Congress on telework. From January 2012 through May 2012, OPM will be verifying, analyzing, and drafting its report to be submitted to Congress in June 2012. [End of figure] OPM officials believe the information they obtained from the 2011 data call will enable the agency to satisfy some of the act's reporting requirements, but for this report to Congress, OPM cannot fulfill other requirements. OPM officials stated that it is not feasible for OPM to measure agencies' progress against their Telework Enhancement Act of 2010 goals, since these goals were established in June 2011, and the time period for agencies' actual participation data was September/October 2011--just 3 to 4 months after agencies established their goals. However according to OPM officials, based on the information collected, OPM will be able to report the percentage and frequency of telework at individual agencies. To communicate changes to the 2011 data call, OPM officials increased their training efforts to aid executive agency officials in developing a common understanding of terms, key concepts, and the objectives of the data call. According to OPM officials, in July 2011, OPM officials responsible for the data call met with agency respondents to provide an introduction and overview to the 2011 data call. The meeting covered the new requirements under the act, and the planned timeframe for agency reporting and OPM processing and analysis of the data collected through the data call. OPM could not require agency officials to attend. Nonetheless, OPM reminded agency officials responsible for the data call, that it was important that they attend both September and October training sessions being offered by OPM. The September training session covered the data call questions and incorporated specific content of the near final data call. The October training session reviewed specific instructions on how to enter information into the online data call form, in addition to reviewing instructions and questions from respondents. While some of the information provided at the two training sessions was similar, each session contained some new information usually in response to questions raised at a previous session. OPM staff also maintained and disseminated via email a list of the most frequently asked questions posed by data call respondents. Various Methods of Data Collection and Ongoing Changes to the Data Call Make It Difficult to Assure Data Reliability and to Make Year-to- Year Comparisons: The act requires OPM to report, among other things, year-to-year executive agency progress on the number of employees who telework. To accomplish this, OPM needs reliable baseline data on telework participation to be able to make year-to-year comparisons. However, OPM officials expressed continuing concerns over the reliability of quantitative participation and frequency data submitted by the agencies through the 2011 telework data call. OPM officials explained that 2011 is a transition year for telework programs across executive agencies, and some agencies made changes to their policies to bring their telework programs into compliance with the requirements of the act. For example, some agencies were implementing new data collection systems while collecting data on telework. These agencies first needed to create and implement new policies, and then consider and establish processes related to the collection of telework data to report to OPM. OPM officials responsible for preparing the report to Congress stated that because of these changes, it would not be appropriate to compare the 2011 data to data collected in prior years. OPM will not be able to make comparisons to prior years due to the previously discussed modifications to the data call, such as changes to definitions. Definitional changes, including the change to specifically exclude mobile workers, could result in agencies interpreting and reporting telework participation and frequency of telework differently than they did in the past. While these changes should improve the consistency of data going forward, according to OPM officials there is no way that OPM can assure that all agencies would provide comparable responses to the same data call questions. Moreover, OPM officials have said that key terms and definitions for the 2012 data call may continue to evolve. If OPM reports agency telework progress based on data collected using definitions revised year to year, OPM may reach erroneous conclusions, although OPM officials have said they are taking steps to try and prevent this possibility. Another reason why comparisons to prior data calls would be invalid is because of the changes OPM made to the time period when agencies reported telework data. Maintaining consistent data series over time necessitates using consistent data collection procedures for ongoing data collections. OPM said that changing the reporting period from 1 calendar year to 4 weeks may result in greater data consistency because in previous data calls, agencies may have reported data from different time periods within a calendar year. Now all agencies will report data from a narrower and more similar time frame. However, there are no available studies to support that the new time period is representative or "typical" of other months in total or of the experience of particular agencies. OPM officials said that they will need to indicate this in their report to Congress. While OPM made changes to the 2011 data call to allow it to meet some of its reporting requirements under the act and better assist agencies in responding to the data call, our analysis found the 2011 data call did not fully meet some generally accepted survey standards. According to these standards, to be valid, survey questions must adequately represent the concept or behavior in question and consistently predict outcomes. Moreover, questions must be designed and asked so that each recipient will understand and answer the same question in the same way. But for the 2011 data call, there can be no assurance that all respondents were aware of associated definitions and instructions provided by the training sessions and the frequently asked questions (FAQ). Although OPM invested more in providing training in preparation for the 2011 data call than in previous data calls and it disseminated training slides to all invitees of its final training session, attendance lists were not recoverable for all training sessions so there can be no assurance that all respondents received training or reviewed the slides. Consequently, some data call respondents may not have been aware of the definitions and instructions provided in the training sessions or in the FAQ. Additionally, some of the information provided in the training sessions was inconsistent. For example, OPM officials said that during the last training session they instructed respondents to report an employee "telework day" if the employee teleworked for any portion of a work day.[Footnote 12] However, this clarification of "telework day" was not given in either the July overview meeting or in the other training session, and not clearly included in the instructions in the survey instrument or distributed in the FAQ. This information may have been important to agencies reporting on situational telework through automated systems intended to capture more precise data. OPM officials also explained that responses to certain questions should reference the same time period, and this information was not available in the data call instructions.[Footnote 13] During one training session, an OPM official said she instructed those participants who determined employee participation by counting telework agreements to limit the agreements counted to those in effect after the agency implemented its telework program under the act; however, this information was not available in the online data call instructions. Uncertainty about whether data call respondents attended both the September and October training sessions, and variations in training sessions, could cause agencies to have different understandings of data call concepts and terms. OPM recognizes that the existing measures of federal telework participation vary in validity and reliability, which affects agencies' ability to report accurate data, and it is taking steps to verify data submitted by respondents to provide a more accurate picture of telework in the federal government based on current definitions and collection methodologies. However, as a result of issues raised above, respondents may have provided inconsistent or inaccurate data on topics required by the act. OPM officials anticipate that telework data will be more reliable next year because of the expected governmentwide implementation of automated data collection based on time and attendance records. As we have reported, OPM has concluded from research that the most reliable telework data are collected through time and attendance tracking systems.[Footnote 14] Data collected through automated systems eliminate the need to track telework data by counting telework agreements or relying on estimates. Since 2003, OPM has consistently expressed concerns about the methods agencies use to collect telework data. In its previous telework reports to Congress, OPM has advocated for the development of an automated data collection system. OPM officials noted that OPM does not control what telework data responding agencies maintain, or their methods of data collection. Executive agencies provide telework participation and frequency data using a variety of methods, such as relying on estimates, counting telework agreements, and using automated time and attendance records to track telework participation. In an effort to collect more uniform data across agencies, OPM officials are standardizing definitions and data elements for use in automated time and attendance systems. For example, OPM has identified routine telework hours in a pay period as a data element for automated data collection and provided a standardized definition that will be used by all agencies using the Enterprise Human Resources Integration (EHRI) system.[Footnote 15] OPM has introduced a timeline for modifying the existing EHRI system to allow OPM to collect telework data from executive agencies. According OPM, agencies will begin piloting these automated data collection systems for the 2012 telework data call. OPM began to discuss automated governmentwide data collection with the ITMG in July 2011. According to OPM's timeline, OPM began to communicate internal requirements for automated telework data collection to telework managing officers in March 2012, but an OPM official has stated they do not expect full automation of telework data until 2013. This official also noted that different agencies have varying abilities to implement this new type of data collection and reporting mechanism, and considering different levels of comfort with new systems, it will take time to adjust to this method. However, continuous improvement efforts sometimes result in a trade-off between the desire for data consistency and a need to improve data collection and maintenance of a consistent data series over time. Because of the eventual planned move to automation, OPM may not be able to use 2011 data as a baseline. With the planned change to the method of data collection, it may not be possible to compare the 2011 data to future data. The 2011 data call requested data for a 2-month period, and some data call respondents relied upon estimates. Planned automation can provide a more uniform and accurate method for collecting telework data, however it may make comparisons using the 2011 telework data as a baseline difficult. However, OPM officials believe the 2011 data will provide an improved report of telework status because of standardized definitions and the more uniform time period of data collection. According to OPM officials, some executive agencies will need time to become comfortable with automated reporting systems and, during a period of transition to a new system, there could be initial reliability issues. However, these officials said that the 2011 data, notwithstanding its limitations, will be useful in identifying and understanding any major agency changes in reported participation that could occur during a transition to automated data collection. Such changes could alert OPM to possible transition related issues in agencies' conversion to automated governmentwide data call collection efforts in 2012 and 2013. In addition, for those agencies already responding on the basis of time and attendance reporting, major changes in agency responses provide OPM the opportunity to confirm with agencies that the uniform definitions are being consistently applied. Conclusions: The Telework Enhancement Act of 2010 requires OPM to collect telework data and report annually to Congress, which emphasizes the need for telework data to be valid and reliable. OPM revised the 2011 telework data call in order to allow it to meet some of the act's reporting requirements and assist agencies in responding to the data call. This revision resulted in changes to terminology, as well as changes in the collection time period of requested telework data. OPM provided greater assistance to agencies through training on changes to the 2011 data call to improve accuracy of agency reporting of telework participation and frequency for the data call reporting period of September and October 2011. However, agencies use various methods, which OPM does not control, to report, collect, and maintain telework data, and this could affect the reliability of the telework data submitted. In addition, variation in training sessions and OPM's uncertainty as to whether all respondents attended training, could lead to respondents' potential misunderstanding of important terms and instructions. The validity and reliability of the reported 2011 telework data for some of the responding agencies may be questionable, and therefore agency telework participation and frequency data will not likely be comparable with previous data calls because of differences in definitions used, time periods of reporting, and individual agency tracking methods. With the revised 2011 data call, OPM establishes a baseline it could use to conduct a limited crosscheck of data collected through a governmentwide automated telework data collection system, which OPM plans to implement over the course of 2012 and 2013. OPM expects that automated data collection will provide it increasingly more reliable data on which to report progress. However, these efforts to improve future automated data collection may result in changes to agencies' methods of data collection and a trade-off between the desire for consistency with previous data calls for comparison purposes and a need to improve overall data collection. Recommendations for Executive Action: To improve OPM's annual reporting of telework to Congress, we recommend that the OPM Director take the following two actions: * Ensure that the reliability limitations related to the 2011 telework data call are clearly reported in its June 2012 report to Congress by fully describing how existing measures of telework participation vary widely in validity and reliability and limit the capability of OPM to reliably report the actual level and frequency of telework participation. * Continue efforts to improve data collection and gather information that allows for the appropriate qualification of year-to-year comparisons and informs users about the effects of data collection changes going forward. Agency Comments and Our Evaluation: We provided a draft of this report to the Director of OPM for review and comment. The Associate Director of OPM provided written comments, which we have reprinted in appendix III. In summary, OPM partially concurred with our first recommendation and fully concurred with the second. OPM highlighted a number of actions the agency has under way or plans to undertake in response. For the first recommendation, OPM noted that inadequate methods of data collection exist at the agency level and OPM continues to address this data reliability issue through training on evaluation and measurement. While this is an important step in addressing data reliability issues, OPM should ensure that telework data reliability limitations are clearly reported in their annual reports to Congress. For the second recommendation, OPM noted its continued plans to automate collection of telework data, and to regularly meet with telework managing officers and telework coordinators to keep them updated on changes to telework policy and data collection. OPM also provided a number of technical comments, which we incorporated as appropriate. We are sending copies of this report to the Chairman and Ranking Member of the Subcommittee on the Federal Workforce, U.S. Postal Service and Labor Policy, Committee on Oversight and Government Reform, House of Representatives; and the Director of OPM. In addition, this report will be available at no charge on the GAO website at [hyperlink, http://www.gao.gov]. If you have any questions about this report, please contact me at 202-512-6806 or jonesy@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributions to this report are listed in appendix IV. Signed by: Yvonne D. Jones: Director, Strategic Issues: [End of section] Appendix I: Objectives, Scope, and Methodology: This report (1) describes the Office of Personnel Management's (OPM) actions to respond to the requirements of the Telework Enhancement Act of 2010 and (2) assesses how OPM is handling and addressing identified data reliability issues in the 2011 telework data call. To address these two objectives, we reviewed relevant reports and guidance published by OPM that describe the status of telework programs across executive agencies and previous telework data calls and their instructions. We also reviewed previous GAO reports on telework and the reliability of OPM's telework data. Lastly, we interviewed the OPM officials responsible for the planning, design, implementation, and analysis of the 2011 telework data call. This included discussions on the role of the Interagency Telework Measurement Group (ITMG), the process for developing the definitions and key terms used in the data call, the training and assistance provided to executive agency officials responsible for completing the data call, and agency plans to address outstanding data reliability issues associated with the data call. We conducted additional analysis to answer selected objectives as described below. To assess the extent to which the 2011 telework data call met generally accepted survey methodology, GAO internal experts in survey research identified principles from the Office of Management and Budget's (OMB) Standards and Guidelines for Statistical Surveys relevant to assessing the 2011 telework data call.[Footnote 16] We also used relevant aspects of GAO's guide to Developing and Using Questionnaires.[Footnote 17] Using the OMB principles, two analysts independently reviewed the data call, supporting documentation, and clarifying information provided in interviews to assess the extent to which the data call methodology met research practices. The initial rate of agreement across 15 rated practices, including those rated as having insufficient information to judge, was 12 of 15, or 80 percent agreement. The rating for the three practices on which there was initial disagreement were reconciled by the two analysts conducting the review and the reconciled rating was then reviewed by a third analyst with survey expertise. The third analyst did not recommend any changes to the reconciled ratings. We also simulated completion of the Web-based data call, accessing and responding to the data call in the same manner as executive agency respondents. Table 1 outlines the generally accepted survey research principles, derived from OMB's guidelines, which we used in our assessment. This is not an exhaustive list of all OMB guidelines.[Footnote 18] When we completed this review in April of 2012, OPM had not yet completed analyzing and reporting on the results of telework data call. Based on this, we could not yet assess whether the data call met all of the OMB principles related to data analysis and reporting. Some OMB principles, such as those related to sample design, were not appropriate to apply to the telework data call. These principles were therefore excluded as not relevant to our review. Table 1: Principles Used to Assess the Data Call: Development of Concepts, Methods, and Design: 1. The survey had a clear rationale; 2. Potential users were consulted to identify their requirements and expectations; 3. A review of existing, surveys, studies, and reports, or other literature informed the survey; 4. The scope of survey data items are defined and justified; 5. To ensure data remain consistent over time, key variable and survey procedures remained consistent; 6. A clear, logical, and easy-to-follow flow of questions from a respondents' point of view; 7. The questionnaire was pretested and all components of the final survey system were field tested. Collection of data: 1. The sample frame was appropriate; 2. The respondents were provided appropriate informational materials; 3. Questions are clearly written; 4. The questionnaire design minimized respondent burden and maximized data quality; 5. Response maximization efforts were appropriate; 6. Agency staff developed proper protocols for handling respondent questions to help ensure data quality; 7. Protocols to monitor data collection activities were developed. Processing and editing of data: 1. Appropriate checks and edits on the data collection system mitigated errors. Source: GAO and OMB, Standards and Guidelines for Statistical Surveys (Washington, D.C.: Sept. 2006). [End of table] We conducted this performance audit from June 2011 through April 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. [End of section] Appendix II: Revisions to OPM's Telework Data Call to Collect Information to Prepare for Reporting under the Telework Enhancement Act of 2010: Instructions: 2010 Telework data call: Time period: Based on the agency's calendar year (Jan. 1 to Dec. 31, 2009); 2011 Telework data call: Time period: Based on the months of Sept. and Oct. 2011[A]. 2010 Telework data call: Participation: agencies invited to participate in data call; 2011 Telework data call: Participation: agencies required to submit telework data to OPM. 2010 Telework data call: Reporting deadline: March 2010; 2011 Telework data call: Reporting deadline: Dec. 2011. Definitions: 2010 Telework data call: Telework: "Telework refers to any arrangement in which an employee regularly performs officially assigned duties at home or other work sites geographically convenient to the residence of the employee"; 2011 Telework data call: Telework: "Telework is a work arrangement that allows an employee to perform work, during any part of regular, paid hours, at an approved alternative worksite (e.g., home, telework center). This definition of telework includes what is generally referred to as remote work but does not include any part of work done while on official travel or mobile work. See the following clarifications on remote and mobile work as telework." 2010 Telework data call: Employee: Not defined; 2011 Telework data call: Employee: Employee refers to federal civilian employees excluding military personnel and contractors. Employee may also include full-time, part- time, and/or intermittent employees. 2010 Telework data call: Eligibility: Agencies have the discretion to determine the telework eligibility requirements for their employees. For reporting purposes, in this Call for data, ALL employees are considered eligible EXCEPT those employees whose: * positions require, on a daily basis (every work day), direct handling of secure materials, or on-site activity that cannot possibly be handled remotely or at an alternate worksite; * most recent federal government performance rating of record (or its equivalent) is below fully successful or conduct has resulted in disciplinary action within the last year; 2011 Telework data call: Eligibility: An employee is eligible to participate in telework if all of the following parameters are true: * The employee has not been officially disciplined for being absent without permission for more than 5 days in any calendar year; * The employee has not been officially disciplined for violations of subpart G of the Standards of Ethical Conduct for Employees of the Executive Branch; * Teleworking does not diminish the employee's performance or agency operations; * Participation and performance complies with the requirements and expectations of his/her telework agreement; * The employee's official duties do not require on a Full daily basis (All Day, every work day): - direct handling of secure materials determined to be inappropriate for telework by the agency head; or; - on-site activity that cannot be handled remotely or at an alternate worksite; * The employee and/or the employee's position are not disqualified based on additional criteria established by the organization. 2010 Telework data call: Types of telework: None provided; 2011 Telework data call: Types of telework: * Routine: telework that occurs as part of an ongoing, regular schedule, and; * Situational: telework that is approved on a case-by-case basis, where the hours worked were not part of a previously approved, ongoing and regular telework schedule (e.g., telework as a result of special work assignments or doctor appointment)[B]. Source: OPM. Notes: This table is intended to illustrate the differences between the 2010 and 2011 data calls. OPM made changes to the 2011 data call in response to the Telework Enhancement Act of 2010. [A] Under the Telework Enhancement Act of 2010, agencies were required to establish telework policies by June 2011. [B] Situational telework is sometimes also referred to as episodic, intermittent, unscheduled or ad-hoc telework. [End of table] [End of section] Appendix III: Comments from the Office of Personnel Management: United States Office of Personnel Management: Employee Services: Washington, DC 20415: April 4, 2012: Mr. William Doherty: Assistant Director: U.S. Government Accountability Office: 441 G St, NW: Washington DC 20548: Dear Mr. Doherty: Thank you for providing a draft of the Government Accountability Office (GAO) report, "Government Telework: Program Measurement Continues to Confront Data Reliability Issues" to the U.S. Office of Personnel Management (OPM). We appreciate the opportunity to provide you with comments about this report. Response to Recommendations: Recommendation: Ensure that the reliability limitations related to the 2011 telework data call are clearly reported in its June 2012 report to Congress by fully describing how existing measures of telework participation vary widely in validity and reliability and limit the capability of OPM to reliably report the actual level and frequency of telework participation. Management Response: We partially concur. OPM will continue to report limitations of the methodology used to collect telework participation data as it has in prior reports, including the 2010 report to Congress. However, with its focus on the data call, the GAO report implies that improvements to that instrument will address reliability issues. This focus minimizes the much larger issues that exist at the agency level and pose the greater threat to data reliability. The data call provides only a structured electronic platform for individual agencies to report the results of their own data collection efforts. The measurement of telework participation currently occurs at the level of the individual agency. While an appreciation of evaluation has grown in Government in recent years, few Work/Life Agency staffs are adequately trained in evaluation methodology or data collection. As a consequence, inadequate methods for data collection prevail within agencies, and many, as GAO noted, do not adequately capture telework behavior. OPM continues to address this much greater threat to the reliability of reported telework participation through agency training on the principles of evaluation and measurement. Also, as discussed with GAO during the engagement, we have developed a common coding system and method for data collection in order to address the threat to reliability posed by widely varying data collection methods for telework participation. In this effort, OPM has worked with the Federal Shared Service Centers to automate the collection of telework data via the Enterprise Human Resources Integration (EHRI) Human Resource (HR) and Payroll data feeds. An employee's telework eligibility and usage will be collected, summarized and reported to OPM via the monthly EHRI FIR-Status data feed and biweekly EHRI payroll data feed. Initiating this effort, OPM recently released new data standards for the telework data being collected via EHRI. In addition, an updated version of the Government- wide reporting requirements contained in the Guide to Human Resources Reporting (GHRR) was recently published. Recommendation: Continue efforts to improve data collection and gather information that allows for the appropriate qualification of trend analyses and informs users about the effects of data collection changes going forward. Management Response: We concur. OPM will continue to hold frequent information meetings with Telework Managing Officers and telework coordinators to apprise them of any changes in policy and data collection. The April 2012 meeting will provide agencies with details regarding the automated data collection system currently being implemented as described above. The automated system will provide data appropriate to more sophisticated analyses, including examination of trends to the extent appropriate and, more importantly, provide OPM with the data necessary to assess important outcomes with telework. Technical comments to the draft report are enclosed. Unless otherwise noted, the suggested revisions are meant to provide technical accuracy. Please contact Ms. Janet Barnes, Deputy Director, Internal Oversight & Compliance on (202) 606-3270, should your office require additional information. Again, my thanks to your office for providing this opportunity to update and clarify information in the draft report. Sincerely, Signed by: Angela Bailey: Associate Director: Employee Services: Enclosure: [End of section] Appendix IV: GAO Contact and Staff Acknowledgments: GAO Contact: Yvonne D. Jones, (202) 512-6806 or jonesy@gao.gov: Staff Acknowledgments: In addition to the contact named above, William Doherty, Assistant Director, and Keith O'Brien, analyst-in-charge, led the development of this report. Virginia Chanley, Patricia Donahue, Robert Gebhart, Jill Lacey, and Joseph Santiago made significant contributions to this report. Karin Fangman provided legal counsel. Shirley Hwang, Jessica Nierenberg, and Kathleen Padulchick verified the information in the report. [End of section] Related GAO Products: Emergency Preparedness: Agencies Need Coordinated Guidance on Incorporating Telework into Emergency and Continuity Planning. [hyperlink, http://www.gao.gov/products/GAO-11-628]. Washington, D.C.: July 22, 2011. Human Capital: Telework Programs Need Clear Goals and Reliable Data. [hyperlink, http://www.gao.gov/products/GAO-08-261T]. Washington, D.C.: November 6, 2007. Human Capital: Greater Focus on Results in Telework Programs Needed. [hyperlink, http://www.gao.gov/products/GAO-07-1002T]. Washington, D.C.: June 12, 2007. Agency Telework Methodologies: Departments of Commerce, Justice, State, the Small Business Administration, and the Securities and Exchange Commission. [hyperlink, http://www.gao.gov/products/GAO-05-1055R]. Washington, D.C.: September 27, 2005. Human Capital: Key Practices to Increasing Federal Telework. [hyperlink, http://www.gao.gov/products/GAO-04-950T]. Washington, D.C.: July 8, 2004. Human Capital: Further Guidance, Assistance, and Coordination Can Improve Federal Telework Efforts. [hyperlink, http://www.gao.gov/products/GAO-03-679]. Washington, D.C.: July 18, 2003. [End of section] Footnotes: [1] Pub. L. No. 111-292, 124 Stat. 3165 (Dec. 9, 2010), codified at chapter 65 of title 5, United States Code. [2] The Interagency Telework Measurement Group consisted of officials from OPM, General Services Administration, United States Patent and Trademark Office, Department of the Navy, Defense Information Systems Agency, National Aeronautics and Space Administration, and Department of Energy. [3] OPM, Report to Congress: Status of Telework in the Federal Government, 2010 (Washington, D.C.: Feb. 2011). [4] OPM considers automated data collection through time and attendance to be a more reliable method of collecting telework data and the number of agencies collecting data this way has increased over the years. [5] OPM, Report to Congress, Status of Telework in the Federal Government, 2003 (Washington, D.C.: 2003) and OPM, Report to Congress, Status of Telework in the Federal Government, 2007 (Washington, D.C.: Dec. 2007). [6] OPM, Report to Congress, Status of Telework in the Federal Government, 2008 (Washington, D.C.: Dec. 2008). [7] OPM, Congressional Budget Justification, Performance Budget Fiscal Year 2012 (Washington, D.C.: Feb. 2011). [8] GAO, Agency Telework Methodologies: Departments of Commerce, Justice, State, the Small Business Administration, and the Securities and Exchange Commission, [hyperlink, http://www.gao.gov/products/GAO-05-1055R] (Washington, D.C.: Sept. 27, 2005). [9] GAO, Human Capital: Telework Programs Need Clear Goals and Reliable Data, [hyperlink, http://www.gao.gov/products/GAO-08-261T] (Washington, D.C.: Nov. 7, 2007). [10] GAO, Emergency Preparedness: Agencies Need Coordinated Guidance on Incorporating Telework into Emergency and Continuity Planning, [hyperlink, http://www.gao.gov/products/GAO-11-628] (Washington, D.C.: July 22, 2011). [11] OPM defines mobile work as work characterized by routine and regular travel to conduct work in customer or other worksites as opposed to a single authorized alternative worksite (i.e., site audits). [12] Under this definition, if an employee teleworked 2 hours, it should be reported as a "telework day". Therefore, a "telework day" is not related to the number of hours of telework but rather the number of work days during which any telework occurred. [13] In the October training session, OPM explained that questions asking about the total number of agency employees and the total number of employees who teleworked (for participation data) should come from the same time period. However, the data call questions asking for these two totals are not drawn from the same time period. [14] GAO, Human Capital: Further Guidance, Assistance, and Coordination Can Improve Federal Telework Efforts, [hyperlink, http://www.gao.gov/products/GAO-03-679] (Washington, D.C.: July 18, 2003). [15] According to the OPM website, the EHRI system offers a standardized approach for human resources record conversion, storage, access, sharing, and archiving. [16] OMB, Standards and Guidelines for Statistical Surveys (Washington, D.C.: Sept. 2006). See [hyperlink, http://www.whitehouse.gov/omb] (last accessed Nov. 10, 2011). [17] GAO, Developing and Using Questionnaires, [hyperlink, http://www.gao.gov/products/GAO/PEMD-10.1.7] (Washington, D.C.: Oct. 1993). [18] OMB standards and guidelines are also not designed to be completely exhaustive of all efforts that an agency may undertake to ensure the quality of its data. OMB encourages agencies to develop additional standards focused on their specific activities. [End of section] GAO’s Mission: The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of GAO Reports and Testimony: The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website [hyperlink, http://www.gao.gov]. Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to [hyperlink, http://www.gao.gov] and select “E- mail Updates.” Order by Phone: The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, [hyperlink, http://www.gao.gov/ordering.htm]. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO: Connect with GAO on facebook, flickr, twitter, and YouTube. Subscribe to our RSS Feeds or E mail Updates. Listen to our Podcasts. Visit GAO on the web at [hyperlink, http://www.gao.gov]. To Report Fraud, Waste, and Abuse in Federal Programs: Contact: Website: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]; E-mail: fraudnet@gao.gov; Automated answering system: (800) 424-5454 or (202) 512-7470. Congressional Relations: Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125 Washington, DC 20548. Public Affairs: Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548.