Research Reliability: Federal Actions Needed to Promote Stronger Research Practices
Fast Facts
While scientific discoveries can be serendipitous, reliable research is more often built on successive studies that employ rigorous design, documentation, and confirmation.
The U.S. government funded over $42 billion in science research in 2019. How can federal agencies help assure the reliability of that research?
Experts and others have suggested that agencies implement practices such as disclosing results and methods earlier, as well as incentivizing researchers to use rigorous methods and make their research more transparent.
We recommended that agencies collect information to learn how best to implement such practices.
Highlights
What GAO Found
The National Institutes of Health (NIH), the National Science Foundation (NSF), and the National Aeronautics and Space Administration (NASA) are the three largest federal funders of basic scientific research in the United States. According to leading experts GAO interviewed, these agencies could do more to increase the rigor and transparency of the research they fund by taking actions to better align awards and recognition for researchers with more rigorous and transparent research practices. Experts suggested, for example, that agencies could incentivize or mandate that researchers preregister their studies as a means to share their research plans before the research is conducted. Doing so would enable other researchers to comment on and strengthen the methodology and analysis plans. Experts further suggested that agencies help improve standards for data repositories where research data are stored publicly, encourage the publication of null research results, and support training in statistical analysis and study design. Although the scientific community has developed many such practices to enhance research reliability, GAO found that they are not widely adopted because of researcher misconceptions and misaligned incentives in funding and publishing, among other things.
Role of Rigor and Transparency in Research Reliability
NIH, NSF, and NASA have taken steps to promote and support additional rigor and transparency in research, such as establishing requirements for researchers to disclose research results and associated data publicly. However, these agencies largely rely on grant application reviews and the prepublication peer review process to help ensure research rigor. GAO found that these agencies do not evaluate the rigor and transparency of the research they fund to help identify strategies for improvement. Specifically, they do not collect indicators of rigorous study design and transparency of research results such as study sample size, adherence to research plans, or the extent to which research data are findable, accessible, and usable. As a result, the agencies lack information to support changes to the grant making process and research funding priorities. Federal guidance and Standards for Internal Control in the Federal Government call for agencies to prioritize making federally funded research more rigorous and transparent and to use quality information to achieve agency objectives. Without this information on the research they fund, agencies are limited in their ability to take effective actions to improve research reliability, like those the experts described to GAO.
Why GAO Did This Study
In 2019, the U.S. government funded more than $42 billion in basic scientific research across a wide range of scientific disciplines. Unsuccessful attempts to reproduce and replicate research results have been documented across many scientific disciplines, including those funded by NASA, NIH, and NSF. The scientific community has expressed concern over the difficulty of replicating prior research results.
GAO was asked to review strategies to improve the reliability of federally funded research. Among other things, this report (1) examines what actions, according to experts, federal agencies could take to foster rigor and transparency in the research they fund; and (2) assesses the extent to which selected federal science funding agencies have taken actions to improve rigor and transparency. GAO conducted a literature review; reviewed NIH, NSF, and NASA documents; and conducted four roundtable discussions with 22 experts. GAO also interviewed agency officials as well as stakeholders from academia, professional societies, publishing, and other parts of the scientific community.
Recommendations
GAO is making six recommendations, two each to NIH, NSF, and NASA to evaluate research using indicators of rigor and transparency, and to use this information to inform further actions. NIH and NSF concurred with the recommendations. NASA did not concur with our first recommendation and partially concurred with our second. GAO continues to believe the recommendations are valid.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| National Institutes of Health | The Director of NIH should collect information on relevant indicators of rigor to assess the research projects the agency funds, and implement steps, as needed, to promote strong research practices in future work. (Recommendation 1) |
HHS agreed with this recommendation. In December 2022, HHS said it plans to issue a Notice encouraging grant applications to include elements of rigorous study design in NIH-supported publications describing vertebrate animal and cephalopod research. The agency believes this will increase consistency of reporting for this type of research. HHS is also exploring whether machine learning or natural language processing approaches could be used to identify elements of rigor in NIH-funded publications. HHS will provide a status update in July 2023.
|
| National Institutes of Health | The Director of NIH should take steps to collect information to determine whether current policies and requirements are adequate to achieve transparency by ensuring research results and data are findable, accessible, and usable, and implement programmatic or policy changes, if needed. (Recommendation 2) |
HHS agreed with this recommendation. On January 25, 2023, a new NIH data-sharing policy went into effect. Under this policy, researchers--through submission of a Data Management and Sharing Plan--are required to prospectively plan for how scientific data will be preserved and shared. HHS notes that the plan encourages data management and data sharing practices be consistent with the Findable, Accessible, Interoperable, and Reusable (FAIR) data principles. NIH will monitor grant recipients' compliance with the data management plan at regular intervals. In addition, NIH intends to assess the process and outcomes associated with the new policy, which will inform potential future policy changes. NIH has issued several pieces of guidance on implementation of the new data sharing policy, and has created and developed initiatives to promote FAIR data sharing.
|
| National Science Foundation | The Director of NSF should collect information on relevant indicators of rigor to assess the research projects the agency funds, and implement steps, as needed, to promote strong research practices in future work. (Recommendation 3) |
NSF agreed with this recommendation. On January 17, 2023, NSF stated that it will support research and workshops aimed at developing an understanding of possible candidates for robust indicators of rigor in different types of science and engineering research. The information gathered from the workshops, and research on scientific rigor indicators, will inform the policies and procedures that NSF can use to enhance the rigor of the research projects the agency funds and promote the use of sound and improved research practices. This work will take place between September 2023 and September 2025.
|
| National Science Foundation | The Director of NSF should take steps to collect information to determine whether current policies and requirements are adequate to achieve transparency by ensuring research results and data are findable, accessible, and usable, and implement programmatic or policy changes, if needed. (Recommendation 4) |
NSF agreed with this recommendation. On January 17, 2023, NSF stated that it will conduct a study to assess the degree to which current policies and practices are ensuring that research results and data are findable, accessible, and reusable. NSF will explore strategies to improve public access to research outputs. The information gathered from these efforts will inform potential updates to policies and procedures which support transparency and public access to the products of research made possible through NSF awards. As part of this activity, NSF anticipates that the next update to its Public Access Plan will occur in FY 2023.
|
| National Aeronautics and Space Administration | The Administrator of NASA should collect information on relevant indicators of rigor to assess the research projects the agency funds, and implement steps, as needed, to promote strong research practices in future work. (Recommendation 5) |
NASA did not concur with GAO's July 2022 recommendation, stating that while NASA is committed to ensuring research reliability for projects it funds, NASA believes the best way to ensure reliability is through applying the peer review process to proposed studies submitted to NASA and to NASA-funded studies seeking publication. As stated in its July 2020 report, GAO continues to believe the recommendation is warranted.
|
| National Aeronautics and Space Administration | The Administrator of NASA should take steps to collect information to determine whether current policies and requirements are adequate to achieve transparency by ensuring research results and data are findable, accessible, and usable, and implement programmatic or policy changes, if needed. (Recommendation 6) |
NASA partially concurred with this recommendation. On January 17, 2023 NASA stated that it is already engaged in two efforts that will accomplish the objective of this recommendation. First, NASA's Science Mission Directorate has initiated its Year of Open Science, whose goals include enhancing access to NASA-funded publications by applying the Findable, Accessible, Interoperable, and Reusable (FAIR) principles. Second, NASA is revising its Scientific Integrity policy with the aim of increasing transparency and accessibility of NASA-funded research.
|