Let Unfunded Grant Applications See the Light of Day

By Stuart Buck, Christopher Steven Marcum

In 2022, as part of an ongoing assignment from Congress, the National Academies of Sciences, Engineering, and Medicine set out to evaluate if a set of National Institutes of Health (NIH) grants to fuel science start-ups worked as intended. The aim was to determine whether Small Business Innovation Research and Small Business Technology Transfer grants spur productive collaborations, technology transfer, and economic benefits—but NIH refused to share how applications to both grant programs were ranked (and thus funded) by the expert review panels tasked with evaluating them, hindering the Academies’ efforts. “Although the committee requested priority score information from NIH, this information was not provided because of confidentiality concerns,” the report read. “If future analyses are to be more robust and enable stronger statements on program impact, NIH will need to find a way to provide this information to researchers, as it and other agencies have done in the past.”

Without knowing what proposals go unfunded, there is no way to know whether agencies are supporting a wide range of ideas or favoring a narrow theory.

Something very similar happened with the National Science Foundation (NSF). In 2023, the Academies called out NSF for not meeting its legal obligation to share data: “Granting access to data on all applicants for program assessment purposes, as is called for in the legislation mandating this review, and establishing processes that would allow for structured evaluation of policies and procedures would help NSF understand the effectiveness of its initiatives and how its programs could be improved.”

Despite expectations from policymakers and statutes that data from science agencies be available for analysis, both of us—longtime open science advocates who have worked in various government and industry positions and who are writing only in our personal capacities and not on behalf of anyone else—have heard top researchers complain that they can’t get access to information on unfunded proposals, or, on the rare occasions when they do, access is conditional on allowing the agency to veto any publications using the data.

Without knowing what proposals go unfunded, there is no way to know whether agencies are supporting a wide range of ideas or favoring a narrow theory. Are “high-risk, high reward” proposals getting a chance? Have hard-won changes in grant policies actually helped early-career researchers? Do the questions researchers ask change in response to demands from Congress or calls from citizen groups? These questions seem both valuable and straightforward. Yet metaresearchers (those who research how research is done) are unable to address such topics with any certainty. The public cannot know, for example, how many NIH grant applications come from historically Black colleges or universities, or how many researchers propose to study gain-of-function in viral genomes, without knowing what is included in all R&D grant applications, funded and not.  

Put another way, understanding federal R&D investments requires knowing about both the numerator (awarded applications) and the denominator (awarded plus unawarded applications). But federal agencies—in the United States and elsewhere—almost never show what they choose not to fund. (In contrast, core information about funded proposals are generally and laudably available on databases such as NIH RePORTER and NSF’s Award Search.) This lapse runs counter to all evidence-based policymaking and, in our view, threatens scientific competitiveness. In contrast, shining light on grant applications offers huge potential benefits against comparatively little risk.

Discernment demands denominators

At a very basic level, we’re looking for more disclosure than is currently given. Yes, making unawarded proposals available raises myriad operational questions about what level of information would be made accessible, who should have access, and what provisions would need to be made for dual-use and other sensitive research. Proposal information contains many components, including abstracts, full text, and reviewers’ comments and scores. Right now, simply having abstracts and reviewer scores available by default to American researchers in appropriate fields would be a fantastic advance. Perhaps the most metascience mileage could come if funding agencies made core data available to any qualified researcher without reserving the right to veto publications. Considerations about appropriate and additional availability could be tabled until disclosure systems are worked out and benefits become clear.

Perhaps the most metascience mileage could come if funding agencies made core data available to any qualified researcher without reserving the right to veto publications.

We do know that the relatively few researchers who were supplied even scant information about unfunded proposals have produced significant insights. They’ve done work on how ethnicity affects likelihood to receive funding, whether higher grant scores predict higher publication rates, how race and ethnicity tracks funding rates, whether certain scoring metrics have outsize influence, and whether grant recipients are more likely to attract venture capital. These are all important questions, and just a few of many that could be asked and answered with greater access and transparency.

A broader set of analyses could reveal how the research system fails. Take, for example, the controversy around the amyloid hypothesis in Alzheimer’s research, which links the disease with a protein called amyloid beta. Several papers, including an extremely influential 2006 Nature study that seemed to prove an aspect of the hypothesis, came under scrutiny after a neuroscientist found evidence of image tampering in 2021. The papers in question had encouraged what many people in Alzheimer’s research consider to be a kind of groupthink around the amyloid hypothesis that deflected attention from other potential strategies for understanding and treating the disease. And while the evidence of misconduct doesn’t necessarily disprove the amyloid hypothesis, Science reported in 2022 that about half of all NIH Alzheimer’s funding that year went toward work that mentioned amyloids. One argument is that this emphasis led to overwhelming confirmation bias that disfavored contrasting results or hypotheses, but if researchers could retroactively examine funded and unfunded proposals, they might get a more precise understanding of whether and how the amyloid hypothesis crowded out other promising research options—enabling the system to learn from its mistakes. In fact, NIH could potentially exonerate itself from perpetuating such a bias if it provided public access to the full portfolio of applications. Furthermore, if funders realized their decisions were public, they might be more likely to hedge their bets across different research avenues. This could promote a more rigorous approach to hypothesis-setting within research proposals.

We recognize that many researchers will be uncomfortable making part or all of their proposals publicly available. In our view, however, any researcher ready to ask for public tax dollars to be directed to their laboratory should be willing to let at least some materials (abstract, peer review scores, etc.) be available so that other researchers can evaluate how the funding agency is doing. 

The potential benefits of disclosure

In 2022, scientific integrity researchers Serge P. J. M. Horbach, Joeri Tijdink, and Lex Bouter argued that disclosing information about unfunded grant applications was the next step for open science and transparency, articulating a raft of potential benefits. We believe these benefits would accrue to funding agencies, researchers, and taxpayers.

Access to a complete picture of the research landscape across the federal government would allow funding agencies to make more informed decisions about funding priorities.

Researchers could make more efficient progress by using past grant proposals to refine their approach and so avoid wasting months or even years on unrealistic proposals, particularly if reviewers’ comments and scores are also shared. They could learn whether they are pursuing projects already deemed unpromising by funding agencies, which could prompt them to try other areas, or to have a pre-application conversation with a program officer to gain a better understanding of an agency’s interest. Researchers with similar interests would be able to discover each other’s work and potentially join forces, leading to stronger proposals, more impactful research, and collaborations formed much earlier than those enabled by publications and conference presentations.

Though agencies can look across their own applications for insights, access to a complete picture of the research landscape across the federal government would allow funding agencies to make more informed decisions about funding priorities. As articulated in the Foundations for Evidence-Based Policymaking Act signed into law by President Trump in 2019, when agencies can share and access information across the government, they are able to more efficiently identify research trends, understand the derivative impacts of their own work, and craft decisions and policies informed by evidence. They can also build more effective, cross-agency initiatives, such as NSF’s Smart Health funding opportunity, designed to support cross-agency efforts to incorporate information science in health care. 

And as metascientists are able to show that these practices give taxpayers improved outcomes, these data could build public trust.

Dread of disclosure

Agencies and researchers fear the denominator for many reasons. First, and perhaps most familiar, is that public disclosure could allow others to scoop researchers’ proprietary and innovative ideas. This is a justifiable fear—scooping happens, sometimes intentionally and sometimes by accident. However, we think public disclosure actually protects intellectual property. In the current system, anonymous peer reviewers have privileged access to others’ ideas. If proposals were publicly available, there would be a time-stamped version of who had a particular idea first—enabling those authors to protect both credit and intellectual property. (An aside: Should some particularly rapacious lab seek to scoop another’s idea, they might not want to bother with ideas that had already been rejected by a panel of peer reviewers.)

We think our idea can build on several relatively new practices to promote transparency and eliminate bias in science. One of these is preregistration, where researchers can opt to submit plans for experiments and analyses to a public registry or journal before beginning work, so that the experimental question and design can be assessed independently of results. And there is no evidence that preregistration raises risks of scooping. The Center for Open Science’s Open Science Framework, one of the largest preregistration platforms, reports that they have never heard of even one example of work being stolen because it was on their platform. Also, the rise of placing preprints (research articles that have not yet undergone traditional publication in a journal) into publicly accessible repositories has demonstrated that such timestamps are protective against scooping while enabling rapid review, sharing, and credit, as researchers now list prepublished work on their CVs. Perhaps assigning funding applications a digital object identifier (DOI), as Horbach, Tijdink, and Bouter suggest, might have similar benefits.

Disclosing unfunded proposals and their reviews could render agencies more open to criticism of their decisionmaking processes, funding priorities, perceived biases, or errors in evaluation.

Fears of being scooped and losing credit could cause some researchers to write their proposals differently, perhaps withholding data and potentially limiting insights. But to the extent that a researcher has an idea worth funding, it would be counterproductive to hide that idea entirely in their application.

Another potential objection is that researchers from other countries might scoop ideas from the United States. However, the goal of science funding is not to burnish the curricula vitae of American scientists, but to increase human knowledge for the public good; if other nations’ researchers are able to make useful discoveries, that will benefit all of humanity. The perceived threat could also increase pressure for American scientists and funders to think more critically about their research priorities—an idea worth scooping by a foreign actor is probably worth funding in the first place.

For agencies and funders, fear of scrutiny may be an obstacle to transparency. Disclosing unfunded proposals and their reviews could render agencies more open to criticism of their decisionmaking processes, funding priorities, perceived biases, or errors in evaluation. For example, a 2022 paper found consistently lower grant scores and funding rates for non-white applicants at the National Science Foundation. But we contend that such findings are critical for policymakers to identify problems and make improvements. 

And then there are the forces of inertia alongside agencies’ fear of blowback from researchers. Change is hard and scary, even for science agencies charged to be at the bleeding edge of innovation. Science policy scholar Frank N. Laird writes that “sticky policies” are the primary force preventing reform at federal science agencies. Though his argument is specifically about stubbornly low funding rates and wasted grant-writing efforts, the idea applies much more broadly. Getting agencies to shed light on the denominator will require considerable effort and infrastructure, including resources allocated to redact sensitive information, support infrastructure for sharing proposals, and ensure compliance. Nonetheless, we argue that institutional inertia should not get in the way of transparency and good governance. In fact, we would argue that federal agencies should bear the burden of justifying why they are not disclosing information. Agencies typically argue that they have statutory exemptions to the Freedom of Information Act, or FOIA. And yet our closer examination suggests the case for these exemptions is thin to nonexistent, with public interest nearly always outweighing the risks of disclosure. (See text box.)

Encouraging disclosures

The same agencies that insist their applications be kept confidential have, ironically, begun to emphasize the need to share experimental data, regardless of outcome. Last year, NIH issued a request for information to encourage publication of null studies (studies that lack statistically significant results for a stated hypothesis), arguing that formal dissemination is “vital for scientific progress and accurate assessment of cumulative evidence.” That follows a 2020 data sharing policy requiring deposit of all NIH-funded data in a repository, regardless of whether data support the hypotheses stated in the grant proposal or ended up in a scientific publication. It also is consistent with the 2023 NIH Data Management and Sharing Policy.

The same agencies that insist their applications be kept confidential have, ironically, begun to emphasize the need to share experimental data, regardless of outcome.

NSF has, for its part, funded workshops and issued Dear Colleague letters discussing the importance of sharing null results. They have also funded studies on metaresearch, the same field clamoring for access to unfunded proposal data in order to understand and improve the grant-awarding process.

Federal agencies and Congress both greatly value the transparency achieved by current practice for all outputs of research they fund. But NIH and NSF could do more to enable metaresearchers to dig into agencies’ own null results. The lack of sunlight here has caught the attention of government watchdogs. A 2023 report from the Government Accountability Office (GAO) critiqued NIH for failing to make enough data available to evaluate NIH’s contribution to drug development and recommended releasing application data on funded and unfunded research, including scores and thresholds for funding.

Show the denominator, see progress

Despite widespread hesitancy, several efforts have worked to share data from unfunded proposals. In 2017, statistician Jeffrey T. Leek applied to a Howard Hughes Medical Institute (HHMI) call to support undergraduate teaching, then publicly posted his rejected proposal “so at least the work I put into it doesn’t just disappear entirely.” Before that, in 2012, biologist and data scientist Ethan White began compiling his own and others’ unfunded grant proposals in his field, resulting in the searchable database Open Grants. To date, only 290 proposals made to over a dozen global agencies have been voluntarily submitted to the project, 65 of which were listed as being unfunded. (For scale, NIH receives over 50,000 research project grant applications a year.) Though laudable, these grassroots efforts suffer from a lack of visibility, which in turn discourages participation.

A few funders have experimented with making unfunded proposals available. In 2021, with researchers’ consent, the Open Science Fund within the Dutch Research Council (NWO) published unfunded applications as a way for other funders to look for applications that might be a good match for their programs. Submitters for 67 of 167 programs agreed to have at least part of their proposals made available but NWO seems to have abandoned the practice in subsequent years. Similarly, the Wellcome Trust specifically established the (now closed) Open Research Fund and the Learned Society Curation Awards to make submissions accessible with the consent of the applicants. The former program did so for 137 of 172 eligible submissions. The latter, a small joint program with HHMI, provided summaries of all seven proposals submitted along with information on the decision process (three were funded). To our knowledge, funders have not explained publicly why the programs weren’t renewed or why researchers did or did not participate. (Perhaps launching such innovations around the time of the pandemic sapped momentum.)

In any case, broader participation would be required to capture broader benefits. When programs rely on applicants to actively opt in, efforts suffer from selection bias, a form of systematic error that preempts generalizability. For example, we’d guess that researchers submitting slapped-together proposals or proposing the same idea again and again are less likely to want their proposals to be public. It would be fascinating (but currently impossible) to know how such submissions worsen low acceptance rates.

By embracing openness and releasing information on both successful and unsuccessful research proposals, federal agencies can foster a more efficient, collaborative, and innovative scientific ecosystem.

There is incremental progress. In response to the GAO report, the Department of Health and Human Services described a pilot program “to provide researchers with access to agency’s internal administrative data” and added that it would consider expanding the program, with an update expected in October 2025. In December 2024, NIH announced a pilot program for science of science scholars, promising to provide access to internal agency data “if appropriate” for their studies.

We believe such pilot studies would build an evidence base that would encourage funding agencies to work together to not just allow but actually support study of unfunded applications. If full accessibility is too high a bar, there are tiered modes of secure access that could be tried. The Census Bureau, for example, uses a designation called “special sworn status” that requires background checks and training to access confidential data; something like this as well as disclosure avoidance training and data use licensing agreements could allow researchers access to data on unawarded applications as well as review scores of all applications via virtual secure access enclaves or research data centers.

We anticipate reform—through mandates or guidance—is coming to federal R&D funding agencies, especially given that US funding agencies are, as we argue, already required to provide a great deal of data about unfunded applications. For instance, both Section 10502 of the CHIPS and Science Act of 2022 and Section 303 of the Foundations for Evidence-Based Policymaking Act of 2018 require data to be shared about the full range of applications submitted. US agencies should be accountable to the law. Taking initial steps toward more transparency on their own accord could help make coming reforms smoother and more effective. Nonetheless, nontransparency has been in place for the history of grant-making, and so no one is certain how to go about reversing it. Doing anything must push against the entrenched default of doing nothing. By embracing openness and releasing information on both successful and unsuccessful research proposals, federal agencies can foster a more efficient, collaborative, and innovative scientific ecosystem. That will ultimately strengthen the United States’ position as a global leader in science and technology.

Transparency Is the Law

The Freedom of Information Act (FOIA) lists nine reasons to exempt information from disclosure and requires agencies to balance these with the public interest, pointing to guidance from the Department of Justice in making decisions. Here we consider the chief ones. Though these cases did not necessarily result in project proposals or reviewers’ evaluations being released, we argue that there is a much stronger legal case for disclosure than currently appreciated.

Exemption 4: Trade secrets or commercial or financial information that is confidential or privileged.

This is likely the key exemption agencies point to to explain why they don’t share applications, even on an anonymized basis. After all, it could be unfair if disclosure allowed a third party to scoop a researcher’s unfunded (presumably unpublished) idea or technique.

But not even NIH seems to think that every proposal counts as a trade secret with proprietary information. The NIH Grants Policy Statement notes that “applicants are instructed to identify proprietary information at the time of submission of an application…. If an applicant fails to identify proprietary information at the time of submission as instructed in the application guide, a significant substantive justification will be required to withhold the information if requested under FOIA.” And, to the extent a grant application contains routine but sensitive information, such as a researcher’s salary, that could be automatically redacted.

Caselaw, though limited, suggests grant proposals are not protected by Exemption 4. In Washington Research Project, Inc. v. Dept. of Health, Education and Welfare et al., 504 F.2d 238 (D.C. Cir. 1974), the Washington Research Project sued to get access to “eleven specifically identified research projects that had been approved and funded by the National Institute of Mental Health.” This information included the grant application, a site visit report from the agency, and a summary report on the application.

The court discredited the claim that research designs were subject to exemption, arguing that “it is clear enough that a non-commercial scientist’s research design is not literally a trade secret or item of commercial information, for it defies common sense to pretend that the scientist is engaged in trade or commerce.” What’s more, the court found that “all types of applications,” including progress reports, were subject to disclosure. Indeed, it seemed to ridicule arguments for nondisclosure: “The government has been at some pains to argue that biomedical researchers are really a mean-spirited lot who pursue self-interest as ruthlessly as the Barbary pirates did in their own chosen field.”

Exemption 5: Privileged communications within or between agencies, including those protected by: deliberative process privilege (provided the records were created less than 25 years before the date on which they were requested); attorney work-product privilege; and attorney-client privilege.

No one has claimed that grant applications are privileged communications, but there is caselaw that privileges advice from peer reviewers as deliberative process. See Formaldehyde Institute v. Department of Health & Human Services, 889 F.2d 1118, 1121 (D.C. Cir. 1989); Judicial Watch, Inc. v. US Department of Commerce, No. 15-cv-2088 (D.D.C. Aug. 21, 2017); Washington Research Project, Inc. v. Department of Health, Education and Welfare et al., 504 F.2d 238 (D.C. Cir. 1974). Nonetheless, we would argue that reviewer ratings and comments are hardly confidential, as they are used to determine how to distribute government funds. Even if not strictly required to release this material, agencies could readily anonymize or aggregate it for release to independent scholars.

Exemption 6: Information that, if disclosed, would invade another individual’s personal privacy.

Agencies could argue that unawarded grant applications are exempt from FOIA because they contain personally identifiable information, such as the authors’ names, institutions, email addresses, etc., but this very same information is released with awarded applications via www.usaspending.gov and on agency websites. 

In fact, courts have found that information must be released. In Kurzon v. Department of Health and Human Services, 649 F.2d 65 (1st Cir. 1981), plaintiff George M. Kurzon “wanted to test his theory that the peer review method by which the National Institutes of Health (NIH) evaluate grant applications is biased against unorthodox proposals.” When the agency refused to release the data, he filed a lawsuit asking the National Cancer Institute for names and addresses of unsuccessful applicants. The court noted that most Exemption 6 cases contained highly personal details, but that, because Kurzon was seeking “slight informational content,” the loss of privacy would be “minimal.” Further, there would be no risk of embarrassment because the vast majority of applications are rejected.

Perhaps most importantly, the court emphasized that there was “an obvious public element” attached to “efforts to secure government funds, especially in a field so much in the public eye as cancer research,” and that NIH itself recognized this by releasing information about funded grant applications.

Finally, the court found that there was no promise of anonymity, that “the best the government can do is to assert a general implied promise of confidentiality based on its policy statement, published in the Federal Register, that ‘[i]nitial research or [a] research training grant application on which award is not made’ is ‘generally not available’ to the public.” Kurzon, 649 F.2d at 69-70 (quoting 45 C.F.R. Part 5, App. (1980)). We would argue further that even a promise of confidentiality would not justify an exemption to FOIA. Indeed, it would allow agencies to override requirements with unilateral promises. Such a practice would run counter to stated values of transparency in science, impede open government, and raise agencies’ risk of litigation.