What peer review can actually do
In machine learning, and more broadly in computer science, we already know that peer review is noisy. Every conference season brings the same mixture of frustration and resignation: inconsistent scores, obvious misses, strong papers rejected, weak papers accepted, and long discussions about how strange the outcomes were. For example, the NeurIPS 2021 consistency experiment measured this point: 10% of submissions were evaluated by two independent committees. The committees disagreed on accept versus reject for 23% of the duplicated papers, and among papers accepted by the first committee, 50.6% were rejected by the second. The experiment has limitations, and many disagreements concerned papers near the decision boundary, but that is precisely the point: at the boundary of a selective conference, acceptance is a noisy outcome. The earlier NeurIPS experiment in 2014 reached a very similar conclusion.
This raises a question we don’t often ask: what is a conference decision actually for?
We currently ask one rushed signal to answer several different questions. Is the work technically sound? Should the field pay attention to it? Can its claims be trusted? Is its author a strong researcher? A few reviewers reading under deadline can provide useful evidence about each of these questions, but they cannot answer all of them with the confidence we often assign to acceptance.
At its best, conference review asks whether a paper’s claims are adequately supported, its reasoning is clear, and some part of the field is likely to learn from the result. For example, TMLR expresses the first part well: its primary criterion is whether the claims are supported by accurate, convincing, and clear evidence, while interest to the journal’s audience is treated separately. That is already a substantial task. Acceptance should increase our confidence in a paper, but only by a bounded amount. It should mean that the work deserves serious attention, discussion, criticism, use, or extension. It does not necessarily mean that its important claims have already survived independent verification.
The stronger interpretation persists for an understandable reason. Conference decisions do not only allocate scientific attention; they also allocate scarce opportunities. They influence hiring, grants, invitations, letters, visibility, and what early-career researchers are told counts as success. Ideally, anyone evaluating a researcher would read and understand that person’s contributions as a whole and in context. In practice, few people have enough time. Acceptance counts, citation totals, and venue names become cheap substitutes for the harder judgment. One noisy decision has therefore come to stand in for several slower and more expensive ones.
Some researchers would like conference acceptance to mean much more: serious vetting, high confidence, something close to “probably correct.” I understand that instinct. Evidently, scientific venues should not become indifferent to correctness. But a nontrivially large conference operating on a compressed timeline cannot independently verify every important claim, rerun every experiment, inspect every dataset, and explore every undocumented choice. For instance, NeurIPS 2025 received 21,575 submissions, and major machine learning conferences often move from submission to decision in only a few months. At that scale and speed, review is necessarily a best-effort first pass, not an exact certification process. That does not make it worthless or excuse weak reviewing. It means that we should read the decision more carefully and improve what happens after acceptance.
From peer review to test-of-time
Even excellent peer review cannot settle the question we eventually care about most: does the claim remain supported once other people inspect, use, and extend the work? Some weaknesses are visible on a careful first reading, and others only appear when a paper leaves the review system and meets sustained attention. Here are a few concrete examples: in reinforcement learning, Henderson et al. (2017) showed how nondeterminism, high variance, sensitivity to implementation choices, and inconsistent reporting could make apparent improvements difficult to interpret; in natural-language inference, Gururangan et al. (2018) found that models could often predict labels using the hypothesis alone, revealing annotation artifacts that weakened the broader interpretation of benchmark performance.
That is why “do better peer review” is only part of the answer. Better reviews are valuable, but they still happen under scarcity: limited time, limited attention, limited compute, and limited willingness in general. Test-of-time awards recognize a related point from much farther away. They are not verification exercises, and influence is not the same as correctness, but they acknowledge that important properties of a paper can only be judged after years of follow-up work and actual use.
What we lack is an ordinary, visible version of that later scrutiny for the average accepted paper: something earlier and lighter than a test-of-time award, but more systematic than hoping volunteers eventually check important work on their own.
Proposal: an audit lottery
My suggestion is a post-acceptance audit lottery: after decisions are made, a conference would select a sample of accepted papers for a bounded public check that adds evidence to the record, without questioning acceptance or implying misconduct. The practical question is how to make such a system work at conference scale: which papers should be selected, what should be checked, who should perform the audits, what resources should be provided, and how should the findings be reported?
Most papers should be selected randomly, so later scrutiny is broad and predictable rather than reserved for controversial or famous work. Random selection would also change incentives across the full submission population. The sample should be stratified because papers with modest public artifacts, expensive training runs, private data, human-subject studies, or external API dependencies pose different audit problems. A separate, smaller stream could examine unusually influential or consequential claims.
Each audit would need a clearly declared scope. An artifact check might ask whether the archived code, data, models, and instructions are functional. A result reproduction might ask whether an independent team can recover a principal result using the authors’ artifacts and declared setup. A robustness or replication study might vary implementations, seeds, datasets, models, metrics, or other consequential choices. A claim-evidence audit might instead ask whether the title, abstract, and headline conclusions are appropriately bounded by what the experiments establish. These questions should not be collapsed into one badge: code can run while the scientific claim remains weak, and a result can reproduce under the authors’ setup while failing under reasonable variations. Compute tiers could limit what an audit attempts, but they should not define what it means. The outcome should describe what was checked and found - for example, that the main result was reproduced under the declared setup, a material discrepancy was identified, the claim exceeded the available evidence, or the audit was inconclusive within its budget - rather than assigning a global label such as “audited-confirmed.” The report should include an author response, and acceptance would normally remain unchanged: the purpose is to update the evidence attached to a paper, not to rerun the original decision or informally adjudicate misconduct.
The main point is not to punish weak papers but to change what papers are optimized for. A system dominated by one compressed decision naturally rewards work that is easy to evaluate favorably at that moment: broad claims, polished narratives, favorable setups, and complexity that is difficult to inspect under deadline. A credible possibility of later scrutiny would instead reward narrower claims, cleaner artifacts, clearer documentation, and fewer hidden degrees of freedom. For this to work, audits must be treated as scientific labor, with citable reports, visible credit, conflict screening, and compute support where needed. The system should also avoid creating another crude prestige signal: being selected should not be treated as an honor, not being selected should remain neutral, and every finding should stay attached to the specific claim and protocol examined.
Moving forward
We would not be starting from scratch. A few examples come to mind: artifact badging is already established in parts of computer science; TMLR gives reproducibility studies a formal publication path and certification; MLRC became an official NeurIPS track in 2026; and OpenReview can keep reviews, responses, and revisions linked to the original submission. What is still missing is a routine, venue-initiated audit of a representative sample of a conference’s own accepted papers. An audit lottery would not replace voluntary reproductions or deeper follow-up studies, but would add a modest layer of ordinary scrutiny across the accepted population.
I have empirical machine learning papers mainly in mind. Theory, datasets, and systems would require different checks: formal verification may help with some proofs, dataset audits may focus on provenance, consent, contamination, or licensing, and systems work may require independent evaluation under realistic conditions. AI tools may lower the cost of some scrutiny by helping researchers navigate code, appendices, and claims, while also making it easier to produce polished submissions whose evidence remains difficult to assess. Both effects strengthen the case for separating initial selection from later scrutiny.
I do not think the answer is to abandon (or constantly criticize) peer review or ask reviewers to verify entire research programs under deadline. Review should do the job it can realistically do: assess whether a paper’s claims appear supported, its reasoning is clear, and its contribution merits serious attention. The field should then have a normal way to update the record when later scrutiny produces new evidence. That is: (1) conference review should decide what enters the conversation; (2) the scientific record should make visible what remains supported after the conversation begins.
Acknowledgements
Thanks to many stimulating discussions with several colleagues and friends, including Gautam Kamath, Sanmi Koyejo, Mahdi Haghifam, John Duchi, Andreas Haupt, Rylan Schaeffer, Lydia Zakynthinou and many others.