RESEARCH QUESTION
Move feedback from post-decision to pre-submission
Generate actionable criticism while authors still have time to revise.
THE PAPER IN ONE PATH
Can AI broaden peer-review feedback before submission?
Human feedback is expensive and often arrives months after submission, when authors have little time to react. We built a paper-only review engine that searches broadly for possible concerns. Its goal is not to reproduce one finite reviewer panel exactly: it should anticipate much of that panel's feedback while also surfacing grounded issues the panel may not have mentioned.
Select any stage to open its evidence and paper-ready paragraph.
10,000ICLR submissions
37,993human reviews
84.9%historical weighted coverage · 512
Top 32initial blinded manual audit
CENTRAL RESEARCH CLAIM
A broad AI review pool overlaps strongly with—not simply contains—the human panel.
At 512 candidates per paper, the generated pool covers 78.7% of the historical atomic issues and 84.9% of their seriousness weight on the ten-paper diagnostic. The remaining human-only region measures feedback the generator missed. The AI-only region is different: it mixes useful novel concerns with possible false positives and therefore requires independent validity assessment rather than automatic rejection.
AI-only hypotheses requiring validation
Human-only missed historical feedback
01 Dataset construction Recover what reviewers could have seen 3,398 timestamp-qualified pre-review manuscripts; grouped 2,446 / 272 / 680 split.
10,000ICLR 2026 submissions
37,993reviews
2,123version-pinned arXiv inputs
1,548pre-review OpenReview PDFs
- The two historical sources overlap for 273 papers, yielding a strict union of 3,398.
- All 3,398 parse successfully: 2,114 source-first LaTeX projects and 1,284 PDFs.
- Paper families stay together in train (2,446), validation (272), and unopened test (680).
Paper-ready paragraph
We construct a leakage-controlled benchmark from 10,000 ICLR 2026 submissions and 37,993 reviews. To approximate the manuscript available before reviewer feedback, we recover 2,123 version-pinned arXiv submissions and 1,548 timestamp-qualified OpenReview PDFs, producing 3,398 unique pre-review papers. Paper families are assigned to a frozen grouped split of 2,446 training, 272 validation, and 680 held-out test papers.
02 Experiments 05–10 · evaluation protocol Turn prose reviews into atomic, weighted targets Candidate × human-issue matrices replace whole-review lexical overlap.
Human reviews→atomic issues→Sol High match probability→one-to-one coverage
- The central ten-paper diagnostic contains 178 corrected atomic human issues.
- A concern covers an issue when the frozen GPT-5.6 Sol High same-issue probability is at least 0.70.
- We report both strict recall and seriousness-weighted recall; complete matrices remain inspectable.
Paper-ready paragraph
We evaluate review generation at the atomic-issue level rather than with BLEU or ROUGE. Human reviews are decomposed into deduplicated concerns, assigned seriousness scores, and compared with every generated candidate using a frozen same-issue judge. This yields a complete candidate-by-human matrix and supports strict, seriousness-weighted, major-issue, and one-to-one coverage metrics.
03 Experiments 10–17 · pool-size curve Coverage keeps rising, but the tail gets expensive 512 candidates cover 140/178 issues and 84.9% of seriousness weight.
Per paperIssues coveredStrictWeighted
12894 / 17852.8%58.1%
256111 / 17862.4%69.2%
384131 / 17873.6%80.7%
512140 / 17878.7%84.9%
The final 128 candidates recover only nine additional issues and 4.2 weighted-coverage points.
Paper-ready paragraph
Increasing the retained pool from 128 to 512 candidates raises strict human-issue coverage from 52.8% to 78.7% and seriousness-weighted coverage from 58.1% to 84.9%. The curve does not saturate completely, but marginal yield declines sharply: the last 128 candidates recover nine of 178 issues. Broad candidate generation is therefore effective, while its late tail is increasingly inefficient.
04 Experiments 11, 14–19 · generation strategy Nominal count is not semantic breadth At 512 candidates: independent 49.5% weighted coverage; dedup + refill 84.9%.
Only 656 of 5,120 outputs (12.8%) are post-hoc semantically unique.
16,143 raw concerns are generated to retain 5,120 unique candidates.
Breadth costs 5.2× more tokens (96.61M vs 18.66M) and 3.6× more requests (1,905 vs 525) across ten papers. Historical wall time is incomplete, so no wall-time speedup is claimed.
Paper-ready paragraph
Independent calls repeatedly rediscover obvious criticisms: only 12.8% of 5,120 nominal outputs are semantically unique, and weighted coverage reaches 49.5%. Rejecting near-duplicates and refilling each quota raises weighted coverage to 84.9%, but requires 3.6 times as many requests and 5.2 times as many tokens. Generation strategy therefore matters more than nominal pool size, at a substantial compute cost.
05 Experiments 15, 18, 19 · semantic saturation Easy concerns arrive first; novelty becomes harder Adaptive stopping is cheap, but leaves 33 full-pool issues behind.
74–173adaptive candidates / paper
66.0%weighted coverage
15.0 minobserved concurrent wall time
- The adaptive novelty rule saves calls and tokens but misses 33 issues found by the 512 pool.
- Fine-grained persona quotas reach a 74.5% duplicate-rejection rate; rigid evidence slots return
no_issue56.7% of the time. - The evidence supports diminishing yield, not a fitted exponential law.
Paper-ready paragraph
The first concerns are easy to elicit, whereas later semantic novelty becomes progressively rarer. Early stopping reduces cost but sacrifices coverage, and increasingly rigid persona or section quotas either regenerate duplicates or return no issue. These observations demonstrate sharply diminishing marginal yield; we deliberately do not claim an exponential scaling law because no parametric law was fitted.
06 Experiments 10, 23 · heterogeneous generation Different models explore different review regions A practical 256 pool mixes 128 DeepSeek, 64 Inkling, 32 Kimi, and 32 Qwen candidates.
DeepSeek 128
Inkling 64
Kimi 32
Qwen 32
- On 93 papers, size-matched 32-candidate Sol and DeepSeek pools reach 38.7% and 42.3% macro strict recall; their union reaches 53.5%.
- There are 265 Sol-only and 347 DeepSeek-only covered targets.
- On five validation papers, the provisional 256-candidate mixture reaches 76.7% weighted coverage.
Paper-ready paragraph
Model heterogeneity supplies a second route to semantic breadth. Size-matched Sol and DeepSeek pools cover many non-overlapping targets, and their union substantially improves recall. We therefore form a practical 256-candidate pool from DeepSeek V4 Flash, Inkling, Kimi-K3, and Qwen3.8-2.4T. This result establishes complementarity, although differing prompts and serving budgets prevent a causal claim that model mixture alone explains the gain.
07 Experiments 12, 13, 23–25 · top-k ranking Thirty-two are enough; automatic selection remains difficult The oracle establishes capacity, while a preliminary blinded manual audit tests practical usefulness.
79.3%hidden oracle weighted @32
43.5%best anchored Qwen weighted @32
≈36 ptsselection gap
The candidate pool contains a compact covering set. Current automatic judges cannot reliably identify it, motivating the blinded manual Top-32 audit.
Paper-ready paragraph
A retrospective maximum-coverage oracle shows that 32 candidates can preserve essentially all human-issue coverage available in the 256-item pool. Yet listwise, cross-batch, pointwise, pairwise, and ensemble LLM rankers recover only about 40–44% weighted coverage on average. A preliminary ground-truth-blind manual audit suggests that a useful compact set can be selected while retaining concerns not recorded by the historical panel. We treat this as qualitative appendix evidence and reserve quantitative validity claims for a larger expert study.
08 Error source 1 · Experiments 26–33 Semantic clustering is highly useful, with room for improvement It organizes the candidate pool effectively, while manual audits reveal remaining merge and split errors.
- Agglomerative
- Ward
- bisecting spherical k-means
- spectral
- DBSCAN
- HDBSCAN
- OPTICS
- BIRCH
- Gaussian mixture
- DeepSeek clustering
- tag-aware clustering
Across 160 confirmation clusters, 70 have no historical-review edge and 41 connect to at least two human issues. Manual inspection also finds distinct concerns merged into one cluster and equivalent concerns split across clusters.
These counts do not make every unmatched cluster an error: some may contain useful AI-only feedback. They show that an unlabeled geometric partition cannot perfectly reproduce the latent structure of a small historical reviewer panel.
Paper-ready paragraph
Semantic clustering is a highly useful way to organize a large candidate pool and preserve reviewer-relevant concerns. We compare geometric, density-based, probabilistic, divisive, LLM-generated, and tag-aware clustering, then inspect clusters through both human-review links and paper-level examples. The high hidden ceilings show that most useful concerns survive clustering. Remaining errors—occasionally merging different technical angles or splitting paraphrases—identify clear opportunities for paper-conditioned and context-aware improvement.
09 Error source 2 · Experiments 34–38 The right cluster often contains the issue; we choose the wrong representative Hidden within-cluster selection is high, but paper-only centrality and LLM scores are weak.
SelectionWeighted coverageInterpretation
Qwen top one / cluster38.7%deployable baseline
Kimi tournament45.2%best confirmation selector
Qwen top four + hidden choice70.9%shortlist ceiling
Ward hidden representative74.1%cluster ceiling
- In 49.2% of issue-bearing clusters, only one member matches the assigned human issue.
- Qwen score–hidden utility Spearman correlation is only 0.158 within clusters.
- Representative choice causes 36.1 weighted points of loss versus 6.1 points from the one-per-cluster constraint.
Paper-ready paragraph
The second source of error is representative selection. If hidden human-review links identify the best member of each cluster, weighted coverage reaches 74.1%, and retaining four candidates per cluster preserves a 70.9% ceiling. In contrast, medoids, centroid proximity, density, graph centrality, learned rankers, and LLM tournaments remain much lower on average because the reviewer-relevant member is often not the semantic center. A retrospective single-paper case nevertheless reaches 83.3% weighted Top-32 coverage with spherical bisecting K-means and paper-only pointwise selection, demonstrating substantial headroom for improved representative models.
10 Error source 3 · Experiments 23, 35, 37–40 The candidate–human matching score is itself noisy Judge disagreement, paraphrase sensitivity, and lossy atomization can all hide real semantic overlap.
660Sol-positive matrix edges
8DeepSeek-positive edges
0.015cross-judge kappa
1.14human issues / matched candidate
At the same 0.70 threshold, the Experiment-23 DeepSeek matcher marked only 8 of 45,568 candidate–issue pairs positive while Sol High marked 660. In addition, atomic extraction can remove context needed to recognize equivalence, and two concerns may describe the same underlying weakness from different technical angles. Measured coverage therefore contains evaluator false negatives as well as genuine generation and selection misses.
Across the ten-paper 256 pool, the Sol matrix contains 3.71 matching candidates per human issue and 1.14 human issues per matching candidate. Separate importance-compression experiments retained 78.1% of parent weighted coverage at 64 candidates and only 59.7% at 32, so matcher noise does not erase the observed selection bottleneck.
The main table uses regex extraction for non-reasoning models but an LLM evaluator for reasoning models, creating a possible evaluation-protocol confound.
Paper-grounded; no human issue matched (max probability 0.045).A guarantee stated only “for sufficiently large T” may not support the motivating few-step regime without a finite-T threshold.
Paper-grounded; no human issue crossed 0.70 (max probability 0.58).- Paraphrased criticisms can receive different scores when they emphasize different mechanisms, remedies, or evidence.
- Atomic issue summaries improve evaluation tractability but can discard entities, qualifiers, or surrounding reasoning required for a match.
- Manual Top-32 inspection finds both strong overlap with human feedback and plausible AI-only concerns, suggesting that some measured “errors” are reference or matcher limitations.
The manual audit is promising but small. It supports the hypothesis that AI can both anticipate and extend human feedback; it does not yet quantify the validity rate of AI-only concerns.
Paper-ready paragraph
The third source of error is evaluation noise. LLM matchers disagree sharply, semantically equivalent concerns may use different wording or emphasize different causal angles, and atomizing a full review can remove context required for equivalence. Consequently, measured human-review coverage is a noisy lower-bound agreement signal rather than a complete quality metric. Our preliminary blinded manual audit finds that selected AI concerns recover much of the human feedback while also surfacing plausible issues absent from the recorded panel. Scaling this audit with independent experts is necessary to separate genuinely novel feedback from matcher false negatives and invalid concerns.
PAPER THESIS
More than mimicking reviewers
An LLM review tool can surface much of the feedback a human panel would raise—and additional grounded hypotheses—early enough for authors to act. It complements rather than replaces peer review: breadth is attainable, while trustworthy compression and large-scale independent validity evaluation remain open problems.
THE NARRATIVE IN THREE SENTENCES
- Generate broadly. Deduplicate-and-refill and heterogeneous models turn nominal outputs into semantic coverage.
- Select cautiously. Thirty-two concerns are sufficient under an oracle, but present paper-only rankers discard too much useful breadth.
- Evaluate on two axes. Historical overlap measures prediction; blinded independent validity determines whether AI-only criticism adds useful feedback.