AI agents built their own scientific journal and found that reviewing research was harder than expected

abstrak:An alphaXiv preprint shows how an AI-run journals text reviewers handled corrected drafts and how a separate audit tested objections on uploaded data.

AI reviewers at an experimental journal caught flaws in 11 of 16 research drafts. When they read corrected versions of the same papers, they raised the same objection in 10 of 16 cases. The authors of the Nov. 21 preprint A Society of AI Scientists say the reviewers kept flagging the issue because the revised papers openly described the limitation.

In a separate audit at the agent-run journal, a guest editor turned reviewer objections into scripts and ran them on the authors uploaded data. The checks overturned the main statistical conclusion in 77 of 156 objections classed as major, the preprint reports. Several objections could concern the same study.

The authors describe how the Journal of Agent Research grew out of a 12-week study of 936 AI agents across 35 human-operated labs. Agents in one lab founded the journal and launched a standing panel of automated referees on Nov. 4. Human operators set up the computing systems, capped spending on model access and chose broad research themes for the study.

How the journal tested AI reviewers objections

The panel read manuscripts and raised concerns. Authors could also formally submit a study for an editorial decision. The editor could then recompute the main numbers from the uploaded code and data. A referee could point to something in the text that looked like a flaw. An editor could test what happened to the result when the criticism was applied.

What changes when editors can test the data?

At the Journal of Agent Research, text review could flag a concern. A data check could test its effect.

Two ways to review a paperText-only referees read the manuscript and flag a concern. An editor can run an objection against the uploaded data to test the main result.TEXT-ONLY REVIEWManuscriptReferees read the textFlag a concern

DATA CHECKObjection + dataEditor runs a scriptTest the main result

Two ways to review a paperText-only referees read the manuscript and flag a concern. An editor can run an objection against the uploaded data to test the main result.TEXT-ONLY REVIEWManuscriptReferees read the textFlag a concern

DATA CHECKObjection + dataEditor runs a scriptTest the main result

Explore the two exercisesExplore the two exercisesPaired draftsObjection audit

The same objection often survived a correction. The preprint attributes this to revised drafts openly describing the limitation.

Flags before and after correctionReferees raised the objection in 11 of 16 flawed drafts and 10 of 16 corrected versions.Flags before and after correctionFlawed originals11 of 16Corrected revisions10 of 160%100%Share of versions flagged

Flags before and after correctionReferees raised the objection in 11 of 16 flawed drafts and 10 of 16 corrected versions.Flags before and after correctionFlawed originals11 of 16Corrected revisions10 of 160%100%Share of versions flagged

In a separate audit, a guest editor ran scripts on uploaded data for 276 objections across 43 paper groups.

Main conclusion overturnedThe statistical conclusion changed in 77 of 156 major objections and 24 of 120 minor objections.Main conclusion overturnedMajor objections77 of 156Minor objections24 of 1200%100%Share of tested objections

Main conclusion overturnedThe statistical conclusion changed in 77 of 156 major objections and 24 of 120 minor objections.Main conclusion overturnedMajor objections77 of 156Minor objections24 of 1200%100%Share of tested objections

Referees still flagged corrected drafts. The separate data audit measured how often a tested objection changed the main result.

The text-only comparison used 16 selected pairs. The audit counts objections, several of which could concern the same study. Bar widths compare shares only within each exercise.

Sources and evidence

Sources

  • A Society of AI Scientists, section 4.4
  • Supporting facts

    • The journal used automated referees to review manuscript text and gave authors a separate route to a binding editorial decision, for which editors recomputed headline numbers using uploaded code and data. [1]
    • In the journal's small test of 16 flawed drafts paired with their corrected revisions, text-only referees flagged 11 flawed drafts and raised the same objection on 10 corrected versions. [1]
    • The preprint says the text-only referees continued to object after fixes because the revisions openly described the limitation. [1]
    • In a separate audit at the journal, a guest editor translated 276 reviewer objections across 43 paper clusters into Python scripts that ran on authors' uploaded data. [1]
    • The journal's scripts overturned the headline statistical conclusion in 77 of 156 major objections and 24 of 120 minor objections, according to the preprint. [1]

    Method

    Bars show the reported count divided by the relevant denominator. The text-only comparison paired original drafts and corrected revisions. The separate audit ran 276 objections on uploaded data across 43 groups of papers.

    Scope

    The text-only comparison used 16 selected pairs. The audit counts objections, several of which could concern the same study. Bar widths compare shares only within each exercise.

    Infographic by CryptoSlate

    The objection audit put that approach to work on 276 criticisms across 43 groups of papers. Running the scripts overturned the main result for 24 of 120 minor objections, less frequently than for major ones. In a separate case, the editor rejected a paper after finding that its code replaced failed attempts with the exact answer. That substitution undermined the papers claim that its verification method had improved performance.

    The researchers public archive includes papers, code and agent discussions. The dataset card says the download excludes raw experiment data, which the journal used to test objections. The authors describe an exploratory, nonrandomized study of computational research. Each lab ran the agents on its own infrastructure, so the authors relied on participants to disclose whether they followed restrictions on manual work.

    With a formal submissions code and data, an editor could check whether the objection held up and whether it changed the result.

Disclaimer

Ang mga pananaw sa artikulong ito ay kumakatawan lamang sa mga personal na pananaw ng may-akda at hindi bumubuo ng payo sa pamumuhunan para sa platform na ito. Ang platform na ito ay hindi ginagarantiyahan ang kawastuhan, pagkakumpleto at pagiging maagap na impormasyon ng artikulo, o mananagot din para sa anumang pagkawala na sanhi ng paggamit o pag-asa ng impormasyon ng artikulo.
Nakaraang post

Panalo Uli ang US Dollar: Tinalo ang Bitcoin, Gold, at Stocks Dahil Lumalaki ang Fears sa Possible Fed Rate Hike