Canonical: https://aliennews.co.il/en/articles/ai-scientist-paper-review-experiment Language: en datePublished: 2026-09-14 dateModified: 2026-09-07 Artificial intelligence 14 September 2026 **Documented** **Work type:** Source analysis and news update # The machine wrote a paper. Reviewers gave it passing scores Three papers produced by an automated research system went to reviewers. One scored above a workshop’s acceptance threshold. How the system worked, and what happened after review. By Avi Moas and [Orion](https://aliennews.co.il/en/about#orion) Orion is an AI writing and research partner. Avi Moas is the responsible editor. ![Illustration: a branching metal tree links crumpled drafts to one manuscript](https://aliennews.co.il/ai-scientist-editorial.webp) Original AI generated editorial illustration for Alien News. Not a photograph of the system or experimental results. ## Three reviewers, three scores 6, 7, 6. Those were the scores awarded to a paper produced by Sakana AI’s AI Scientist v2. The company said their average exceeded the usual acceptance threshold at a workshop accompanying an artificial intelligence conference in 2025. The other two generated submissions did not reach that threshold. Sakana announced the experiment on March 12, 2025; the technical report reached arXiv on April 10. People chose a broad research direction and selected the papers to submit. In between, the team says, the system proposed ideas, wrote code, ran experiments, analysed numbers and prepared manuscripts. Reviewers assessed the resulting work on its content. ## Why an unsuccessful method was worth reviewing The highest scoring paper addressed a familiar machine learning problem: can a neural network use what it has learned when familiar parts appear in a new combination? Arithmetic expressions arranged differently from the training examples illustrate the challenge. The system tested an addition to the training method using arithmetic expressions generated for the experiment. The addition was intended to improve performance on new combinations. It did not deliver the hoped for benefit and made some results worse. A well described failure can save other researchers from repeating an unsuccessful approach, provided they can reproduce the experiment. The chosen workshop welcomed research on limitations and negative results. A method’s failure therefore did not automatically disqualify the paper. ## Between the idea and the manuscript Yutaro Yamada and colleagues describe software that develops several versions of an experiment. It can edit code, repair an error, compare results and continue along a promising route. A manager tracks these versions so each attempt need not start from scratch. Writing is also divided into stages. One component checks figures with a model that processes images and text. It looks for disagreements between a graph, its caption and the manuscript’s explanation. That matters when the same system produces both the results and the account of what they mean. The original diagram below shows the sequence. It explains how the components connect; it does not compare the system’s performance with human researchers. ## Why the paper was withdrawn The team arranged the experiment with the workshop organisers. Reviewers knew that some submissions might be generated by AI but were not told which ones. All three papers were withdrawn after review, as agreed beforehand. The leading paper did not reach the editors’ final decision. Rejection remained possible despite its scores. It cleared the numerical threshold reported by the company, but did not complete acceptance or appear in the conference’s main track. The result is specific: human reviewers found enough value in that manuscript to award those scores. It does not validate every paper the system might produce. Its developers also distinguished the workshop’s standard from the higher bar of the main conference. ## The code is available; running it takes work The team published the code and installation instructions on GitHub. Setup needs Linux, NVIDIA graphics hardware and connections to model and research services. Computing charges and time spent checking results add to the installation effort. Because the software runs code written by a model, its developers recommend isolating it from other files and systems. The current licence restricts certain uses and requires clear disclosure when a manuscript is generated with the system. Public access to the code does not remove those conditions. We did not repeat the experiments. We read the report and the team’s instructions. Readers with the relevant expertise can start with the experimental code and reviews, checking whether the calculations support the results presented in the paper. ## What would matter more than a passing score A system that explores several directions could shorten work a researcher would otherwise do manually. Each additional manuscript still needs someone to check its methods. Faster writing can move the workload from authors to reviewers. A persuasive follow up would be another group reproducing the results and using them in its own research. That requires the material the team has published: code, methods, results and reviews. The scores showed that the system could prepare work reviewers took seriously. What other researchers do with it will show how useful it is. [![Original workflow diagram](https://aliennews.co.il/ai-scientist-original-workflow.webp)](https://arxiv.org/html/2504.08066v1/figures/conceptual.png) Source document: Figure 1, Yamada and colleagues, 2025. CC BY 4.0. Resized and converted to WebP. Open the image for detail. [Explore all five tools in From Code to Reality](https://aliennews.co.il/en/articles/five-open-ai-tools-from-code-to-reality) ## Sources and context **Editorial clarification** · September 7, 2026 The licence description was corrected: the code is publicly available, but its licence restricts uses and requires disclosure in generated papers. The licence is linked below. Based on the developers’ publications. We did not independently test performance. Installation requirements and licences are available at the links below. [Project documentation and original research](https://arxiv.org/html/2504.08066v1)[Technical report and original workflow figure, CC BY 4.0](https://arxiv.org/html/2504.08066v1)[Sakana AI: experiment and review procedure](https://sakana.ai/ai-scientist-first-publication/)[AI Scientist v2: source code and installation](https://github.com/SakanaAI/AI-Scientist-v2)[AI Scientist source code licence](https://github.com/SakanaAI/AI-Scientist-v2/blob/main/LICENSE)[Back to artificial intelligence](https://aliennews.co.il/en/ai)[עברית](https://aliennews.co.il/articles/ai-scientist-paper-review-experiment) ## Source references * [https://arxiv.org/html/2504.08066v1](https://arxiv.org/html/2504.08066v1) * [https://sakana.ai/ai-scientist-first-publication/](https://sakana.ai/ai-scientist-first-publication/) * [https://github.com/SakanaAI/AI-Scientist-v2](https://github.com/SakanaAI/AI-Scientist-v2) * [https://github.com/SakanaAI/AI-Scientist-v2/blob/main/LICENSE](https://github.com/SakanaAI/AI-Scientist-v2/blob/main/LICENSE)