Concerns Rise Over AI’s Impact on Scientific Research and Peer Review

OpenAI has ignited controversy within the mathematics community by reportedly using AI to solve over 700 longstanding mathematical problems in a brief period. A recent open letter signed by 28 Fields Medal recipients, one of the highest honors in mathematics, sheds light on these concerns.
Mathematicians reviewing some of the AI-generated solutions describe them as opaque in their reasoning. The open letter states, “Solving problems is only a tool and proxy for achieving the primary goal of conceptual understanding and insight.”
Prominent mathematician Terence Tao remarked, “These companies are dumping carcasses of raw meat onto our communal village table and saying: ‘Here you go, I solved your food problem.’”
OpenAI claims it consulted with the community before announcing its latest results, which include findings relevant to computer science.
Despite this, mathematicians must quickly adapt to the disruptive influence of AI in their field, a challenge that extends beyond mathematics to all scientific disciplines.
The impact of AI on software development is evident, with tools like Claude and Codex now routinely utilized by industry software teams. However, AI is also significantly affecting scientific practices. Academic journals and conferences are experiencing exponential increases in submissions from researchers, which strains a system largely reliant on volunteer efforts for reviewing and editing.
Breaking Peer Review?
This year, the International Conference on Learning Representations, a leading academic meeting on AI, received 60,000 submissions, a significant increase from 490 in 2017. One researcher commented, “AI is breaking traditional academic peer review.”
This trend is not isolated. In 2024, Mark A. Hanson from the University of Exeter and colleagues published a paper titled “The strain on scientific publishing,” noting the exponential growth in available academic articles in recent years.
This increase partly stems from a long-standing trend, predating modern generative AI, driven by pressures on researchers to publish extensively, often referred to as “publish or perish.” In 2022, just before the launch of ChatGPT, the number of published papers was approximately 47% higher than in 2016, despite little growth in the number of practicing scientists. AI is undoubtedly accelerating this process.
However, this raises concerns about the sustainability of this trajectory and the potential decline in scientific quality. One academic conference is exploring whether AI could help address these challenges. This year, the NeurIPS meeting introduced a voluntary AI-assisted review for conference abstract submissions.
Following the review process, scientists assessed the quality and usefulness of the AI-assisted evaluations. Yet, some experts believe this could exacerbate existing problems.
Protecting Critical Thinking
Anecdotal evidence suggests that some non-professional scientists are using AI to generate and submit academic papers and write science-related books. While citizen scientists contribute significantly in fields like biodiversity monitoring and astronomy, AI should not replace the formal training, ongoing research, critical thinking, and expert collaboration that academia provides.
AI and related technologies can democratize knowledge, but the author notes that it took them 13 years from their PhD to write their first book, while now it seems possible to complete such a task in weeks. However, human oversight is essential to verify and interpret AI-generated results.
As this situation evolves, new methods of scientific dissemination may be necessary to accommodate emerging research creation techniques.
The author emphasizes the importance of critical thinking in research. They increasingly rely on their AI assistant for various tasks, but the results can be unclear or incorrect, as highlighted by Terence Tao’s quote. Human expertise and time are crucial for sorting through these complexities.
The author questions the implications for researchers and the training of future scientists, mathematicians, and engineers. They express concern that long-term student projects, which typically take months, could be nearly fully automated through AI prompts. This raises questions about the learning outcomes for students and the authenticity of the author’s own research.
One of the author’s favorite science fiction stories, Isaac Asimov’s “Profession,” depicts a future where education occurs through direct brain uploads, akin to AI. The narrative critiques this approach, illustrating the risks of easily acquired knowledge without deeper understanding.
The author reflects on the arrival of a science fiction-like future where they can generate ideas while AI handles much of the work. While this is exhilarating, it raises concerns about the future of their discipline.
Related link: mathandai.org
Source: The Conversation Technology