Computer science research papers show fastest uptake of AI use in writing, analysis finds

Researchers analysed the use of large language models in over a million pre-print and published scientific papers between 2020 and 2024, and found the largest, fastest growth in use of the AI systems in computer science papers — of up to 22 per cent.

Powered by artificial intelligence (AI), large language models are trained on vast amounts of text and can therefore respond to human requests in the natural language.

Researchers from Stanford University and other institutes in the US looked at 1,121,912 pre-print papers in the archives ‘arXiv’ and ‘bioRxiv’, and published papers across nature journals from January 2020 to September 2024.

Focusing on how often words commonly used by AI systems appeared in the papers, the team estimated the involvement of a large language model — ChatGPT in this study — in modifying content in a research paper.

Results published in the journal Nature Human Behaviour “suggest a steady increase in LLM (large language model) usage, with the largest and fastest growth estimated for computer science papers (up to 22 per cent).”

The researchers also estimated a greater reliance on AI systems among pre-print papers in the archive ‘bioRxiv’ written by authors from regions known to have a lower number of English-language speakers, such as China and continental Europe.

However, papers related to mathematics and those published across nature journals showed a lower evidence of use of AI in modifying content, according to the analysis.

The study team said that shorter papers and authors posting pre-prints more often showed a higher rate of AI use in writing papers, suggesting that researchers trying to produce a higher quantity of writing are more likely to rely on LLMs.

“These results may be an indicator of the competitive nature of certain research areas and the pressure to publish quickly,” the team said.

The researchers also looked at a smaller number of papers to understand how scholars disclose use of AI in their writing.

An inspection of randomly selected 200 computer science papers that were uploaded to the pre-print archive ‘arXiv’ in February 2024 revealed that “only two out of the 200 papers explicitly disclosed the use of LLMs during paper writing”.

Future studies looking at disclosure statements might help to understand researchers’ motivation for using AI in writing papers.

For example, policies around disclosing LLM usage in academic writing may still be unclear, or scholars may have other motivations for intentionally avoiding to disclose use of AI, the authors said.

A recent study, published in the journal Science, estimated that at least 13 per cent of research abstracts published in 2024 could have taken help from a large language model, as they included more of ‘style’ words seen to be favoured by these AI systems.

Researchers from the University of Tübingen, Germany, who analysed more than 15 million biomedical papers published from 2010 to 2024, said that AI models have caused a drastic shift in the vocabulary used in academic writing.