Human VS AI: comparison of scientific paper drafting capabilities
TL;DR
This study took an article written by the authors of the current study and ran it through the models to help build prompts for the article to be written by the AI, and compared the results of the human-written article and the AI-generated ones.
Abstract
Large language models (LLMs), a type of artificial intelligence (AI), are increasingly popular tools used for everyday and work-related activities and tasks. Their application in medicine is widely researched and has been used recently to help write scientific papers in various fields. LLMs can draft sections of manuscripts or whole papers far more quickly than human writers. However, they need appropriate prompting to draft near-complete and worthwhile papers. In the current study, we used two different LLM models: ChatGPT-4o and Claude 3.5 Sonnet, to test AI’s scientific writing capabilities. We took an article written by the authors of the current study on the topic of fluorescent cholangiograms and ran it through the models to help build prompts for the article to be written by the AI. After that, we loaded the data and references used for the article into both AI tools and prompted them to write sections of the article (or a whole article if possible) on the same topic while allowing for independent choice of statistical analysis. The results of the human-written article and the AI-generated ones were compared, evaluating the information used from the references, the types of statistical analysis methods used, the conclusions drawn, and the time it took to complete the task.