Zero-Shot Detection of IndoT5-Synthesized Indonesian Scientific Abstracts Using mDeBERTa v3

Abstract

Human evaluators struggle to distinguish original scientific abstracts from AI-generated text, as AI-produced formal language appears neat and convincing; prior studies report reviewers correctly identify only 68% of AI-generated abstracts while misclassifying 14% of human texts. This study presents an exploratory, generator-specific evaluation of mDeBERTa v3 using zero-shot Natural Language Inference (NLI) classification, applied to Indonesian scientific abstracts synthesized via IndoT5-base-paraphrase rather than AI-generated text in general. A balanced 2,274-abstract dataset paired human abstracts (SINTA 3 journals) with IndoT5-base-paraphrase outputs as the AI class. Mann-Whitney U analysis on seven linguistic features revealed significant differences (p < 0.001) across all. A critical anomaly emerged: AI texts showed higher sentence-length variation (SD = 15.44) than human texts (SD = 7.99), contradicting the assumption that AI text is more uniform, attributable to context-window exhaustion in IndoT5 producing semantic hallucinations when synthesizing dense abstracts. Testing three NLI scenarios showed a single instruction targeting this fluctuation achieved the highest Recall (76.52%) but with 790 false positives among 1,137 human abstracts, limiting accuracy to 53.52%; added complexity further degraded AI-class recall due to vocabulary overlap. A Random Forest classifier trained on the same features achieved 91.21% accuracy (F1 = 0.9130), substantially outperforming the zero-shot approach and confirming the anomaly as a strong, learnable signal. These results indicate zero-shot NLI can partially track a generator's mechanical artifacts through a single targeted instruction, but remains insufficiently precise to separate machine-error fluctuation from natural human variation, and is not recommended for standalone academic-integrity screening without further refinement