This study evaluates six AI medical translation systems using a mixed-methods approach, integrating BLEU scores, user surveys (N=775), and behavioral data. A standardized bilingual corpus was constructed from authoritative sources including the WHO and NMPA, while an AHP-BLEU hybrid model was developed to combine subjective user evaluations with objective scores across word, sentence, and paragraph levels in both Chinese-English and English-Chinese tasks. Results show Atman and Youdao outperform others in overall quality, with DeepL excelling in terminology. Spearman correlation analysis confirms a strong positive association (ρ=0.943, p=0.005) between BLEU scores and user satisfaction, validating the model. Despite rapid advances, current AI medical translation tools still struggle with term accuracy, context adaptation, and document complexity. The proposed AHP-BLEU framework helps align evaluation with user priorities, offering a more balanced view of performance. Future improvements should include semantic-aware metrics and human-verified baselines to better support multilingual medicine applications, from Traditional Chinese Medicine globalization to virtual consultations.
Xiangyu Wu, Yongqi Zeng, Qing Wang et al.· Advances in Engineering Tech...· 0 citations
Language models (LMs) often acquire various biases during pre-training and may express them in interactions, potentially causing social harm. Existing methods often rely on counterfactual augmentation or representation projection. These strategies remain limited in practice due to their high computational costs and difficulty in scaling to larger models. Additionally, many of these strategies require manual data annotation, narrowing their scope to specific cultures and bias categories. To overcome these limitations, we propose HEIMAT, a HEurIstic-style autoMATic debiasing framework for LMs. HEIMAT consists of two main steps: bias disclosure and debiasing fine-tuning. In the first step, it uses simple templates to construct heuristic prompts, which are applied to reveal model biases and generate corresponding context prompts. In the second step, it fine-tunes the model by minimizing the Jensen-Shannon divergence of predictions on these context prompts to reduce bias. Extensive experiments show that HEIMAT effectively mitigates bias in different cultures while maintaining the model's natural language understanding (NLU) performance.
Tian Lan, Yemin Wang, Chuancheng Shi et al.· 0 citations