Skip to content
Log in
Sponsored Have an interesting project or opportunity? → Hire me ↗
Sponsored Design Multi-Agent AI Systems Using MCP and A2A: Engineer your own Pyt Design Multi-Agent AI Systems Using MCP and A2A: Engineer your own Pyt
Preprint

MGAL: A Multilingual Granularity-Aware Long-Context Benchmark

Chunhan Li Chenglin Xu Zongyang Zhang Jiale Liu Zhuoxi Rao Xudong Jia Junxiu He Menglin Yang Wenjuan Gong Zhengzhe Liu Cheng-Wei Qin
Aug 2026 · 0 citations · 38 references
Computer Science

TL;DR

MGAL is the first multilingual, granularity- and position-aware long-context benchmark, constructed from United Nations reports spanning 8K to 128K tokens across the six official UN languages, and finds that LLMs perform well at word-level tasks but struggle with coarser-grained ones.

Abstract

Evaluation of long-context Large Language Models (LLMs) has advanced rapidly. However, most existing benchmarks are limited to the document level and focus mainly on high-resource languages, leaving many fine-grained challenges insufficiently evaluated. To address this gap, we present MGAL, the first multilingual, granularity- and position-aware long-context benchmark. MGAL is constructed from United Nations (UN) reports spanning 8K to 128K tokens across the six official UN languages. It covers four coherent levels of linguistic granularity (word, sentence, paragraph, and document) and further stratifies entries by their position within the document (begin, middle, and end), indexed at both the document and paragraph levels. This design enables systematic diagnosis of multilingual long-context comprehension across different granularities. Through extensive experiments and analyses, we find that: (1) LLMs perform well at word-level tasks but struggle with coarser-grained ones; and (2) Closed-source models retain a clear performance advantage in lower-resource languages. We further identify two new challenges: (1) Under local semantic crowding, where neighboring sentences share topics and entities, models tend to follow surface cues (e.g., connectives like ``however''or repeated entities) rather than the discourse role of the sentence in surrounding context (e.g., background, outcome); and (2) A gap between fluency and consistency in generated outputs, where models produce text that reads smoothly but drifts from the source facts. In addition, we observe several patterns in line with prior studies, including reliance on nearby evidence and reuse of options under uncertainty.

arXiv Semantic Scholar
View source

Similar papers

Preprint Jul 2026

Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders

MTEB-PT is presented, a Portuguese benchmark constructed from a subset of MMTEB, comprising 14 existing datasets across Semantic Textual Similarity (STS), classification, retrieval, and reranking, and shows that language-specific fine-tuning still improves model performance in Portuguese, especially on task types that match the adaptation data most closely.

Lucas H.T. Okamura, Alexandre Alcoforado, A. H. R. Costa · 1 citation
Preprint Jul 2026

Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench

PredicateLongBench is proposed, a benchmark that stress-tests long-context reasoning by asking models to identify the longest contiguous subsequence of words in a long input that satisfies given predicates/constraints drawn from a broader predicate class.

Siddhartha Jain, A. Velingker · 0 citations
Preprint Aug 2026

VoxSumm: A Multilingual Corpus of Long-Form Spoken News for Joint Summarization and Translation

This work formalizes joint speech summarization and translation (JSumT), the generation of a succinct, faithful target-language summary directly from a long spoken document in a source language, and establishes a foundation for developing and evaluating multilingual systems capable of jointly interpreting, compressing, and translating long-form speech.

Yejin Jeon, Marie Maltais, Virginia Ceccatelli et al. · 0 citations
Book Open access Jul 2026

Speak Beyond English: Multilingual Prompts Improve Query Classification in Small Language Models

This paper tests whether prompting the same (frozen) SLM in three typologically diverse languages and aggregating the outputs can improve classification without retraining or translation, and suggests that cross-lingual diversity rather than surface-level input variation drives the gain.

Pratyay Banerjee, Panthadeep Bhattacharjee, Angshuman Jana · 0 citations
Book Open access Jul 2026

From Single- to Cross-Document: Benchmarking Multi-Granularity Event Analysis of Large Language Models

Event analysis is an essential and fundamental direction of information extraction, involving various event-centric tasks at different granularity of documents. While large language models (LLMs) have preliminarily achieved promising performance in part of these tasks individually, their capability in event analysis still lacks comprehensive understanding due to restricted document granularity, task designs, and data source of existing benchmarks. To address these limitations, we introduce MiGUE-Bench, a systematic benchmark for assessing the performance of LLMs in multi-granularity event analysis. To support large-scale evaluation, we first develop an LLM-driven self-correcting annotation framework called MiGUE-Pipeline, enabling scalable acquisition of high-quality source data of events with automatic labels. Then, we design four core tasks in our benchmark, i.e., event detection, relation reasoning, structure induction, and future prediction, to probe model competence at different levels, from atomic event details to complex cross-document narratives. Extensive experiments on state-of-the-art LLMs and retrieval-augmented generation (RAG) methods delineate the current capability boundary and identify critical deficiencies, providing insights into the the future improvement of LLMs in challenging event analysis tasks.

Tao Wen, Shuai Shao, Pei Ke et al. · 0 citations