The Sensitive Benchmark (SenBen), a large-scale scene graph dataset of 13,999 movie frames from 157 films, is introduced and a multi-task distillation recipe targeting vocabulary imbalance in autoregressive scene graph generation is proposed targeting vocabulary imbalance in autoregressive scene graph generation.
Abstract
Automated content moderation typically assigns binary safe/unsafe labels without revealing what sensitive behavior appears, who participates, or where in the image it occurs, limiting both auditability and cross-platform adaptation. Grounded explanations require grounded annotations, which existing moderation datasets do not provide at scale. We introduce the Sensitive Benchmark (SenBen), a large-scale scene graph dataset of 13,999 movie frames from 157 films. Each frame is annotated with Visual Genome-style scene graphs (25 object classes, 28 attributes covering affective states such as pain, fear, aggression, and distress, 14 predicates) together with 16 sensitivity tags spanning 5 categories. To evaluate predictions, we define SenBen-Score, a recall-focused composite metric over tags, objects, attributes, and predicates, macro-averaged across categories. All reported scores are measured against single-reviewer-corrected reference labels and are best read as relative comparisons across models on this shared ground truth. A coverage analysis against WordNet shows that 75% of SenBen’s sensitive vocabulary lacks adequate synonym coverage in general lexical resources, motivating a hand-curated domain synonym map released with the dataset. To train a lightweight student model, we propose a multi-task distillation recipe targeting vocabulary imbalance in autoregressive scene graph generation. The recipe combines suffix-based object identity, Vocabulary-Aware Recall (VAR) Loss, and a decoupled Query2Label tag head with asymmetric loss, gaining +5.2 percentage points (averaged over six seeds) in SenBen Recall over cross-entropy training on a 241M-parameter Florence-2-base student. Scaling to Florence-2-large (794M parameters) further raises performance to <inline-formula> <tex-math notation="LaTeX">$\mathrm {R}_{\mathrm {SB}}=0.441$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$\text {F1}_{\mathrm {SB}}=0.458$ </tex-math></inline-formula>, with precision gains exceeding recall gains as model capacity grows. On grounded scene graph metrics, the base student surpasses every evaluated VLM except Gemini and every commercial safety API, while recording the highest object detection and captioning scores across all models at <inline-formula> <tex-math notation="LaTeX">$7.6\times $ </tex-math></inline-formula> faster inference and <inline-formula> <tex-math notation="LaTeX">$16\times $ </tex-math></inline-formula> lower GPU memory, though relational (predicate) recall remains the principal open challenge.
The results demonstrate that demographic prompting is not a monolithic intervention: its utility is highly context-dependent, shaped by attribute signal quality, task characteristics, and model architecture.
M. Kamruzzaman, Shrabony Das, Gene Louis Kim· 0 citations
The widespread circulation of abusive online content has increased the need for reliable moderation of Chinese social-media text. Existing Chinese benchmarks support label classification, fine-grained toxicity categorization, and target-aware extraction, but do not provide a unified representation for deterministically verifying the stated basis of a moderation decision. We introduce VARM-Bench, a benchmark for field-anchored chain-of-thought rationales in Chinese abusive-speech moderation. Each instance contains a concise natural-language rationale with explicit anchors for six decisions: target, target type, target explicitness, author stance, harmfulness label, and fine-grained category. Our deterministic protocol evaluates field correctness, target alignment, output validity, complete-record agreement, and hidden record errors conditioned on correct final decisions, without relying on an LLM judge. Under a common structured-output protocol, we evaluate language models across multiple model families using zero-shot prompting, taxonomy guidance, and structured CoT supervision, and analyze lexical-cue sensitivity and field-level errors. Results show that strong label-level performance can conceal substantial errors in complete moderation records. VARM-Bench provides an auditable and reproducible benchmark for evaluating verifiable moderation rationales in Chinese abusive-speech moderation.
Mingyu Yuan, Shengtao Wen, Lingbing Guo et al.· 0 citations
Reliable confidence estimation remains a key limitation of test-time adaptation in vision-language models (VLMs), where prompt tuning improves zero-shot accuracy but often degrades calibration due to entropy-driven overconfidence. Prior approaches mitigate this using LLM-derived class attributes and contrastive regularization, yet treat attributes independently, ignoring their relational structure. We propose ARGTCA, which represents (class, attribute) pairs as nodes in a Symbolic Attribute Graph and trains a Graph Attention Network (GAT) using contrastive objectives to produce structurally informed embeddings that capture inter-attribute dependencies. We introduce two attribute selection strategies: ARGTCA-DIV for intra-class diversity and ARGTCA-DISC for inter-class discrimination. Experiments across nine benchmarks show that ARGTCA-DIV reduces average Expected Calibration Error (ECE) by approximately ~37% over baselines, while ARGTCA-DISC consistently performs as the second-best variant, reducing average ECE by approximately ~17% over baselines. These results suggest that modeling symbolic attribute interactions provides a principled approach for reliable test-time adaptation in VLMs.
Tanay Sodha, Aditya Sharma, Ramya Hebbalaguppe et al.· 0 citations
Financial disclosures may contain numerical, temporal, referential, factual, and policy inconsistencies that require different evidence and reasoning to diagnose. We study \emph{fine-grained inconsistency classification}: given a passage known to contain a conflict, the goal is to identify its type among 11 categories. Using a fixed snapshot of the synthetic SBID-FD benchmark, we compare frozen and fine-tuned encoders, evidence-augmented classifiers, prompted large language models, and LoRA-adapted generative models under a shared evaluation protocol. Task-specific adaptation yields large improvements over frozen representations, and a fine-tuned 300M encoder performs competitively with substantially larger prompted and adapted models. We further study whether localizing the conflicting claims improves classification through matched predicted-span, reference-span, and distractor-span conditions. The results show that automatically extracted evidence provides additional signal but recovers only part of the benefit obtained from reference spans. Per-class and confusion analyses further reveal that some inconsistency types are especially sensitive to localization quality, whereas others remain difficult even when the relevant evidence is supplied. These findings identify evidence localization and fine-grained type discrimination as distinct challenges and show that compact supervised encoders are strong baselines for this task.
Aman Kumar, Lasitha Vidyaratne, Dipanjan Ghosh et al.· 0 citations
Recent work has focused on improving explicit natural-language descriptive reasoning traces for generative recommendation. This includes systems that augment semantic ID (SID) prediction with chain-of-thought reasoning. However, because SIDs are opaque learned identifiers rather than natural language, they require costly alignment before an LLM can reason over them. This provides a controlled experimental setting in which both item representation (Title vs. SID) and semantic grounding (minimal vs. extensive SID alignment) can be varied independently. We therefore present the first controlled comparison of descriptive reasoning trace quality across semantic IDs and natural-language titles in a 2 x 2 factorial study on three Amazon product domains using a shared Qwen3-1.7B backbone. We find that introducing explicit descriptive reasoning traces reduces traditional offline recommendation effectiveness under standard SFT and RL training, even though natural language titles produce substantially more grounded and interpretable traces. Extensive SID alignment improves descriptive trace quality but not traditional offline recommendation effectiveness, while a richer reward signal partially recovers performance. Overall, our results show that improving descriptive reasoning trace quality is not, by itself, sufficient to consistently improve traditional offline recommendation effectiveness under the training objectives and evaluation protocols studied here.
Gustavo Penha, Juan Elenter, Claudia Hauff et al.· 0 citations
