LLM Post-Training to Enhance Knowledge Extraction from Specialist Domains: Teaching LLMs User Forum Creole
TL;DR
Whether preserving the full recursive structure of user forum threads during post-training is a more effective first step toward knowledge extraction than flattened question-answer pairs is investigated andEncoder–decoder architectures with bidirectional cross-attention are identified as a promising next step for exploiting the full collaborative structure of forum discourse.
Abstract
Frontier large language models achieve broad linguistic competence but degrade on specialist domains underrepresented in pre-training corpora. Domain-adaptive post-training (DAPT) on curated professional text partially closes this gap, yet the dominant approach flattens structured discourse into isolated document units, discarding the collaborative reasoning embedded in multi-party exchanges. This paper investigates whether preserving the full recursive structure of user forum threads during post-training is a more effective first step toward knowledge extraction than flattened question-answer pairs. Four open-source decoder-only models (TinyLlama 1.1B, Phi-2 2.7B, LLaMA-2-7B 6.8B, LLaMA-2-13B 13B) are post-trained using parameter-efficient LoRA adaptation on 4970 threads from AgTalk, an agricultural producer forum, under three conditions: flattened Q → A pairs, full recursive threads preserving reply order, and shuffled recursive threads with randomly permuted intermediate replies. Five hypotheses are tested through paired Wilcoxon signed-rank comparisons across 29 thread positions. DAPT significantly reduces perplexity relative to pretrained baselines across all architectures (H0 supported). Recursive training reduces perplexity relative to flattened training, an advantage clearest for the two LLaMA-2 models under matched-context evaluation (Wilcoxon win rates near 72%) and present but obscured by outlier variance at the 1.1B and 2.7B scales (H1 supported). However, ordered recursive training provides only a marginal advantage over shuffled (H2 inconclusive), attention analysis reveals identical U-shaped endpoint-weighted profiles regardless of training condition (H3: architectural not learned), and perplexity shows no systematic decrease with accumulating thread depth (H4 not supported). These results are attributed to Rotary Position Embedding constraints in decoder-only architectures that systematically underweight middle-thread content. Encoder–decoder architectures with bidirectional cross-attention are identified as a promising next step for exploiting the full collaborative structure of forum discourse.