Quantifying the Intensity of Online Social Support via Large Language Model–Based Evidence Extraction: Development and Validation Study

Abstract

Abstract Background Online social support, the interaction among individuals in which one helps another during difficult situations through online platforms such as online forums or social media, has proliferated as a vital tool for personal mental health care. Despite the growing usage and importance of online social support, prior studies have mainly focused on either understanding the characteristics of support seekers or merely identifying types of support, which leaves room for improvement in 2 key areas. First, the intensity of support, indicating the strength of willingness provided by supporters, has been studied little. Second, existing models rely heavily either on handcrafted features or on simple text embeddings of the entire message, which, respectively, fail to capture comprehensive textual features or to provide explainability in classification model decisions. Objective This study proposes a deep learning model that not only predicts the intensity of online social support but also provides human-readable evidence that explains the reasoning behind the decision made by the proposed model. While prior studies mainly concentrated on the task of identifying support types, this research quantifies the strength of support, which may better explain how strongly supporters seek to deliver informational and emotional support. Methods The proposed model categorizes informational and emotional support into 3 levels—strong, moderate, and weak. It collaborates with a large language model (LLM) to extract key sentences or phrases expressing supportive intent and to compute sentiment scores from both the original post and the reply. All extracted texts are then encoded as a vector representation whose elements indicate the presence of text segments corresponding to predefined features. All the computed features are finally fed into the neural network layers for intensity classification. The proposed model was evaluated on 2 independent, human-annotated datasets collected from previous studies, which consist of 1000 and 400 pairs of support-seeking posts and their associated supporting comments, respectively. Results The model achieved superior accuracy compared to baselines on both datasets (0.728/0.694 and 0.709/0.598 for informational support and emotional support, respectively). Through a case study of representative examples, we show that the extracted texts serve as useful evidence explaining the decisions made by the proposed model. For example, explicit intent to support (eg, providing a detailed solution or showing strong empathy) is successfully extracted and used to detect strong support, whereas only indirect support (eg, telling personal stories or offering general advice) is present and can be captured as weak support. Conclusions Through cooperation with LLMs, quantifying the intensity of online social support is highly achievable. In addition, the sentences extracted by LLMs can be used to explain the decisions made by the proposed model.