SP 9.02 Robustness of Retrospective Surgical Complication Grading for Time-Series Data Capture: An Inter-Observer Agreement Analysis Using Clavien-Dindo Classification
TL;DR
It is demonstrated that a supervised dual junior rater approach yields substantial-to-excellent inter-observer agreement in non-structural clinical data coding, however, agreement is likely over-estimated when only the highest grade is recorded over an extended interval (e.g. 30 days).
Abstract
Non-structural clinical data, such as post-operative complications, are susceptible to inter-observer disagreement, undermining data validity. This study assesses the validity of multi-timepoint Clavien-Dindo complication grading in a single-procedure cohort of Whipple resections. Complication grading within 7, 14, 30 and 90 days for 130 Whipple resections were independently coded by two final-year medical students and validated by a senior clinician. Inter-observer agreement was assessed using Cohen’s Kappa (κ). Disagreements were analysed by category (within minor, between major and minor, and within major grades). The senior clinician reviewed disagreements and identified potential systematic grading errors. Across 1040 coding episodes (130 patients, two coders, four time intervals) agreement results showed: κ (days 1-7) = 0.77(95% CI, 0.66-0.88); κ (days 8-14) = 0.91(0.85-0.97); κ (days 15-30) = 0.88(0.81-0.95); κ (days 31-90) = 0.73(0.59-0.88). Recording the highest complication grade within 30 days reduced disagreements from 32 to 15, κ (days 1-30) = 0.84(0.50-1). Most disagreements occurred within minor grades (I–II). Disagreement between minor and major grades (≤II vs ≥IIIa) was only observed in days 1-7. This study demonstrates that a supervised dual junior rater approach yields substantial-to-excellent inter-observer agreement in non-structural clinical data coding. However, agreement is likely over-estimated when only the highest grade is recorded over an extended interval (e.g. 30 days). For shorter time-series intervals, this approach is more prone to inconsistencies, highlighting the need for diagnostic criteria-based, automated data collection system to ensure data robustness for research and care quality improvement.