Validity Evidence for Procedure-Specific Competence Assessment Tools in Obstetrics and Gynecology: A Scoping Review.
TL;DR
While many tools demonstrated strong reliability and the ability to distinguish between levels of experience, evidence across other domains of validity was inconsistent, highlighting the feasibility of procedure-specific assessment in O&G and the substantial work that remains to support its use within CBME.
Abstract
Objective
Competency-based medical education emphasizes direct learner observation, frequent feedback and clear documentation. To support these goals for training residents in Obstetrics and Gynecology (O&G), there is a need for high quality, objective assessment tools to assess performance. Our study aims to comprehensively review the existing literature on procedure-specific competence assessment tools in O&G.
Design
Using the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) guideline, a systematic search was conducted of 6 databases to identify studies containing procedure-specific assessment tools in O&G. Two independent reviewers assessed these tools for validity using Messick's framework, which includes 5 sources of validity: content, response, process, internal structure, relationship to other variables and consequence validity. The educational utility was measured using the Medical Education Research Study Quality Instrument (MERSQI). Methodological quality was measured using the Accreditation Council for Graduate Medical Education tool for evaluating the quality of assessment methods (ACGME). The reviewers compared results and consensus discussion was used to arrive at a single score for each tool.
Results
About 5571 studies were identified, 37 studies underwent full text review, with 14 meeting inclusion criteria. A variety of surgical procedures were evaluated including cesarean section, vaginal hysterectomy, laparoscopic hysterectomy (total and with salpingo-oophorectomy), large loop excision of the transformation zone (LLETZ), Fetoscopic Laser Therapy, laparoscopic salpingectomy, cystoscopy, laparoscopic tubal banding, and myomectomy. Six of these studies assessed simulation-based tools and the remaining 8 studies evaluated live operative or video-based assessments. Most tools demonstrated strong internal consistency (Cronbach's α = 0.91-0.97), interrater reliability (ICC > 0.80), and construct validity through demonstrated ability to discriminate between experience levels. While many tools demonstrated strong reliability and the ability to distinguish between levels of experience, evidence across other domains of validity was inconsistent. These findings highlight both the feasibility of procedure-specific assessment in O&G and the substantial work that remains to support its use within CBME.
Conclusions
With a cultural shift in the design of surgical education toward more formal evaluation of resident competency, there follows a need for objective-procedure specific assessment tools. This review reveals the limited number of existing tools in O&G and demonstrates the need for more robust design and evaluation of assessment tools that emphasize validity, methodological quality and educational utility.