Development and Pilot Testing of a Novel Tool for Evaluating Practical Skills in Hematopathology Residents in Pakistan

Background: The mini-Clinical Evaluation Exercise (mini-CEX) and direct observation of procedural skills (DOPS) are reliable tools for work-based assessment of medical trainees. Tools of this type do not yet exist for evaluation of practical laboratory skills of pathology residents. Objective: We developed and piloted a 9-item instrument for direct observation of laboratory skills (DOLS). Methods: We used the DOLS tool with 10 hematopathology residents (PGY-1 to PGY-5) from Aga Khan University. Each resident was evaluated by 3 faculty members in the laboratory during 4 separate encounters using the DOLS instrument. We assessed construct validity, interrater reliability and G coefficient, feasibility of using DOLS, and learner satisfaction. Results: A total of 120 encounters were observed with a mean score (±1 SD) of 56.7% (±12.44). Assessment scores moderately correlated with the number of laboratory procedures previously performed by participants (r = 0.658 and 0.641; P = .0001) and with PGY level. Interrater reliability ranged between 0.47 and 0.96. Cohen's d was 1.64. Residents accounted for a large component of estimated variance (73%), suggesting DOLS can differentiate residents' laboratory skills; variance associated with assessors was small (0.01%). Residents reported being satisfied with the tool. Mean time (±1 SD) taken for observing and feedback was 17.89 ± 5.89 minutes. Conclusions: The new DOLS instrument could provide reliable scores for observing laboratory skills. Residents were satisfied with the tool, and rating times make the tool feasible for formative assessments.