Monitoring possible effects of examiner leniency/stringency in OSCE with simple statistics
Tor, E., MacNish, J., Wright, A., & Steketee, C. (2012). Monitoring possible effects of examiner leniency/stringency in OSCE with simple statistics. The 15th Ottawa Conference on the Assessment of Competence in Medicine and the Healthcare Professions.
When a conjunctive standard framework is applied for OSCE and students are required to pass every station or a fixed number of stations, the consequential validity of OSCE scores may be undermined by possible effect of examiner stringency / leniency. Summary of Work The possible leniency/stringency in scoring for each OSCE examiner is monitored post-hoc using simple statistics. The student score in a station (x) is compared to their respective average score across all other stations (y). The average score deviation (d=x-y) for all students is taken as a proxy for relative task difficulty for the station (D). Having taken into account the relative task difficulty, the absolute score deviation for each student in the station is determined (d-D). Finally, the average score deviation for all students examined by an individual examiner is compared to the standard error of measurement (SEM), to determine whether there is an effect of stringency or leniency. Summary of Results This statistical monitoring of possible effect of examiner leniency / stringency has helped in enhancing the consequential validity of OSCE test scores, and providing constructive feedback to OSCE examiners. Conclusion With the use of simple statistics which can be easily done using EXCEL, possible effect of examiner leniency/stringency can be monitored as part of quality assurance procedures for OSCE results.
assessment, OSCE, psychometrics, medical education