Monitoring possible effects of examiner leniency/stringency in OSCE with simple statistics
Publication Details
Tor, E., MacNish, J., Wright, A., & Steketee, C. (2012). Monitoring possible effects of examiner leniency/stringency in OSCE with simple statistics. The 15th Ottawa Conference on the Assessment of Competence in Medicine and the Healthcare Professions.
Abstract
When a conjunctive standard framework is applied for OSCE and students are required to pass every station or a fixed number of stations, the consequential validity of OSCE scores may be undermined by possible effect of examiner stringency / leniency. Summary of Work The possible leniency/stringency in scoring for each OSCE examiner is monitored post-hoc using simple statistics. The student score in a station (x) is compared to their respective average score across all other stations (y). The average score deviation (d=x-y) for all students is taken as a proxy for relative task difficulty for the station (D). Having taken into account the relative task difficulty, the absolute score deviation for each student in the station is determined (d-D). Finally, the average score deviation for all students examined by an individual examiner is compared to the standard error of measurement (SEM), to determine whether there is an effect of stringency or leniency. Summary of Results This statistical monitoring of possible effect of examiner leniency / stringency has helped in enhancing the consequential validity of OSCE test scores, and providing constructive feedback to OSCE examiners. Conclusion With the use of simple statistics which can be easily done using EXCEL, possible effect of examiner leniency/stringency can be monitored as part of quality assurance procedures for OSCE results.
Keywords
assessment, OSCE, psychometrics, medical education
Comments
Where possible the Link To Full Text button at the top of this page will link you to a full text version of this research output. Where, due to copyright or licence restrictions, it is not be possible to link to the full text version of this item the link will take you to the website of the copyright owner, who should be able to provide access to the output.
The University Library recommends the National Library of Australia TROVE search service to locate this research output within an Australian library.