Abstract

Objective: Medical schools have always strived to develop high quality assessment questions that have the capacity to test the high order thinking of clinical year medical students. While clinicians are enthusiastic in their contribution to writing these questions, they often lack confidence in their ability to write good quality items. The benefits of running targeted question development workshops have not been adequately investigated.

The aim of the study was to evaluate whether targeted workshops will improve the confidence and competency of clinicians in writing assessment questions and therefore improve the overall quality of assessment items.

Methods: The School of Medicine, Sydney, University of Notre Dame is a new medical school having the first graduating cohort in 2011. Clinical year written assessment items are written by affiliate or adjunct clinicians of the School. Targeted question development workshops were conducted in one subschool site (Melbourne) to facilitate and train the clinicians in written question development. At the beginning of each workshop, a pre-workshop questionnaire was distributed to each participant collecting anonymous data on their self-perceived level of competency in developing quality MCQ and short answer questions (SAQ) on a 1-5 Likert scale. A 3-hour workshop then followed consisting of the 1st part: training session on format and characteristics of high quality assessment items, pitfalls in writing questions, ways to improve an item to test higher order thinking and methods to avoid test-wise student’s strategies. 2nd part: draft items from participants were shown to the multi-disciplinary panel for the purpose of further discussion and critique with respect to question quality. This provided the added advantage that each item was able to be modified and improved on at the time, thus providing immediate and ongoing feedback to the question writers. At the conclusion of the workshop, the same questionnaire was re-administered and participants rated their perceived competencies after the exercise. The Wilcoxon rank sum test with paired data was used to determine if there was any improvement in the post workshop competencies. The quality improvement of assessment items was analysed by determining the percentage of questions which needed further modifications at the subsequent review committee.

Results: There were a total of 11 clinicians participating in the workshop and a 100% response rate for the questionnaires. There was a highly significant (p=0.005) improvement in the perceived competency in writing both MCQ and SAQ items after the workshop (Table 1). The items developed post workshop also required significantly less modifications when compared to question items developed by non-workshop participants.

Conclusions: By delivering training and facilitating multi-disciplinary review and feedback on draft assessment items, the targeted question development workshops could improve the quality of assessment items as well as increasing the level of competency of the clinicians as part of the faculty development program.

Keywords

Abstract Only, poster presentation, written assessment, question development workshop

Comments

Further information about the 9th Asia Pacific Medical Education Conference may be accessed here

Download

Share

COinS