Use of an question-specific algorithm decreases variability in examiner cohorts in written assessment

We evaluated a question-specific algorithm in the correction of short-note questions (SNQs) and compared variability in marking between essay type questions (EQ) and SNQs.190 students were assessed. The EQ was corrected by 2 specialists (consultants in that area) and 2 non-specialists. A model answer was provided. Each of the nine SNQs was divided between 2 examiners – 1 specialist, 1 non-specialist. A question-specific algorithm was supplied for each SNQ. Examiners rated the usefulness of the marking schemes using a 5-point Likert scale.In the EQ, specialists gave a higher grade (54.3+/- 0.9) versus non-specialists (51.5+/- 0.7, p=0.01). There was no difference between specialists' and non-specialists' grades in SNQs (54.5+/-0.3 versus 55.8+/-0.3 respectively. 60% of non-specialists described the algorithm as useful or excellent versus 30% of specialists. 55% of specialists described the EQ model answer as better than previous years versus 68% of non-specialists).We show significant differences between specialist and non-specialist examiners in the correction of EQ but no difference in SNQs, for which question-specific algorithms were provided. The difference may have been due to the contrasting attitudes and thus application of the marking schemes provided.Use of a question-specific algorithm decreases variability in grades awarded by distinct examiner cohorts.