Use of an question-specific algorithm decreases variability in examiner cohorts in written assessment

Spooner, Muirne and Meagher, Frances and Corrigan, Noel and McElvaney, Gerard (2011) Use of an question-specific algorithm decreases variability in examiner cohorts in written assessment. [Conference Proceedings]

[img] Text
NAIRTL 4th Annual Conference (Cork) 2010 Flexible Learning Conference Proceedings_sm.pdf

Download (11MB)

Abstract

We evaluated a question-specific algorithm in the correction of short-note questions (SNQs) and compared variability in marking between essay type questions (EQ) and SNQs.190 students were assessed. The EQ was corrected by 2 specialists (consultants in that area) and 2 non-specialists. A model answer was provided. Each of the nine SNQs was divided between 2 examiners – 1 specialist, 1 non-specialist. A question-specific algorithm was supplied for each SNQ. Examiners rated the usefulness of the marking schemes using a 5-point Likert scale.In the EQ, specialists gave a higher grade (54.3+/- 0.9) versus non-specialists (51.5+/- 0.7, p=0.01). There was no difference between specialists' and non-specialists' grades in SNQs (54.5+/-0.3 versus 55.8+/-0.3 respectively. 60% of non-specialists described the algorithm as useful or excellent versus 30% of specialists. 55% of specialists described the EQ model answer as better than previous years versus 68% of non-specialists).We show significant differences between specialist and non-specialist examiners in the correction of EQ but no difference in SNQs, for which question-specific algorithms were provided. The difference may have been due to the contrasting attitudes and thus application of the marking schemes provided.Use of a question-specific algorithm decreases variability in grades awarded by distinct examiner cohorts.

Item Type: Conference Proceedings
Depositing User: National Forum
Date Deposited: 06 Dec 2015 19:09
Last Modified: 12 Dec 2015 15:36
URI: http://eprints.teachingandlearning.ie/id/eprint/3737

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year