Is the Reality of Multiple Graders Inconsistency: our experience in reducing this with Undergraduate Radiography Thesis Work

Background It is widely mooted that moderation of grades is crucial to monitoring consistency, and hence validity, in the grades awarded by separate lecturers. Typically moderation involves a second lecturer grading a sample of work, often chosen as being at the top, middle and bottom of the grade range. We had applied this approach for many years, but were concerned that there remained an underlying inconsistency in the grades awarded by different lecturers for work of a similar standard. In Radiography, we use a reference image example to ensure consistent multi-observer analysis of images, and we decided to try applying this concept of a reference example in the grading of student work in our Stage 3 and Stage 4 research modules. METHOD To establish our start point, we constructed an objective grading rubric, and had seven lecturers grade a sample piece of work. Grades were spread, emphasising the need for some further reference point. We then applied three approaches to moderation (in three different modules): 1. Seven lecturers and two moderators graded coursework in sight of an example piece of work graded by the module coordinator; 2. Seven lecturers graded coursework without a reference example, then two moderators graded a sample of this work with the comparator of a graded piece of work; 3. Six lecturers graded theses without a reference example, and a moderator ranked the theses into deciles without reference to either the primary grades or the detailed rubric. In each case, as this was research work, supervisors were also asked to comment on whether the grades awarded were as anticipated. Where considerable difference (more than three grade points or more than two deciles) existed between the grades/rank given, or the grade given and the grade anticipated, the module coordinator reviewed the grading and a consensus was achieved between the three involved lecturers. RESULTS In the initial pilot study on the same piece of work, the seven lecturers awarded grades separated by several grade points. With each system of moderation, there was low disparity between grader and moderator, and between grades awarded were generally similar to those anticipated by supervisors. The highest consistency was achieved when a graded example was circulated to all lecturers in advance of grading. DISCUSSION AND CONCLUSION The importance of moderation of grades is well established. Our experience has been that even with moderation, grading standard can vary across a number of lecturers. We propose that to improve consistency across multiple graders, the Introduction of a graded piece of work as a reference standard is useful