Search Results

The consistency of assessment demands is important to validity. This research investigated the comparability of the demands of college-assessed units within a vocationally related qualification, drawing on methodological approaches that have previously been used to compare assessments. Assessment materials from five colleges were obtained. After some initial familiarisation and the revision of a framework of demands, 15 expert judges compared the assessment materials from pairs of colleges in turn. For each pair, each judge evaluated which college’s assessment tasks they considered more demanding on a number of dimensions and overall. Analyses suggested that assessments were more demanding at some colleges than others but that differences were not large. An interview study was also conducted in four colleges. The interview data suggested substantial similarities between colleges in the assessment tasks conducted. However, some differences that could affect demands were identified (e.g., degree of authenticity of tasks, availability of exemplars when writing reports).

Published abstract reprinted by permission of the copyright owner. Show less

The Angoff method is a widely used procedure for setting pass scores in vocational examinations, in which the awarders estimate the performance of minimally competent candidates (MCCs) on each test item. Within the context of some UK vocational examinations, the procedure consists of two stages: after making the first round of estimates, awarders make final decisions after discussion and after receiving statistical data about candidate performance. This study investigated the relative effects of discussion and performance data on awarders’ estimates relating to a UK vocational qualification. The results of the study showed that performance data had more influence on the awarders’ estimates than discussion alone. However, neither discussion nor performance data had the power to drastically sway the awarders from their original decisions, nor did they significantly reduce the variability of individual judgments. These data were compared to what the awarders reported in questionnaires and interviews. The comparison revealed that there was often a discrepancy between what the awarders said about the effect of discussion and statistics on their estimates and what one can conclude by looking at quantitative data alone.

Published abstract reprinted by permission of the copyright owner. Show less

This article is based on a presentation given at the Journal of Vocational Education and Training 8th International Conference held in Worcester College Oxford, UK, in July 2009. The paper was written at the beginning of a wider research project, conducted within the Research Division. The aim of the wider project is to develop a research instrument for comparing the syllabus demands of cognate units from different types of qualifications. The specific aims of the present article are to review the theoretical approaches, methods and research instruments used to compare vocational qualifications (VQs) in England, with the view to gauging their appropriateness for comparing the demands of different types of qualifications. The wider project is still a work in progress.

Published abstract. Show less