The evaluation of electronic marking of examinations

Abstract

This paper discusses an approach to the electronic (automatic) marking of examination papers, in particular, the extent to which it is possible to mark a candidate's answers automatically and return, within a very short period of time, a result that would be comparable with a manually produced score. The investigation showed that there are good reasons for manual intervention in a predominantly automatic process. The paper discusses the results of tests of the automatic marking process that in two experiments yielded grades for examination scripts that are comparable with human markers (although the automatic grade tends to be the lower of the two). An analysis of the correlations between the human and automatic markers shows highly significant relationships between the human markers (between 0.91 and 0.95) and a significant relationship between the average human marker score and the electronic score (0.86).

References

1. Burstein, J., K. Kukich, et al. (1998). Computer Analysis of Essays. NCME Symposium on Automated Scoring, USA.
2. Burstein, J., C. Leacock, et al. (2001). Automated Evaluation of Essays and Short Answers. Fifth International Computer Assisted Assessment Conference, Loughborough University, UK, Learning & Teaching Development, Loughborough University, 41--45.
3. Callear, D., J. Jerrams-Smith, et al. (2001). CAA of Short Non-MCQ Answers. Fifth International Computer Assisted Assessment Conference, Loughborough University, Learning & Teaching Development, Loughborough University, 55--69.
4. Christie, J. (1999). Automated essay marking for both style and content. Third International Computer Assisted Assessment Conference, Loughborough, University, Loughborough, UK, 39--48.
5. Deerwester, S., S. Dumais, et al. (1990). Indexing by Latent Semantic Analysis. JASIS 41(6): 391--407.
6. EAP (2002). Electronic Assessment Project. http://mcs.open.ac.uk/eap
7. Jurafsky, D. and J. H. Martin (2000). Speech and Language processing, Prentice Hall. ISBN 0--13--095069.
8. Leacock, C. and M. Chodorow (2000). Automated Scoring of Short-Answer Responses, ETS Technologies. http://www.etstechnologies.com
9. Salton, G. (1989). Automatic text processing: the transformation, analysis and retrieval of information by computer. Reading, Mass, USA, Addison-Wesley.
10. Thomas, P. G., B. Price, et al. (2001). Experiments with Electronic Examinations over the Internet. Fifth International Computer Assisted Assessment Conference, Loughborough University, Loughborough, UK, Loughborough University, 487--502.
11. Thomas, P. G., B. Price, et al. (2002). Remote Electronic Examinations: student experiences. British Journal of Educational Technology 33(5): 537--549.
12. Whittington, D. and H. Hunt (1999). Approaches to the Computerised Assessment of Free Text Responses. 3rd International Conference on Computer Assisted Assessment, Loughborough University, Loughborough, UK, 207--219.

Index Terms

• Applied computing

  • Education

    • Distance learning

    • E-learning

• Social and professional topics

  • Professional topics

    • Computing education