Making assessment more effective and efficient
The Reflections on Assessment Theme stressed the fact that assessment needs to perform a number of tasks, which Boud (2000) calls 'double duty'. He maintains that assessment activities have:
- to encompass formative assessment for learning and summative for certification
- to focus on the immediate task of, and the implications for equipping students for lifelong learning in an unknown future
- to attend to both the learning process and the substantive content domain.
This double duty is no less true when computers are involved, but the introduction of technology into assessment practices offers the opportunity for a review of them all. The assumption that computer aided assessment will enhance practice on its own is false; it is just as possible for computers to be deployed in support of bad practice as in good.
An interesting perspective on transparency in assessment was examined in one of the workshops for the Reflections on Assessment Theme. It was suggested that transparency contained both explicit and implicit messages and that both staff and students need to be aware of the implicit as much as the explicit ones. The example was given of an assessment brief that reminds students to 'make sure your essay includes a wide range of sources to support your argument', which appears at first sight to be transparent, but which in fact contains implicit assumptions about the nature of discourse in an academic essay; something that might be quite foreign to both first year and overseas students.
The objective of making assessment more effective and efficient was examined in depth in workshop series No 1: Streamlining assessment - how to make assessment more efficient and more effective of the Reflections on Assessment Enhancement Theme, which also emphasised the need to reduce the assessment load on staff and students. A number of suggestions on strategies for streamlining assessment are discussed.
In the overview to the workshop on making assessment more efficient and effective, a number of key issues and dilemmas were identified and these might provide a template for action. In a later workshop in the same Theme, a contributor suggested the following as a guide to keeping assessment cost-effective and manageable:
- Focus your assessments. It's better to do a few assessments well than many poorly. Concentrate on assessing just a few key learning goals rather than every goal of your course or programme.
- Make maximum use of existing information before creating or purchasing new tools.
- Focus on those assessment strategies that give the greatest dividends for time and resources invested.
- Limit the volume of assessment information you collect from students. Perhaps a one-page chart will give you just as much information on students' analysis skills as a three-page essay. Perhaps a two-page abstract will give you just as much information on students' writing skills as a 20-page term paper.
- Use rubrics - they really speed up the process of evaluating student papers and projects.
- Stop doing something else. Consider dropping your mid-term examination to give you more time to assess student projects. Consider moving some of your more straightforward lectures to handouts that students read on their own, creating more class time for students to collaborate on assignments and for you to review assignments with individual students.
- Look at samples rather than censuses of student work. If students maintain journals in your course, for example, spot check a random sample of them each week rather than read them all. If all students in a programme complete a senior thesis, evaluate just a sample of them for writing and critical thinking skills.
- Stagger your assessments. Stagger the due dates for assignments so each class's assignments are turned in a few weeks apart and you're not overwhelmed with papers
- at any one point in the term. Similarly, stagger programme assessments across a multi-year period. A three-year assessment cycle might include an examination of student portfolios every first year, a survey of alumni every second year and exit interviews of graduating students every third.
- Adapt your assessment schedule to meet your evolving needs. Suppose that focus groups show high levels of student satisfaction but senior theses show poor organisational skills. You may want to put the focus groups on a back burner, conducting them only once every three years just to make sure student satisfaction isn't slipping, and begin reviewing theses every term to monitor the effectiveness of your efforts to strengthen organisational skills.
It's important to establish realistic expectations for the quality of dissertations. Assessment is a form of action research, a branch of research that, while disciplined and systematic, is inherently imperfect, so don't expect perfection. While it would be wonderful if every assessment project were designed to meet standards for publication in peer-reviewed research journals, realistically most staff don't have the time - or interest - to do this.
"Aim not for replicable, generalisable research but for results that are simply good enough to use with confidence to make decisions about teaching and learning in your course, programme or institution".
Source: Reflection on Assessment II
The Integrative Assessment Theme examined the 'backwash' effect of assessment, that is to say the impact upon learning produced by a particular method of testing. Evidence on the use of multiple-choice questions (MCQs) provides a good illustration. In research by Scouller (1998), for example:
"Multiple-choice questions (MCQs) tended to be perceived by students as calling for relatively low-level learning outcomes and the adoption of a surface approach which favoured memorisation over understanding, while those students who had taken a deeper and more analytical approach in preparing for the MCQ exams actually performed less well. In the students' essay assignments, by contrast, a deep approach was much more likely to be seen as appropriate, and those who had used a surface approach did less well".
Source: Integrative Assessment - Guide No 3
The accusation of fostering a surface approach to learning was also aimed at ICT and related technologies. The availability of and speed of access to vast quantities of data, it was argued, might lead to a lack of authority of sources and discourage reflection and critical thought.
Alongside the 'backwash effect' of assessment, the impact of tests in providing feedback (or perhaps more properly 'feedforward assessment'), was examined in a number of places in the Themes. The Integrative Assessment Theme argued that it was possible to convert feedback into feedforward by interconnecting assignment and assessment tasks and creating a recursive cycle, a 'feedback loop', in which feedback comments on one task, draft or set of questions could be fed directly into a subsequent task or draft, or will aid preparation for an exam.
The importance of consistent marking and feedback was emphasised, and such consistency can only be achieved through systematic oversight, analysis and evaluation.
In one of the workshops in the Reflections on Assessment Series, it was suggested that there were a number of actions that could be taken to make assessment, in the contributor's words 'more accurate and truthful'. These were:
- Start with clear statements of the most important things you want students to learn from the course or programme.
- Teach what you are assessing. Purposefully help students learn the skills needed to do the assessment task.
- Because each assessment technique is imperfect and has inherent strengths and weaknesses, collect more than one kind of evidence of what students have learned. If you are assessing learning across an entire programme, for example, rather than only give students a culminating examination, you might also look at samples of papers they've written and perhaps internship supervisors' ratings of their skills.
- Before creating an assignment, write a rubric: a list of the key things you want students to learn by completing the assignment and to demonstrate on the completed assignment.
- Likewise, before writing test questions, create a test 'blueprint': a list of the key learning goals to be assessed by the test and the number of points or questions to be devoted to each learning goal.
- Make assignments and test questions crystal clear. Write them so that all students will interpret them in the same way and know exactly what you want them to do.
- Make sure that your assignments and test questions clearly relate to your key learning goals. Each test question, for example, should clearly correspond to the learning goal you've identified for it in your test blueprint. A writing assignment intended to assess how well students organise an essay shouldn't be graded primarily on grammar and spelling.
- Ask colleagues and students to review drafts of your assignments, rubrics and (using former students) test questions to make sure they're clear and appear to assess what you want them to.
- Try out surveys and similar tools with a small group of students before using them on a larger scale. Check students' responses to make sure they are giving answers that make sense. Ask them if they found anything unclear or confusing. Ask some students to 'think out loud' as they answer a test question; their thought processes should match those you intended.
- Collect enough evidence to get a representative sample of what your students have learned and can do. Collect a sufficiently large sample that you will be able to use the results with confidence to make decisions about a course or programme.
- Score student work fairly and consistently. Before scoring begins, have a clear understanding of the characteristics of meritorious, satisfactory and inadequate papers. Then use a rubric to help score assignments, papers, projects etc consistently.
- Use assessment and quality assurance results appropriately. Never base any important decision on only one assessment. (Failure to adhere to this maxim is one of the major shortcomings of many high-stakes testing programmes.) Assessments shouldn't make decisions for us or dictate what we should teach; they should only advise us as we use our professional judgment to make suitable decisions.
- Evaluate the outcomes of your assessment efforts and revise your strategies to address any shortcomings.
The Integrative Assessment Theme argued that conventional student surveys tend to focus on specific courses or modules and do not typically ask students about their assessment experience across units, across and between subjects, and across and between successive years of study, which is far more typical of their actual experience. Find out more about the assessment experiences overall.
This omission could and should be rectified in order to improve the student experience. Certainly there was a view strongly expressed in the Flexible Delivery Theme that students should be allowed to take more control of the assessment process and that technology offered a number of opportunities for them to do just that via computerised tests and effective interactive teaching tools.
The Integrative Assessment Theme raised a number of questions pertaining to the speed of change of teaching, as opposed to assessment, citing, inter alia problem based learning, where, perhaps, assessment has not kept pace with new styles of teaching, or the converse, where innovative assessment has not been matched by teaching. It was also suggested that students may need some training or at least better preparation for some of the new assessment techniques that are being introduced.
One form of assessment that has considerable appeal, but which is often fraught, is that of group work with an oral presentation at the end of the assignment. One of the post-workshop reports on the conference relating to issues of validity, reliability and fairness offered the following suggestions:
- students should be required to undertake/submit both a group and an individual piece of assessed work and should be required to pass both components
- peer marking (appropriately annotated) should be incorporated within any group work, enabling peers to indicate the contribution (by effort) made by members of the group
- mixing groups episodically may prove useful to enable patterns of student working to emerge, particularly for indicating where a student encounters problems in more than one group scenario
- students and assessors should be involved jointly in setting criteria, thus allowing students to have a role in determining how to achieve the learning outcomes, and generating a student contract for the assessed work; this could be particularly important in the case of students with special needs, though it was also stressed that appropriate advice and information should also be sought through alternate means (special needs advisory staff, learning needs identification documents produced in response to the student's disclosure of disability etc) to ensure that teachers/lecturers make informed decisions on student support needs
- to avoid unfair balance between 'workers' and 'passengers' within the group, many modes of assessment should be employed such as the use of group diaries, an attendance record, the setting of clear and equal goals to be achieved by each student, anonymous peer review, tutor moderation, one to one presentation, video recordings etc
- a mixture of marks should be used: group mark, peer assessed mark, individual mark opportunities to find out about the development of the final assignment, what ongoing work/discussions have taken place, how was the work allocated and completed, should be built into the course in order to get a sense of who was/wasn't involved (it was felt that this background material could be assessed, but wouldn't necessarily have to be)
- students should be encouraged to use online discussion boards, if these exist, to talk about their assignment, share ideas etc, and give the tutor access so they can use the facility to get a sense of how the group is working together, who is having an input and who isn't
- the student with the speech impairment should be asked if they wanted to participate in the delivery of the presentation (staff should not automatically assume they can't, or wouldn't want to); if arising from this the student felt that they couldn't, then the tutor should consult with the student as to what they would see as an alternative, and endeavour to provide it
- alternative methods of presentation should be explored such as video, Microsoft Power Point etc.
The issue of privacy, for both the students and teachers was raised at the conference on validity, reliability and fairness in assessment and the following suggestions were made:
- Anonymous marking should be used wherever possible (although this may not be feasible where individual focus identifies individuals e.g. placements etc).
- Enabling students to view their marks individually e.g. through an online portal (rather than positing on a board - even in an anonymised format) will allow students to decide how widely they wish to share their marks.
- Results could be given by number, not by name.
- Anonymous peers' assessment is another option (and should be possible if the criteria are explicit and detailed), with the tutor moderating/marking a sample.
- Evaluating student feedback in an anonymised fashion, removing any reference to individual tutors (by name) will protect professors' anonymity in published evaluation results; specific issues relating to individual tutors should be taken up through an alternate, developmental route (such as appraisal) - not through 'naming and shaming'.
- Providing generic, anonymised feedback to the group (on collated issues) can permit for issues to be tackled without individuals being identified explicitly.
- On the other hand, individualised feedback may be necessary (see first bullet point above).
One aspect of CAA that was explored in Reflection on Assessment II was the way in which its definition could and should be extended to include administrative and management aspects of assessments. In particular, CAA offers staff the opportunity to analyse the students' results; the examinations' validity and overall teaching effectiveness.