Keynote Changing the culture of assessment and feedback through TESTA Professor Tansy Jessop, Professor of Research Informed Teaching, Southampton Solent University
Tansy was one of the original project team for this methodology started at Winchester. During the project time 75 programmes were audited, 2000 assessment experience questionnaires were used and 90 focus groups. A case study was then produced for each programme and given to each programme team.
Post HEA project funding this has continued to be a popular project and grows across institutions with the top cities for use being London and Edinburgh in UK. This methodology shifts the focus of discussion about assessment and feedback positively from ‘my’ module to ‘our’ module and programme and away from individual modular design to coherent team design. There is also a shift from teacher focus to student experience.
TESTA addresses 3 key problems which are:
1 Knee Jerk
The NSS assessment and feedback sections do have reduced scores nationally and individuals have done many things to improve this but these are not connected so the scores are perpetuated.
2 Curriculum Design
This accentuates the ability to be disconnected and divided and the curriculum privileges knowing stuff.
3 Evidence needed to action problems
A study in the US by Wobash found that there are some misguided assumptions which include the problem is a lack of data. Proving something is different to improving.
TESTA found in the projects that some programmes had 40 – 48 summative assessments usually with 6,000 words per module but only 5 – 19 formative assessment which was often a ratio of 1:8. When feedback was analysed it was found that in three years students had from 936 – 22,000 words of feedback. There were 11 = 15 types of assessment. Too much effort was in the wrong place. Examinations were 22 – 31% of the assessments. There was a need to streamline and reduce summative assessment and connect formative assessment. It is usual to have two assessment per module and we know that attendance dips around assessment.
The language needs to be shared with students and in the public domain. We need to use risky, creative and challenging assessments but related to the real world. Mass education has led to anonymous feedback which has no relationship with the student. Technology offers a range of formats for assessment and feedback.
What is needed is iterative cycles of reflection over modules. Feedback needs to change from telling students to feedback as asking questions. Assessment has not developed with the teaching to draw on the latest pedagogies and move from transmission to socially constructed models of assessment.
Parallel Session Experiences of using a peer marking system for formative assignment grading Sundari Joseph, Audrey Stephen
The module team were in nursing and taught both an undergraduate and postgraduate research module. They wanted to introduce something formative to help with summative assessments. They also wanted to raise awareness in the team of the benefits and challenges of online marking systems. They explored a range of options but settled for the option in moodle which can be used for peer assessment. They set up a rubric that could be used which was matched to the summative assessment. Each student then had to submit 800 words which appraised three research articles. The submission was as a word document which was uploaded. The marking is then by grade A – E and free comments can be added. Staff could override the student mark if needed and identify why they had done this. The team also published exemplars for grades. The formative assessment was voluntary so not all students engaged in this. Each student was allocated three peers to mark. Individual student engagement was then reviewed using learning analytics. There was a face to face meeting initially to explain this activity and how to use the rubric.
Students found this helpful and made them think about their work more and staff said it did engage students in asking questions about their assessment.
Parallel session Testing TESTA success at Dundee? Does TESTA Pass the TEST? One University’s Perspective Lynn Boyle, Neil Taylor
Transforming the Experience of Students Through Assessment (TESTA). This paper focused on early experiences of using TESTA. There are some key steps involved in this which Tansy Jessop (2010) calls the 10 steps to TESTA but they were adapted for this and included:
- Programme audit – so gathering information for the whole programme
- Gathering students views – 28 item questionnaire with nine scales
- Focus groups with final years usually 1-2 group
Two programmes were used for a pilot and the outcomes were robust, positive and relevant. Phase two was then undertaken and this involved 5 programmes, 0.5FTE quality enhancement officer and 2 lecturing staff. There were issues related to time and scale. So the 10 steps are an issue as was getting a whole programme team into a room for a day and undertaking the work module by module.
There are 341 programmes and so there did need to be some consideration of how to make this scalable. To support this the team created a TESTA brick which is a one page document what contains all the key information about a module when the team complete this. There is also a need to collect samples of feedback for each year. The samples are reviewed and the words are counted to develop an estimate of how many words feedback a student gets each year. The themes are also drawn out as points to consider. Phase 3 involved 16 programmes and giving feedback to the programme teams has always been the hardest part as there is a certain level of concern about the feedback. With the increase in the programmes resources are an issue and so the 12 Associate Deans Education have taken part in reviewing feedback and have done this across discipline. This has worked well.
This process whilst good does create some issues. At the initial meeting where findings from focus groups and analysis are done staff are very angry and question your right to say things are not quite right.
It is now proposed to adopt TESTA across the whole institution but to do programmes the year prior to the programme periodic review. There is enough data now in the institution to use this as the data set rather than using the TESTA data. The bricks have been well received and is an easy way to see the programme.
In terms of feedback some of the language used was emotive and not all related to the learning outcomes and did not always provide guidance for development. This process does facilitate discussion around this and the NSS in this area has increased by 6% on the programmes that have been involved in this process.
Parallel Session An evaluation of summative assessment feedback forms: students as co-creators of knowledge Jane Rand
This session focused on students being researchers and focusing on two questions which were:
- How do students engage with summative feedback?
- How comprehensive is their understanding of strengths and areas for development?
Focus groups were undertaken with 30 students from both 1st and 2nd year students with 1/3 1st year students and 2/3 2nd year students. Themes that came from these were that there are inconsistencies in the assessment descriptors and narrative, there are inconsistencies between lecturers, vagueness in feedback and students want personalised feedback. The marking records at that time had the final grade at the end and students reporting looking at the grade and not the feedback with many not reading it at all and some if they failed read it the next day. In the main most students do not use feedback for their next assessment.
There was a need to rethink how feedback was provided and used so engagement was achieved. The team decided to amend the marking record so the grade was at the top and they also wanted to know what feedback students wanted. Students created a new template which had the grade at the top but asked students what feedback they wanted so lecturers could respond to that and then there was section on there where students had to say what they had done with the feedback and this was discussed with their personal tutor. Further focus groups were undertaken and students are now using the feedback and feel some ownership of this.
Keynote The feedback conundrum: finding the resource for effective engagement Professor Margaret Price, Oxford Brookes University
Margaret favours discussing assessment as ORA which is ownership, Responsibility and Autonomy. We know that assessment is a key driver for learning and yet we do not draw on pedagogic developments for assessments. The discourse used is so often focused on checking the grade, grade inflation and fairness. Many have moved to anonymous marking and yet we know students want personalised feedback. Measurement in higher education has now got an unintended consequence of fragmentation because of modularity. Authentic assessment is what the employer market wants and yet we continue to use traditional approaches. Our assessment cultures come from a collusion between staff and students (Eccleston 2006).
There is an assumption that designing assessment at the module level is sufficient but assessments need to be viewed at programme level and we have to move to this. Research has shown that where the where the assessments are designed across the programme students do well and yet where they are a collection of modules students did poorly. Variety in assessment can be good but too much is disconnecting for students.
There is an assumption that constructive alignment is enough for coherent assessment and feedback. There is also an assumption that standards are straightforward but we know students do not understand criteria and consistency between markers is not robust. Students go through programmes mostly seeing only their own work and so have no chance to compare. We need to create a community of practice that includes students.
Assessment judgements are complicated with markers interpreting criteria in light of their experience and sometimes using other criteria. There is an assumption that there is a common view of assessment however we know that students don’t read it, sometimes it is vague, it is open to interpretation and it damages self-efficiency. Students have emotional responses to the feedback.
So what makes good feedback? Staff from ASKe and Cardiff under took and HEA funded project. Students became researchers and did semi-structured interviews with students using one example of good feedback and one example of bad/poor feedback. Students also undertook analysis with support. 3 key findings were developed which were:
- The feedback itself was often very technical and looked at presentation, legibility and levels of explanation. There was recognition of effort and time spent on work but we don’t know what time students have spent just based on the work
- The context of the feedback was important so the assessment design was important in terms of purpose and relevance. Feedback was given related to criteria but this was not a dialogue.
- Students’ development and expectations were the last area. Students showed the mark had some influence but some were upset but comments. Some students can self -evaluate but others can’t.
The summary of the findings was that you don’t need to get it right all the time but students perceptions are shaped by pre-feedback conditions. There is a need to improve student learning development so they are better equipped to engage with assessment.
Parallel session Linking skills, feedback and assessment to develop student agency and achievement Laura Ritchie
This session focused on wanting to give students freedom in assessment. We as lecturers can invite students to do things but we can’t make them. We have to engage them and develop authentic assessment for learning. This has to be unique to the discipline and encourage students to seek help as they need it. We need to prepare them so they are self-regulating.
Laura then asked all the participants to undertake an activity based on the session so far. She asked us all in groups to produce an essay on what she had said so far. This was really interesting with five groups exploring a range of things such as what is an essay? Did the essay have to be written or could pictures be used? What was being assessed? We were like students and did not understand what we needed to do.
This led to a discussion about whether students could choose the format of their assessment as long as they had the criterion and meet these. Essays could become more multi-dimensional with video clips and pictures inserted.
Parallel session A Marking App: from student engagement to feedback and grading Victoria Jack, Tom Smith
The session focused on the development of a marking app which could be used to grade work and also provided some learning analytics about marking activity. This was developed because lecturers took part in a summer school with 100 students and marking of a piece of coursework had to be done in 3 days. The digital tool made tis easier and quicker.
The app was developed following some exploration of feedback given previously and discussions with the markers. What emerged was that 5 categories which marking comments were made around. The team then wrote descriptions for each grade that could be awarded for each category. These were then taken to whole team meeting and revised. The app then had all these added. Markers then used the app to do all marking online.
I6 staff took part in focus groups after the marking and were very positive about this. They said it saved time and you could refer students to the comments in the grade descriptors. They did not however like the calculator and so some changes will be made for next time. A positive feature tutors felt was the analytics where they could see their performance over the time they marked and what grades they gave at different times but also how they were marking compared to the overall tutor performance. The app enables anonymous marking as well.
Parallel session – But how do I know what you really want us to do?: using exemplars to develop assessment literacy and competence using readily-accessible exemplars Sally Brown, Kay Sambell
Assessment literacy is crucial to students’ retention and success. Making assessment related information explicit involves more than publishing this it requires students to actively engage in dialogue. Sadler (1989) noted that for students to be able to develop quality work they need to know what quality they are aiming for.
Projects have looked at giving students pre-assessment activities that examine exemplars but these are often finished products This can lead to students mimicking these, feeling terrified or a lack of craft in developing the product. It is however useful to engage students in actively making judgements around a range of exemplars so they understand what is required. A project at Northumbria used part developed exemplars rather than full products. These were one side of A4 which students all print (there are 3) and bring to the session. Students are given the criteria and then in groups they rank order and do feedback. Tutors then tell them what the feedback would be for each and discuss the rationale and then students revise their feedback.
This has been a 2 year project and the workshop has 120 students in their first year and it takes 2 hours. 2 researchers observe the groups and the essays and feedback were collected in. Students were found not to have understood the question and 52% did not rank the fail exemplar as the worst. In fact students found this to be the most useful. Students found this activity really helpful in examining assessments tasks more closely.
Keynote Assessment and feedback – why do we bother? Ian Pirie University of Edinburgh
The conference topic innovations in assessment was clearly a popular choice despite there having been a lot of dissemination of assessment developments over the last decade. It seems that whilst there has been many advances in this area there is still evidence that practice could be further enhanced. The conference provided an opportunity to hear about practice across the sector and how developments can further enhance practice.
The conference opened with a great keynote from Ian. He outlined the background with reference to the good practice across the sector but the NSS still shows that there are issues.
Assessment needs to be integrated into learning as part of pedagogy and not just added on to the end of modules and programmes but making changes is difficult. Today’s students do need different assessments as the world is rapidly changing and the students need skills that will help them gain employment within this world. Students have different skills now and have access to information that is readily available through the internet. They do not need to have lectures that just provide information as they can gain this elsewhere but they do need to be engaged in active learning that challenges and develops them. Employers today expect one in two graduates to become tomorrow’s leaders and so there is often a mismatch between what we do when preparing them for the world of work. Students like to have personalised academic guidance, advice and support, clarity of expectations, an academic community and 24/7 access to services.
We need to teach students to be more creative and innovative and so assessment needs to focus on this too. The design needs to assess not just the work but their development as a person. The design is therefore critical to this. Assessment is an academic judgement but one that must be based on clarity of what is being assessed and explicit criteria that students understand. One approach to ensuring this is to engage students in re-writing the learning outcomes so that students have these in a language they understand. In terms of feedback this should help students make improvements but this means they cannot be passive recipients of this. We have to help students take responsibility for their learning and engage with the feedback.
Postgraduate and research experiences parallel sessions
The first paper focused on design based research in an educational context by Natalie and Celine. This project required active involvement of schools and teachers in educational research and to see it as beneficial. This was a large scale research design with 5 institutions and 36 schools. The plan was to develop effective interventions and practitioners and researchers worked together in different cycles of the innovations. The researchers went to the schools and stayed as this was action research and would have an impact on the school. Over 40 linked organisations were also contacted. There were four experimental tracks which were teaching methods and assessment, information and communication, networking and mapping existing initiatives. There was a discussion with each school about which track they had an interest in but some schools did not stick to their original track. There was a formulation of tracks in a different way ad this meant there were eventually 9 tracks that were flexible. There were also different schools in terms of approach so E+ was a school that had experimental guidance for the innovation and developed materials with the researchers, E- was an experimental school but got given materials and no guidance and the last school C was a control a school. Schools started to collaborate and so shared information which then impacted on the research. In addition as soon as the researchers stepped into the school other processes of change occurred. There was a question around the culture of the school and mutual expectations and need. Where there was strong leadership there was strong school policy which makes a difference. It took a long time to bring innovation into the classroom and do the research.
The second paper by Annemarie was on article, book format, or both? Shared criteria adopted for the double doctoral thesis format and language in a European/International joint networked PhD programme. This was about which format to adopt and sharing expertise and experience. In 1993 a joint programme was developed and this has changed over the 23 years it has run. The initial network had 13 countries involved but this has continued to grow. This is very positive but does mean that on occasion conflict management strategies are needed and some of the partners are now outside normal academic institutions. In order to decide what format there had to be a view about current institutional practices and what would work from a quality perspective. There is some flexibility so a book with a short article or a collection of articles. These are also often submitted in the national language and English however these are today nearly all in English. The book format is 250 – 400 pages. The articles have to have been submitted for peer review and there is a viva. The mid forma between the book and the article has been 40 – 80 pages and there is substantial pressure to publish. There are also concerns about the journal impact factors, university ranking and citation. There has also been pressure on students to write rather than read. Institutional collaboration must have agreement on the project format.
Marielle discussed her paper walking the teaching, research and community – engagement tightrope: dynamics, reflections and lessons learned. This was about community engagement in research. The university sees knowledge production as important and the order of things is that there is teaching, research and community engagement. This is a shared responsibility of HEI’s which is mutually beneficial. Boyer’s scholarship of engagement is a focus. In 1997 there was a white paper in South Africa, and HE policy became inclusive. Academic duties and responsibilities were clarified alongside performance indicators, a workload breakdown and appraisal. In the department of computer science there was a 3 year BSc, an MSc and a PhD programme. There is also a centre of excellence in ICT for development. There is a focus on research informed teaching and applied research to augment student learning. Service learning included volunteers, field trips and internships. Participatory action research was undertaken. Lessons that were learnt included community engagement must be tied to the context, discipline specific and allow for reflection.
How do academics view the research-teaching nexus? A review of literature was a paper given by Kinga. This focused on a project to explore the links between research and teaching. There was little correlation between research and teaching. The study wanted to look at the impact of integrating research with teaching. A literature review retrieved 38 studies that were similar and 11 of these were excluded and then from the 27 left a matrix was developed. The literature showed that there were values attached to the views, there were differences between academics and others and the context was important. The studies had very diverse outcomes and most academics saw their role as not linked to research. The consequences of this project is that through this nexus students were stimulated to develop critical thinking and academics did gain a better sense of teaching.
Sarah presented the last paper which was about making sense of self-initiated expatriate academic careers in Japan. There was quite a lot of academic mobility in the business and management faculty. The study explored how these academics make sense of their choice to expand their current career. It was an exploratory study with 30 interviews which were 20 males and 10 females. A snowballing technique was used to recruit participants. The coding was done in 3 steps and used bracketing and labelling. There were a range of reason that people had taken a job in Japan which included adventure and a possible long term relationship but most had no intention of staying. They all had a sense of increased marketability with better knowledge of their skills and how these can transfer. They felt they had influence due to their social and cultural knowledge. They took on roles that included administration, teaching and research.
