Assessing phonetics production skills remotely

Like everyone else in Higher Education, I had to adapt my teaching and assessment for remote delivery at the start of the pandemic.  This included having to find a way to assess phonetic production skills online.

Traditionally production skills are assessed by an oral exam.  These oral exams vary, of course, by institution and by discipline, but, typically, students would produce sounds from the IPA chart, label consonants substituted into English words, and describe intonation produced by themselves and their examiner.  Exams are usually around 15 minutes long, and take place individually, with one, or sometimes two, examiners.

In my previous work I’ve look at how students feel about oral exams, and, not surprisingly, these types of assessments are anxiety provoking, although I have tried various methods of reducing that anxiety, such as showing videos of a mock-viva, and asking later years students to come and share their experiences, with some success.  Combined with the difficulties for students, one-to-one exams are also resource intensive for markers.  I’ve certainly spent far too much time over the years booking rooms, laminating stimulus cards, and trying to make everyone tea during examining breaks!

It probably shouldn’t be surprising then that, anecdotally, many courses had stopped formally testing student production even before the pandemic, and, for others, remote assessment created a final, unassailable barrier.

However, I am really committed to production as a crucial skill in phonetics.  It helps us demonstrate a full range of phonetic skills, can solidify our mental representations of speech sounds, and can potentially be of use in future work, such as demonstrating sounds in speech therapy or second language teaching.

So, I was keen to find a way for students to demonstrate their production skills online.  Luckily, during the pandemic, everyone had become much more familiar with using educational technology. However, I didn’t want to simply replicate live oral exams and do the assessments via Zoom or Teams.  I suspected this would be pretty nerve wracking for students, with the additional worries about technology failure, on top of worries about the exam itself.  It would also require many of the organisational aspects that make live oral exams so burdensome for staff (albeit without the laminating and tea making).

Instead, I asked students to make a video recording of themselves.  After materials were released, students had 24 hours to upload their video to the University video hosting platform.  I removed the ‘substitutions’ part of the oral, and instead students demonstrated their awareness of phonetic terminology and VPM  by labelling the sounds they had been asked to produce.  They also recorded and analysed two intonationally different versions of a sentence they produced themselves. Finally, I took advantage of the more flexible timing arrangements,  as students now had 24 hours, rather than 15 minutes.  This allowed me to ask students to reflect on their performance, and say in the video what they felt when well, what could be improved, and actions they would take to improve their skills.

I was incredibly impressed with the videos our students produced.  We had no problems with technology, and all the videos that were uploaded were playable, and with clear enough visuals and sound to be easy to mark.

I am definitely going to keep the video recordings in the future rather than returning to live exams, and have gathered together below some of the things I have learnt, or need to think about further.

1. Provide a model video.  Students might not be familiar with this type of assessment.  I provided a video of me, as if I were a student taking an oral exam in this format.  It allowed me to illustrate features I wanted, including those that formed part of the criteria (e.g. producing two different sentences for intonation analysis), and those that were not (such as just correcting any ‘mistakes’ I made, rather than editing them out (see 4, below).
2. Allow students to upload a practice video.  In line with good pedagogic practice, this formative assessment allows students to test out the technology, and the lecturer can provide feedback on the productions too.
3. Be flexible with technology.  Students used a variety of devices and setups to produce their videos.  Many used their phones, some used their computers, and others has family record them with a video camera.  As these were all uploaded to our video hosting platform there were no issues with compatibility, or with videos not playing. Providing this flexibility means there is one less thing for students to worry about.
4. Encourage students to do the assessment all in one take without editing.  In the face to face orals, we always encouraged students to have several attempts at production and we marked the best version.  Keeping this feature in the orals should be less stressful for students, and means students are not spending time editing videos, which is not one of the assessment criteria.
5. Use a PDF for the assessment instructions.  I wanted to avoid students having to open up an extra document, and would have preferred to put instructions directly into the assessment description in the VLE.  However, it didn’t cope well with the position of diacritics, whereas a PDF obviously preserves this information.  An additional advantage of the PDF is that students can download it if they want to move away from their screen to practise before they record.
6. Use a spreadsheet for marking. I set up a spreadsheet with columns for each mark. I found, once I got into the rhythm of it, that I could have the spreadsheet open and mark in almost real time, just going back to check when I wasn’t sure about something. This had the advantage of totalling the mark automatically, which is infinitely preferable to manually adding up dozens of marks per student, as we always did when examining live.  As most student videos were less than 6 minutes long, even listening to them twice took less time than the equivalent live assessment.

The last academic year has, of course, been unbelievably sad, and challenging for many of us, both personally and professionally.  However, there are some small positives that have emerged.  For me, moving away from a stressful and difficult to organise live exam, to a more flexible and relaxed recorded exam is one of those positives, and I’m looking forward to hearing what others have achieved with their oral exams, and how we can develop them as a community in future.