Quality Assurance in Education parallel session.
The first paper presented by Monica on peer observation in higher education as an agent of change in teaching and learning. The focus was on undergraduate and post graduate students with 40 teachers and 1200 students. The peer reviews were one element of quality. The literature that was reviewed discussed the practice of peer review, how to develop this, sharing practice, taking risks, how it was a powerful learning experience and how it can be inspiring. All aspects of teaching were observed. This was undertaken in a positive climate of dialogue and was open and collaborative leading to a critical friendship. Students were also included as partners. The teachers and students actions as peer reviewers were compared. The rationale for the project was pedagogic excellence and student empowerment and the balance between these. In the train the professor to teach document which is a European publication there were 16 recommendations and these referred to peer feedback.
Staff sessions were organised and there was a protocol and observation sheets that staff and students used. All were told to describe the observation rather than evaluate. There were lectures observed by 15 teacher and 97 students and tutorial classes by 9 teachers and 73 students. This meant that a total of 24 teachers were involved and 170 students. 50% of the teaching staff volunteered to be involved. A qualitative approach was used for the research. The results from the teacher observations showed that there was active student participation, real life cases used, enthusiastic teachers but also low engagement and no participation. There were suggestions for improvement which included more student engagement, provide extensive answers, call students by name, and use pre-class activities. The results from the student activities were loud clear speech and sessions were well planned. Weaknesses noted were extensive information was given in some sessions, some questions were unanswered and the topic was not clear. The discussion about the experiences with the reviewers outlined how the exchange of experiences was valuable but students needed more preparation to be reviewers.
The next paper was about maintaining relevance in education through precise assessment and was given by Kevin Smith. This was focused on engineering students who were mostly 20 years old. Teachers appeared to pay attention only to average levels so students missed key outcomes and students felt this was unfair. Some felt that all information about work on the course was available on the web and so why should they come to university if they did not want to work. The questions related to what should change and what kind of quality do we want? Students often come for traditional education and assessment models. The fixed assessment approach was balanced with predictability and risk. There were multiple objectives and students needed to move. There were defined layers of outcomes but assessment needed to assess the achievement of each outcome. Flexible assessment could make sure each student achieved the outcomes, all were defined in advance and they challenged the students. The class size was 25
Tak presented a review of lecture class evaluation items in a Japanese university. There was increasingly a focus on educational quality and instruction for new teachers. There was also support for improving the curriculum. The lecture class evaluation was a questionnaire which asked students what is needed for lecture improvements. The students felt the questions were too abstract and the timing for the evaluation was too late as well as just being a ritual. Adapting the system led to each lecture having a suitable questionnaire and at a time when changes could be made. To develop this the researcher reviewed 683 items on 49 university evaluation forms. He looked at what was asked, the allocation of questions to the topic and the words used. There were 18 topics overall with many similarities across institutions.
The effect of relational coordination in online education was presented next by Carmen. There are more mobile devices in the world than there are people as many have multiple devices now. This means new skills need to be learnt and training needs to be provided. Gittell 2009 offer a good framework to explain results in IT and factors that impact on technology. Relational coordination is based on mutual adjustment, teamwork and communication. Coordination was also based in relationships and the integration of tasks, respect for knowledge and sharing objectives. Small projects were undertaken with teachers working together with students to learn.
Competence evaluation parallel session
Jennifer Kidd started the session with a paper on the effects of peer review on students learning a comparison of positive and negative feedback. The purpose of peer review is formatively to provide feedback and summatively to give a grade. You do need however to teach the students to give feedback as well as receive it. Formative feedback is effective and promotes learning. The study focused on formative feedback and asked what prompts students to revise their work? The study asked students when submitting their final version of their work to respond to some questions which included:
What changes did you make?
What prompted the changes?
If you had more time what other changes would you make?
What would motivate you to make the changes?
The logistics of the feedback were complex with online support, face to face, computer supported but also pen and paper. There was some ranking and rating and there were feedback checklists. The tools used included e-mail, track changes, blackboard and google docs. The study gained both negative and positive feedback. The negative feedback might prompt revision but it might also lead to students ignoring the feedback. Positive feedback was more effective when given to junior students and students sought feedback from those they know better. The study took place in an undergraduate programme and students were given directions as reviewers and asked to focus on specific bits of essays. The data collection was comparing drafts and final essays. There was then a survey with students where they reported making changes of 70 -80%. Most changes were made in relation to negative comments. The challenges were how to measure the change and defining what prompted the response.
Julia Morris discussed teaching students to give and receive – improving disciplinary writing through peer review. This was a snapshot of a one year project which was at the six months point and involved education, biology, engineering, special education and English. The focus was on improving undergraduate writing and a faculty specific assessment was used with guidelines and a rubric. Students submitted drafts, received feedback and the submitted the final version. There was online reflection and a survey after every round of peer review and this was anonymous. The technologies used were found to have some issues but google docs did allow some colour coding in the peer review process. Expertiza was also used as peer review tool.
Results showed students valued specific feedback, giving and receiving feedback, they did find it uncomfortable sometimes and got contradictory results sometimes. The first round of the reviews was the most helpful. The teachers found that this approach was good and they could see how the quality improved and the grades. Students made submission deadlines and made revisions. Negative comments were about the technology and that some cheating ad occurred where some reviewers did others reviews too and some reviewers were not critical enough.
Miguel then spoke about the contemporary issue knowledge outcome assessment in first year degree students. This was focused on an accredited outcome based programme with 13 outcomes for university programmes. There were 120 students divided into groups of 4-6 and the students chose a topic that they worked on together on google docs. There was an outline presentation provided by all groups. This was assessed using a rubric and assessing as a team worked well. The results from 116 students were that 24 screencasts had been produced for their topics.
Evaluation of interdisciplinary projects in pre-primary education degree was presented by Omar. This focused on first year students and natural sciences and maths. The subjects were combined and students had to solve a problem. The main aim was to get the students working together so 45 students divided into groups of 4-5. They had objectives to achieve and teachers form both the subjects worked with them in class. The assessment was undertaken by the teachers and there were 6 phases. Once a project was chosen, students had to design the project plan and structure it. Then had competencies to achieve and then they were assessed. 2 hours a week was allocated to work with the students and there were 3 control and monitoring sessions. There was self and peer assessment. Students did interviews and they liked the self assessment, expert assessment and teachers rubrics.
Tanju discussed freshman communication students’ development of lexical competence. Projects were undertaken by students to engage them in reading, research, using book chapters and seminar texts. There was often a lack of focus by students on lexis. Academic texts could have a high frequency of words (top 2,000) usually 87%, academic vocabulary was 8%, technical vocabulary was 3% and low frequency words was 2%. The teachers wanted to check the seminar texts and so some were analysed and students needed to exposed to a word 8 – 12 times to learn it. Students had to actively engage with the word and so instructional activities were designed to support this. There was self-assessment with the students and a self-evaluation survey. The students were given a list of words in their seminar texts as a handout. There was a research tool used which was profiler of words. The evaluation reviewed reflective writing papers. The experimental group used more academic words then the control group and had fewer spelling mistakes. Quizzes were also used. The students liked the online tools least and liked the quizzes , games and writing exercises more.
Enhancing Learning and the Undergraduate Experience Parallel session
Linda Reneland-Forsman presented the first paper on Guiding Principles for course design. This project was focused on better support for students and as a three year project involved whole programmes. Contact was made with Heads of departments and teachers and 120 hours per year was given over to develop courses for this project. This was based on three principles that of minimising individual e-mail contact with students, involving students with key concepts from the course literature, assessing students by filming students to assess verbal skills and using Ipads and phones for reflection. All students did video reflections. This project worked well and students were more engaged.
The second paper presented by Sutamsa Rodchua was focused on innovative and active learning strategies in higher education boosting satisfaction and effectiveness. This was centred around graduate programmes with active learning. The project aimed to get students to work as a team although some students did not like working I teams due to variable contributions of students. There were 250 students involved in the project and the team focused on Drummonds twelve best practice steps. The benefits of the project based learning was the relevance to the real world practice. This helped developed collaborative and teamwork skills as well as critical thinking. A student charter was developed and a conference with an expert panel. There was a research poster competition and simulation games, an ethics game and a student portfolio. Personality testes were used with groups and self and group member evaluations. There was also a book donation project. This all worked well and did engage students.
Student oral presentations developing the skills and reducing the apprehension was the third paper presented by Chris Ireland and focused on accountancy students. It was difficult to solve the students problems with oral presentation when teachers did not know what the apprehension was about. Academic presentations are an efficient and effective means of assessment (Bell & Reden 2014). They looked at the factors that impacted on students and had five areas which were novelty, illusion of transparency, cognitive orientation, audience characteristics and inborn individual differences. some students were more apprehensive than others and using interventions gained mixed results. Self-efficacy Bandura was related to one’s belief in capabilities. Performance accomplishment was related to time and space to practice with peers. Vicarious experience was gained through peer observation and all provided feedback. The PRCA 24 item questionnaire for verbal presentations was used to evaluate and some interviews. Ethics approval was sought.
Martin Dvorak then presented the paper I have read the chapters but do not remember – factors affecting intake, retention and recall. The scarcity principle was a focus of this study. 57% of Swedish workers report a change in their work due to the use of technology and 12 – 15 year old students use it all the time. Cialdini (2006) said that people value items they cannot have rather than those they can. Students can browse materials on their ipads and phones but they do not value this but expect it. Mobile technology can be seen to be counterproductive for cognitive strategies. Students lack perseverance and assessment tends to support this by not requiring deep enough learning to be demonstrated. Solutions included encouraging students to use paper based materials, teaching them how to study and plan study time and having weekly quizzes.
Estelle Chamoux then discussed participation activities a tool for improving students’ self-responsibility and engagement in their learning experience. This was focused on real life and students needed to use previous knowledge. Students were to create new knowledge through individual and group work and, develop higher order skills. There were weekly activities and oral presentation but this did fit the students’ needs. This was done with students on both 3 and 4 year programmes. The challenges were other professors thought it was a waste of time, took time to grade and students didn’t know what to do. Students also thought it was a waste of time, they paid for education so should be taught and they didn’t need new approaches. The research was hoped to convince students. Three areas were noted in the research academic impact, engagement and motivation and, which one was most effective? This was undertaken with two groups of introductory students with a background in high school biology. 74 students from Autumn 2014 and 67 from Autumn 2015. For most this was their first time away from home and their exam average was 62%. The students were required to do 12 activities but only 9 were used for grading. If students did below this number then their exam results were lower. Students self-reported study time was 2-4 hours per week. 56.7% always attended class. The activities did help them understand materials and remember them. When checking which activities students remembered these were not complex ones.
The last paper in this session was Higher education for tourism industry traditions and innovations from Olga Burukina and Alexander Yandovskv. There were two problems they were interested in and these were the attitude to education in tourism and technology used for tourism. Tourism and culture in the 1990’s developed closer links and increased cultural activities. Creativity now has a big impact on tourism and education. Students need to learn creativity, empathy, curiosity, resourcefulness and resilience. There is also a need for sustainability. The team therefore developed a more project based approach to their programme and engaged participants from industry who could then use this as CPD.
Learning Spaces and next generation classrooms parallel session
The first paper focused on if you build it will they come? Supporting the development of innovative teaching and innovative teaching space given by Nick Almond and Dave Aldridge. The institution had launched a learning lab last year which could hold 60 people and had 6 screens, 2 projectors, white boards and moving tables. This had been designed for collaboration and bring your own devices. Uses Kramer and creston to connect to the screens but each be used independently. Students learn in a range of ways but some tutors can be resistant to change and don’t engage in new models. This was therefore introduced through a phased approach across 250 lecturers. Initially 15 people used it and now 60 -70 are using it. For the first 15 lecturers someone sat in their sessions and observed keeping a reflective diary. Some interesting themes arose such as the session was either teacher centred or students centred but not often in the middle, some tutors were risk adverse, some could not problem solve any technology issues and some were only intermittent users. Those who were good with the technology did not want to be told what to do. New support is needed but conceptual change does occur.
The next paper was about a model that allowed teachers to reflect on their ICT approaches: the convergent learning space (L P Kieldsen and HW Kiaergaara). The background to this was that there was a range of technology rich environments and a range of student devices being used. In the study there were 7 participants who used new approaches, different tools, different spaces. The participants were linked to a research partner and asked what they planned to do. There was some evidence of new practices but small study.
Bruce Gatenby presented a paper on an e-workshop for academic writing and the active learning space classroom. This was focused on improving academic writing with second language learners. Three classrooms were developed with eight lecturers involved. Traditional classrooms use passive learning and the wish was to have active learning and so the classrooms needed moveable desks, integrated technology, large screens and used a flipped approach to session with the teacher as a guide on the side. Students were encouraged to use their phones in class and the change in furniture was significant. 168 student evaluations were undertaken and they liked the active learning rooms with the moveable furniture, personal device use, screens and whiteboard. The tutors found that not having a focal point in the room was a challenge and multiple distractions being overwhelmed with technology. The students saw the challenges as the lack of a focal tutor point, the small groups and flipped learning. However the classes used workshops approaches with writing in the session so peer and tutor feedback could be given. This was seen as lower stakes and the students benefitted from reduced plagiarism as well as feedback. 85% students wanted more classes.
These two papers were very good but the others were very focused on specific lab spaces.
Tutoring and Coaching Parallel Session
There were five parallel papers I this session. Caroline Brandt presented first on integrating peer tutoring and interdependent learning to enhance academic learning support provision. Caroline is from an engineering university and most of the students are from the United Emirates. The University works closely with the national oil company. Caroline works with communication courses which are student centred and contain literature reviews as one of the assessments. There is a writing centre set up to support students and peer tutors run this experience. Caroline collected data from 11 conversations between peers and these were audio-recorded. In addition data was collected from observing 15 peer tutoring consultations and 4 semi-structured interviews. These were then analysed using linguistics looking at how many words were spoken by each participant. The findings demonstrate that whilst students do consult on a one to one basis the peer spoke more than the student seeking support. Many of the students entered straight from school and so are not use to engaging in group type activity and so Caroline is exploring how students would be better to work in pairs or threes so they develop some independence.
I presented the paper on the personal tutor evaluation undertaken here at City University London and outlined changes to the policy and the staff development support we were providing.
The next paper was about the development and implementation of an educational training programme for teaching assistants in engineering education. The speakers (I Van Hemelrjick, E Londers, M Burman, C Suttels and Y Berbers) introduced a training programme they had developed. The project started by interviewing TA’s about their experiences. They then developed objectives from this and three modules. Each module had an assessment, a face to face session, another assessment and a final face to face session. Specific materials were developed for the sessions. There were 3 major objectives but one was let TA’s experience activities they would practice with their students. This was the constructive approach and some of the types of things they did were guiding a masters’ thesis one to one sessions, coaching for PBL using the 9 coaching roles. The TA’s were very positive about this approach and liked the interaction.
The next paper was was given by Miri Shacham on a Personal academic coaching programme for enhancing student learning. This was focused on engineering students with an average age of 23 years. The drop-out rate is 35% in year one which is an international issue. It is known that personal coaching can improve resilience and student self- esteem. Academic coaching included motivation, time management, study skills etc so successful in studies. This was to promote the students learning skills and reduce attrition. The process was that students got 10 -12 coaching sessions using the Keden (2006) model which focused at the end of establishing capabilities. The findings were that using this model students did gain more self-management skills, took responsibility for their learning, has a sense of self-efficacy, provided support and positive communication to peers, exposed difficulties and had difficulties in making the change. There were both intrinsic and extrinsic factors but family had a big impact. This improved academic achievement in 25 of 37 (67.5%) students involved in the study.
The last paper in this session was about virtual or face to face tutorials: which do university students prefer? Given by Maria J Hernandez-Amoros on behalf of a team of three (M Iglesias-Martinez and I Lozano-Cabezas). The European HE area is changing and innovation is being used. There is a shift from teacher focus to students and away from being the leader and expert to the guide, supporter, creator and tutor. There was a discussion about the difficulty in defining tutor and tutorial and that there were no real limitations in terms of time and space and that some misconceptions arise through poor strategies. The actual process does not matter but the quality does. The team wanted to find out what approaches students used and so undertook a survey with 273 students. They had 8 items on the survey that used a likert scale of 1-5. Students preferred face to face tutorials for questions and used these 1-3 times a term. Virtual tutorials were however used more often. Students tended to seek tutorial support when they needed professional guidance or had academic doubts.
There was a good mix of papers with some issues being raised across the studies.
Gina talked about the forthcoming Teaching Excellence Framework and the promise to place teaching excellence into a comparable place with that of research excellence. There was a discussion of the structures that might be used, strategies and infrastructure. SOTL should provide an evidence base for the teaching excellence and so we need to be clear about this is. As educational developers we should provide our institutions with a definition that should then be used as a basis across the institution. There needs to be development around SOTL outputs to also have SOTL impacts. Teaching excellence has no agreed definition and so we need to work towards what would be an agreed view. The TEF needs to include qualitative measures as well as metrics and peer review and CPD activity are important elements of this. We need to support our colleagues in sharing their good practice from awards and projects and promote dissemination of this through high quality SOTL.
The session aimed to introduce participants to an innovative approach which enables exploration of critical incidents and is designed to maximise the opportunity for learning and development. The origins of poetic transcription sit in research and was developed by Prendergast (2006). This is a new way of working with interview data. However Fiona had developed this further so rather than privilege the researcher’s interpretation this approach uses a group and is dependent upon mutual trust and respect. Fiona had tested this using two case studies one focused on early career academics and one on students in transition. There were five individuals who participated with two facilitators and there had been three workshops. Data was gathered through field notes and a questionnaire. Participants were asked to provide a critical incident and these were then one by one given to the group to interpret which was done though all reading the incident and then reducing this to a poem by using only the words in the incident and no words could be added. The incident did have to be reduced to highlight key thoughts. This was undertaken with each incident.
In the workshop there were about 12 of us and we each had the same incident and had to undertake this task. It was really interesting and then we shared the poems and our thoughts about the incident. Whilst all the poems were different some same key terms and messages came out and we all had shared sense of the incident and how it felt. The participants in the original activity found this approach was important to gain a shared sense and this was through engaging with the text.
Dilly started by focusing on how sometimes language can be a barrier and SOTL is seen by some to have boundaries. Boyer’s (1990) 4 elements have been integrated by many over the years and are familiar as are some of the definitions that Dilly shared in the session. The term education is used by many education leaders rather than teaching or learning because it does reflect the breadth of the educational role. Scholarship can then be linked more easily to research. There is also a need to distinguish between teaching excellence and impact on education. Dilly discussed some theoretical framing and how Bildung’s concept of self-cultivation was useful as we develop both our thoughts around education and philosophy. We develop through authentic dialogue which leads to a collective journey. To continue our journeys “the human mind needs to remain “unsatisfied” with what it imagines it knows (Fairfield 2010:3).
Dilly then focused on the connected curriculum developed at UCL and that this was about learning through research and inquiry. Students connect with staff and their world-leading research. There is a line of research activity built into each programme and students make connections across subjects and out into the world. Students connect academic learning with workplace learning. Students learn to produce outputs directed at specific audiences and they connect with one another and alumni. Expressions of scholarship can be rich and varied and can include:
- formal written argument
- numeric analysis
- presentations
- dialogue and knowledge exchange
- graphical representation
- visual modes of expression
- creative modes such as fine art, film and music
- creative writing…
Our scholarship must allow other cultures to influence our work and we need to look at how we can include other knowledge. We need to play to individual strengths, build on values and passions and be intellectually curious. We need to find ways for others voices to come through and use authentic modes of enquiry and expression. We need to respect diverse modes of expression and so reviewing assessments is a positive action.
We should review role descriptions to ensure education focused scholarship is a desirable activity and reward scholars for their work.
This paper discussed some research undertaken in a university where there were 27,000 students and 1,000 staff with multiple campus sites. The key objective was to increase the proportion of staff seeking internal recognition of effective learning and teaching practice through alignment to the UKPSF and provide support.
In 2014 there had been some HEA recognition targets introduced that the Centre for HE Research and Practice (CHERP) were to support. These were 75% staff to gain recognition. The study aimed to:
- explore the relationship between patterns of engagement with SOTL and the culture around the value and status of L and T at school level
- to make use of the results to inform and enhance subsequent provision of and support for engagement in SOTL
The data sets used for this included staff engagement with various activities and interviews. Interviews were undertaken with 6 faculty UKPSF leads, and then an online survey was undertaken with 250 staff to gain insight into academic staff experiences, perceptions of SOTL and pedagogic research. 25 staff also agreed to be interviewed and then conference attendance data and journal submissions were examined. The award and HEA recognition data was also examined.
From the online survey there was a response of 65 staff (25%) and this covered 29 of the 34 schools. From this the data showed that SOTL was valued in schools by 42% who agreed but 20% disagreed. There was contrasting views across schools. In response to my school encouraging or supporting conversations about Learning and teaching 49% agreed but 20% disagreed.
The conclusions from the evidence suggests that compliance has seen a marked growth of engagement with the natural driver being the target. There is the greater potential in schools where learning and teaching are valued. There needs to be a more consistent culture across the institution. In future the promotion route for academics will include a teaching and learning route and to gain SL promotion you will need to have SFHEA.
This presentation was focused on teaching informed research. The work of Mazur had been explored by Bates and Galloway and they were looking at learning gain. They found there was a positive association between attainment and confidence in performance. Students seem to recognise the power of peer instruction. Undertook some similar work and then shared results with students after class through an e-mail so they could see where they were in relation to the class average. The clickers required the students to own their answers. There was ethical approval for the project. In class students would provide their own answer and then in groups discuss the answer and some students would try to persuade others that their answer was right. The students would then all vote again and some students saw they had gained learning and changed their answer from the discussion. They ran focus groups with the students and students found this useful. Students who usually had low attainment liked the clickers and discussion. There is a need to undertake further work around this but they found students did behave differently when involved in this peer discussion.
