RGU dashboards use colour coding. One feature of this is to show, immediately, how module performance compares with institutional thresholds. Year on Year trend data shows improvement or decline, using coloured bubbles to present the information in a way that is prominent. This offers a high level of insight before detailed consideration of the data is undertaken. Another key feature is that the performance of groups, including certain key protected characteristics, is clearly presented. The dashboards, which also incorporate tables and graphs, are interactive, allowing users to filter the data to support their evaluation for Annual Monitoring Reporting.

Anonymised dashboard

St Andrews has developed a ‘tartan rug’ system to categorise student feedback gathered through module evaluation. The rug shows student satisfaction for module evaluation questions benchmarked against the University mean, using a three-colour system. The colours do not point to an issue, but where satisfaction falls below the University mean, it serves as a trigger for discussion and intervention. The University has recently developed the rug in response to user feedback to include the number of respondents as a percentage of the module cohort. The Academic Monitoring Group considers the rug and provides feedback to the Schools. This includes requests for further information on high performing modules (where good practice can be shared), as well as areas for investigation. Schools are required to reflect within the Annual Monitoring Reporting process on their engagement with the rug.

The University introduced online programme level Annual Monitoring Report (AMR) in 2018. The online template takes the form of a SharePoint document, which is personalised with links to relevant data on:

• student success
• student satisfaction
• student destinations
• programme health.

All programme leaders have access to high level data and the option to view more detailed information, including breakdown by protected characteristics. Programme leaders also draw on module reports, which include detailed information on pass rate and progression. Whilst these reports are helpful, it can be challenging to interpret data for larger modules, which span multiple programmes. Programme leaders need to be mindful of the place of each module within their curriculum, for example whether it is core or optional. An important advantage of the online system is that it is more resource efficient for data users. It also improves oversight, reporting and sharing of practice across UWS on many levels. Some programme leaders use the system as a repository for other data relevant to AMR, as there is the facility to upload appendices. Reports are not currently provided to students through this system. However, the technology is in place to facilitate student access.

Colleges at Edinburgh provide benchmarked data to support Schools’ preparations for Annual Monitoring Report. The data includes information on performance within the Schools benchmarked against the College. Each College may implement benchmarking to suit the local context, so could relate to retention and progression, but this will depend on interpretation of ‘performance’.

Several years ago, award outcomes for the BEd were noted to be lower for students with disabilities. Through their engagement with the data and team discussion, staff identified that there might be a need for additional support. As the numbers concerned were very small, it was determined that a blanket intervention would not be appropriate or helpful. Instead, staff continued to support students on an individual basis around approaches that might help those with disabilities achieve their full potential. Award outcomes have improved to the point where there is no difference in outcomes for this group since the analysis of the data was undertaken. While it is difficult to link this improvement directly to Annual Monitoring Reporting, the engagement with the data through that process undoubtedly heightened awareness of challenges and the importance of early action to support students.

The Conservatoire has a student population of around 1,200 degree students. This means that datasets for progression and retention are small, especially when broken down by student characteristics. The analysis undertaken for Annual Monitoring Reporting has proved valuable when working with small numbers. It has been possible to determine statistical significance, which gives credibility to the process and helps target activity. The Analyst has oversight of a much broader suite of information, within which data is broken down in considerable detail. To give one illustrative example, analysis has been undertaken by gender and musical instrument, for example female cellist. Analysing data to this level helps build up a rich bank of information. Staff can use this data to benchmark their own programme data and test hypotheses with support from the Analyst.

Annual Monitoring Reporting (AMR) supports institution-level planning. The University Committee for Learning and Teaching and the University Committee for Research and Innovation consider this through the AMR Summary Report. The report informs the Committee’s forward agenda for strategic developments in learning and teaching, and research. The links with the Scottish Funding Council Outcome Agreement through our SFC Quality Report, which features an AMR overview.

A 2017-18 School Annual Report found that retention rates had improved due to the implementation of a student retention initiative that had arisen from the recommendations of the previous year’s Report. However, in capturing learner analytics data for the Programme Annual report, it identified that the withdrawal rate remained consistent for the Term 2 re-sits for some programmes. This was against the overall upward trend. In response, the School and Student Services team prioritised this area to monitor engagement and put support in place for students identified as at risk.

A School investigated a disappointing average pass rate for one Division.  This was comparatively low at the main diet in the previous year (against the University average) but recovered after the re-sit diet. However, in the programme-level analysis of the data in 2017-18, it was found that there were areas of both over-performance and under-performance. This was reflected upon in the Divisional Annual Reports and measures were proposed to support teaching staff and improve student performance within the underperforming areas.

This example of practice complements Annual Monitoring Reporting.

The University has a cross-Institutional retention task force who deal with retention and progression with representation from all Schools, Registry, Academic services, IT, Student Recruitment and Admissions. In the last year the task force have reviewed correspondence being sent to students who have become disengaged from their studies. The aim is to provide more support and to be clearer as to what students can do to address their situation. Good practice in supporting students and developing procedures for “early interventions” has been shared between the group. The group has also championed attendance monitoring at lectures with a view to this providing an early warning of disengagement. They are also involved in developing a learning analytics policy to support the future use of learning analytics to support disengaged students.

This example of practice complements Annual Monitoring Reporting.

The University is embedding learning analytics activity into its oversight of retention and progression. Examples include:

• integrating learning analytics into online course design
  
• working with programme leaders to design student engagement dashboards
  
• developing a student assessment and feedback planning tool.

This example of practice complements Annual Monitoring Reporting.

Annual Monitoring Reporting (AMR) dialogue meetings are held annually in October. The meetings are conducted by video-conference to facilitate participation from each of SRUC’s six campus sites. The video-conferences are held over two days, during which there are eight cross-campus meetings, grouped by cognate subject areas. Following the dialogue meetings, a synopsis of key themes is provided to the Learning and Teaching Committee.

The meetings take the form of a peer scrutiny event. A review panel with around eight participants is established to develop questions in advance, for the participating subject areas. Typical attendance at each of the meetings is between 15 and 20 participants. Review panel membership includes academic representation, staff from the SRUC quality and learning engagement teams, and student representation (usually the Student Association Vice President). Students from participating subject areas currently contribute through their involvement in quality processes that underpin AMR. Going forward, SRUC is considering student involvement as members of the participating subject area group. The involvement of external reviewers is also under consideration.

The meetings provide an important structured opportunity for staff and students from the six campus sites to reflect on the AMR outcomes. This ties in well with the continuous development of quality procedures as vehicles for reflection and enhancement at SRUC. Topics covered at the meetings are wide ranging and include discussion of progression and retention initiatives informed by evaluation of relevant data.

During the 2016-17 annual dialogue meetings, the impact of poor mental health on the retention and attainment of students was discussed. This resulted in an ongoing initiative providing training for over 100 staff and students in mental health first aid and the introduction of Therapet Days at each campus. Some staff dogs are now Therapet accredited and able to provide support on a weekly basis. Further to this, SRUC and SRUCSA in partnership have accessed and embraced the SANE Black Dog campaign, which provides a confidential out of hours support line for students (and staff with concerns about students), with analytics provided to SRUC to help shape future wellbeing initiatives. This is being used to inform the SRUC Health and Wellbeing Strategy for both staff and students.

At UWS students take part in the annual institution-level Enhancement and Monitoring (EAM) Event. Annual Monitoring Reporting is the focus within the EAM, which also provides opportunity for updates on key institutional developments through a five-minute thesis approach. AMR discussions are conducted in a world-café format. Typically, the Assistant Deans move around the tables to present key highlights and challenges identified through AMR for their School. Then discussion at the tables informs future enhancement. Conversations can be wide-ranging but are likely to include consideration of progression and retention. Key themes are captured in the EAM Newsletter, which is provided to all staff and students.

Dundee University Students’ Association (DUSA) and the University of Dundee have been working in partnership to explore the possibility of a data sharing agreement to support evidence-based enhancement. This activity is progressing within the wider context of Dundee’s Student Partnership Agreement (SPA). The SPA is used to identify the development of student welfare and pastoral support as a key priority for 2018-19. Related performance indicators within the SPA include:

• swipe card entry to DUSA
• uptake of support and advice from DUSA
• progression and retention statistics broken down by student groups.

The agreement would provide for the sharing of selected data that is currently held separately by the University and DUSA. Under the proposals, University data would include aggregated attendance, retention, progression and attainment statistics. Data from DUSA would include results from the Student Matters Survey and Student-led Teaching Awards. The sharing of this data would enable users to triangulate information on progression and retention from an extended evidence base. The Student Matters Survey provides a rich source of qualitative information on factors affecting the student experience, including financial and housing considerations. If adopted, the data-sharing agreement could be used to good effect within Annual Monitoring Reporting (AMR). A further benefit would be increased student awareness of the AMR process and student contribution to preparing AMR reports.

Edinburgh Napier’s online staff newsletter (The Bones) included, for the first time in 2019, a summary of key themes emerging from Annual Monitoring Reporting (AMR). The newsletter presents information in a format that is engaging, accessible and informative. The AMR article explains to staff how AMR submissions are used for internal and external (ELIR) purposes. Examples of good practice are identified, as well as some areas where further work might be needed. The article reflects not just on the themes emerging from AMR, but also the process. This reflection includes identification of challenges of working with AMR data. Within the article staff are encouraged to continue AMR discussions through formal and informal channels. It is also emphasised to readers that the AMR process is ‘at the heart of the University’s enhancement culture’. The Bones is available to all staff and external audiences, on the University’s website. This means there is potential to stimulate positive engagement with AMR, including discussion around the use of data, across a wider readership than the immediate audience for reports.

An annual dissemination event is held in October with the primary aim of promoting and sharing good practice. The Academic Monitoring Group draws up a list of around 12 potential topics for the event, based on scrutiny of School Annual Monitoring Reporting submissions. Directors of Teaching subsequently vote to narrow the list to five topics. Attendance at the event includes Deans of Teaching plus one other colleague from each of St Andrews’ 18 Schools. School Student Presidents are also invited to attend. The event takes the form of a series of five-minute presentations, after which the presenters facilitate group discussions. A brief report from the event is also prepared for the Learning and Teaching Committee, providing opportunity to share practice with a wider audience. Topics at the meetings are wide ranging and can relate to progression and retention initiatives. For example, there has been discussion around lecture attendance and the benefits for students of participating in timetabled learning experiences. There is good evidence of cross-School transfer of practice following dissemination events. One such enhancement being the adoption, in one School, of graduate exit interviews.

The inclusion of a ‘Question of the Year’ in the report has proven to be an effective and valuable way to gather information on, and respond to, topical issues and/or areas for enhancement. Responses to the most recent question, related to the Careers Link role in academic Schools, were collated and shared with the Careers Centre. They will use this data to:

• structure training for the Careers Links
  
• identify areas of good practice and areas of concern
  
• review against St Andrew’s DLHE data to identify schools whose students need more in-depth support.

The 2019 iteration of the form includes a question on staff and student mental health and wellbeing. Responses will be shared with the University’s Mental Health Strategy Group.

RGU has an 80% threshold for first diet module pass rate. A red cross on the dashboard denotes thresholds that have not been achieved. In cases like this an evaluative commentary is required within the Annual Monitoring Report. Importantly, a red cross does not denote a significant issue with delivery or other aspect of the module under consideration. Instead, it serves as a trigger for further interrogation of the data, and (where appropriate) identification of interventions to support student achievement.