As part of its policy work, AMOSSHE hosts a series of strategy and policy discussions, called AMOSSHE Futures. This AMOSSHE Futures policy breakfast about Student Services and the Teaching Excellence Framework (TEF) took place from 10:00 to 11:30 on Tuesday 23 May 2017 at Woburn House in London.
The discussion group explored the implications of the Teaching Excellence Framework (TEF) for Student Services in higher education. AMOSSHE member research shows that student engagement, retention, and employability are part of the Student Services remit of responsibilities in over half of publicly funded universities in the UK, and aspects of teaching such as study skills, language support and accessible teaching are also commonly part of the Student Services remit. The crossover of these areas with TEF metrics prompted the discussion about how the implementation of TEF will impact the work of Student Services departments, and how Student Services can contribute to TEF in the most effective way.
Nicola Barden (AMOSSHE Executive Member and Director of Student Services, University of Winchester) chaired the discussion. Participants from AMOSSHE member and other sector organisations included representatives from the following organisations:
Anglia Ruskin University
GuildHE
Higher Education Academy
Universities UK
University of Cambridge
University of Derby
University of East Anglia
University of Sussex
Wonkhe
The remit of Student Services responsibilities represented by AMOSSHE members in the discussion included the ‘traditional’ areas such as wellbeing, mental health, student policy, disabled and international student support, and study skills. Other remit areas represented in the room included student engagement and retention, careers and employability, OFFA (Office for Fair Access) access agreements, student discipline, learning and teaching, and accommodation / residences.
Everyone taking part in the discussion had engaged with the progress of TEF to some extent. Wonkhe has published over a hundred articles discussing the merits, limitations and implementation of the framework. Policy experts at Universities UK and GuildHE have worked with the Department for Education and their member institutions to understand TEF and its relevance to different kinds of institutions, update members and influence its future development. The Higher Education Academy has also contributed to the development of TEF, including as part of the Department for Education working group, and is involved in work on the design of the academic discipline-level version of TEF.
At the time of the discussion, with the first TEF results due for publication the following month, Student Services leaders in the room had already engaged with TEF to differing extents. While some had written the TEF submission for the whole institution (in collaboration), others had contributed data, and/or taken part in working groups with greater or more distant influence on the final submission. Others had little or no involvement. Universities UK surveyed its members to identify which roles were involved in the TEF submission, and found that at least half of the roles were within Student Services, or allied through roles such as Pro Vice Chancellor Student Experience or similar.
Anecdotally, some institutional decisions about who should take on responsibility for TEF may have been less thoroughly considered than they might have been, due to the relatively short time in which TEF was introduced, and the sparse, and changing, guidance accompanying it. The group discussed how differences in the extent of Student Services involvement could be reflections of how different institutions view TEF altogether and its potential impact, and institutional attitudes towards student attainment.
Institutions might view TEF as an academic project, in which academics only should be involved (because it concerns teaching), or otherwise a holistic storytelling exercise about the institution’s mission in terms of student experience. Some institutions have seen TEF as an exercise in quality assurance, and others have seen an opportunity to emphasise areas such as student satisfaction or employability in which the institution is already doing well. Some institutions have focused on getting the desired ‘marks’ in the exercise without consideration of gathering a full picture of teaching excellence and its contributory factors.
Accordingly, responsibility for TEF has fallen to planning and strategy teams at some providers, but communications and marketing at others. This has led to some Student Services departments (and other professional areas) potentially being excluded from making a contribution, and/or unsure of how they could contribute. However, where student experience / attainment is key for the institution (perhaps because a low TEF result would have a very negative impact) Student Services’ participation seems to have been more actively invited in shaping the narrative.
Looking at some of the contributions Student Services have made to institutional TEF submissions in terms of metrics, examples included data related to student outcomes and learning gain (such as co- and extra-curricular activities), as well as student feedback and surveys – including the National Student Survey (NSS) – work placements and employability, and learner analytics.
The extent to which Student Services have been able to make narrative contributions has depended on the institution’s overall attitude towards Student Services, and who is coordinating the submission. Some institutions used the Student Services metrics data but were less interested in the Student Services narrative or context, which may have resulted in a loss to the quality of the submission. Some institutions may have considered that a narrative contribution from Student Services would be too ‘fuzzy’ and not punchy enough to influence the TEF panellists, or to influence submission authors within the institution.
Where Student Services is centrally embedded in the institutional structure, and considered core to the mission, there is more opportunity to contribute to and shape the narrative, and also more value is given to the voice / perspective of Student Services as well as the data.
With Student Services potentially contributing to both metrics and narrative TEF submissions, the group considered the relative weights that these might have in terms of awarding the TEF categories of gold, silver or bronze. The Department for Education guidance on this is not entirely clear because the rules changed part way through the process, and were not communicated as well as they might be.
The metrics data is used to benchmark institutions against similar student intakes. If an institution is two standard deviations from its benchmarked score (the mean expectation for what a particular group of students should have achieved), it receives a positive or negative ‘flag’. If the institution is three standard deviations away from the benchmark score, it receives a ‘double’ positive or negative flag. The evaluation combines these flags with a narrative submission from the institution. One purpose (among others) of the narrative submission is to provide more information where performance on the core and split metrics is unclear.
So, will the metrics have more weight than the narratives, or vice versa?
The group did not agree on this. The metrics on their own might provide a dispassionate, fairly objective picture of the relative hierarchy of higher education providers in terms of those metrics; but the review panels, taking into account the narrative submissions, might simply reinforce existing / traditional sector hierarchies and expectations.
On the other hand, the narrative submission is the opportunity for the institution to tell its story in borderline cases, and to account for metrics factors that institutions cannot control (for example, the influence of geography on employability outcomes). Nevertheless, an institution with double negative flags could not attain a gold award, no matter how good the narrative story.
The group concluded that while metrics trump narrative in an absolute sense, the influence of the narrative should not be discounted, particularly where the metrics produce borderline results.
The group speculated about the influence of politics on the construction of meaning around TEF. For example, a measure of ‘highly skilled’ employment outcomes, as opposed to all employment outcomes, might be designed to favour some kinds of institution (with vocational subject specialisms like medicine, or which benefit from reputational / social advantage such as the Russell Group). Alternatively, political desire to use TEF to shake up existing higher education hierarchies may have influenced the benchmarking exercise to ensure that there is, or appears to be, more equality of provision across the sector, to the detriment of traditionally elite establishments.
The difficulties of understanding how TEF awards will reflect the data and narratives submitted led to the question of whether any of this relates to teaching excellence at all. The general consensus was that in its present, initial iteration, TEF is more focused on the excellence of the student experience and graduate outcomes than on teaching. However, the development of subject-level TEF ought to be more relevant to teaching, and draw attention away from the performance of the institution as a whole. Teaching excellence information at subject level ought to provide prospective students with the sort of detailed choice (and therefore promote meaningful competition) that TEF was introduced to encourage. However, unless alternative metrics are also found, or more weight is given to the narrative, the assessment will remain more about student satisfaction than evidenced teaching quality.
Subject-level TEF pilot projects are due to begin this year, and the Department for Education is consulting with the sector about what this iteration of TEF should be like. Some felt that the road to subject-level TEF is a difficult one: an OFSTED-like approach that is potentially expensive, time consuming, and lacking in meaning depending on how granular the categorisation of particular subjects will be. Furthermore, political interest in developing subject-level TEF might dwindle, because politicians are more inclined to consider graduate outcomes and justifying the cost of higher education, than what happens in the classroom.
The group turned their attention to the opportunities that TEF offers for students, for example with regard to split metrics, which aim to establish how students from different backgrounds fare on the various TEF measures relative to their peers.
Student Services leaders in the room indicated that the advent of TEF has not brought new information about differential outcomes to light, because existing involvement with access agreements enables Student Services to understand this area already (or at least should do). However, the institutional profile of TEF means that this work is brought to the fore, highlighting the value of what Student Services are doing, and the impact on student success.
Looking at careers and employability, TEF has already led to the restructure of careers in some institutions to ensure that learning gain is acknowledged and valued.
From the institutional TEF submissions that had been published at the time of the discussion, it was clear that integrated, connected Student Services with a focus on data-driven support was key. Effective Student Services models that are visibly supporting TEF outcomes are good for both institutions and students, so this drive towards integration of services is likely to become more of a priority going forward.
The requirements of OFFA access agreements have led to good work enabling students to participate and succeed, but historically at some institutions this work was seen as more of a goodwill gesture than central to institutional strategy. TEF split metrics have raised the profile of this work, and altogether TEF is an opportunity for Student Services to make the institution sit up and take notice of the value of its work, especially for research-intensive institutions that were criticised in the initial TEF consultation for the quality of their teaching. Institutions that already have good NSS scores are more likely to have embraced TEF and do well, but unfortunately some research-intensive institutions with lower NSS scores have dedicated more energy to lobbying against TEF than implementing positive change on the back of it.
The group discussed the institutional incentives to engage with TEF. The link to fee income is a big driver for this, and although the nature of the link still needs to be decided by a future minister, the Higher Education and Research Act 2017 ensures that institutions will need to participate in TEF in coming years in order to increase fees by inflation. This can be read as a penalty for non-engagement rather than a reward for engagement; nevertheless, fees concerns will continue to drive institutional responses to TEF.
Another incentive is reputation. In a sector preoccupied with rankings and benchmarking, TEF outcomes for English institutions featuring on the HEFCE Register of Higher Education Providers will have a big impact, and the register is aimed at guiding student choice. Institutions will want to appear higher in the register with a good TEF score, having access agreements, and possibly sponsoring a school as well.
The requirements of TEF are likely to lead to increasing calls on Student Services to bolster student engagement, satisfaction, and retention through support. How can Student Services approaches respond effectively?
The discussion returned to the idea that Student Services with joined-up, coordinated models are most likely to be able to contribute effectively to TEF, and also benefit from it by raising the profile of Student Services and securing resources as a key part of the institutional TEF strategy.
Also, targeted, personalised student interventions, with a focus on students less likely to succeed, might be important for focusing resource where it will have the most impact for the institution as a whole. These sorts of interventions need to be driven by good data to understand student characteristics – and with this, perhaps Student Services should focus less resource on student groups more likely to do well anyway in terms of attainment or employment.
This represents a challenge in terms of the quality of the data, how to interpret it, and how to design effective interventions based on it. For example, attendance monitoring data can be problematic. The methods for gathering the data might be flawed (one student might ‘check in’ on behalf of several others); this data might be considered key for measuring retention, an assumption that may or may not be valid; consequent interventions based on this data and this assumption may not be effective.
There is also a challenge in terms of data gathering, with regard to student and/or staff attitudes towards it. Some might consider data tracking par for the course in a cookie-driven digital culture; others might consider the institution to be stalking or taking a Big Brother attitude.
This led the group to consider student expectations in the light of TEF. Students need to understand and engage with the requirements that higher education will make of them when TEF emphasises satisfaction, retention, and graduate employability. Providers need to communicate to students what they are getting into, and what will be expected of them.
Students choosing which institution to attend according to TEF rankings might be led to focus even more than currently on employability outcomes, because TEF will emphasise this, and recruitment marketing is likely to make the most of it too. Anxiety about debt and parental influence (about debt) will reinforce this kind of student decision making, so students may develop assumptions about ultimate salaries and expect the institution to somehow deliver on this.
Institutions need to highlight the benefits of skills development outside of the core curriculum and the value of the broader higher education experience, and teach students how to articulate what they have gained when entering the job market.
The urge to recruit students indiscriminately, prompted by the removal of the numbers cap and the limitations on fees, has the potential to cause problems for both Student Services and institutional TEF outcomes, and may not be in the interests of students for whom higher education is not the right choice. Student Services leaders are likely to find themselves in the position of arguing against keeping students on course (perhaps for the student’s own welfare), in a context where the institution is strongly opposed because TEF has underlined retention and employability as central to the mission.
How should Student Services articulate their value?
When the full set of TEF submissions are published, institutions that see TEF as a key income driver will look to amend strategy and resourcing in response. Some institutions might be inclined to take budget and resource away from Student Services to dedicate to areas with lower TEF scores, so Student Services leaders (especially those who had little input into the TEF submission) need to take part in this assessment to ensure that the value of Student Services is appreciated and protected.
It will be important to show with data how Student Services support TEF across the board, and especially through impact on retention and employability. Even at institutions where the Student Services remit does not include these metrics-rich areas, the work of Student Services still impacts these areas.
A data-driven approach may have more influence within the institution than a focus on qualitative student experience. A challenge is that there is no consistent method to calculate the effectiveness of interventions, and certainly no sector-wide method. However, the group agreed that Student Services leaders should not limit the impact of their message by concerns about accurately ‘proving’ the link between Student Services interventions and student outcomes, to account for every possible contributing factor. Other institutional departments are able to successfully argue their corner with much less reliance on hard data, and probably with much less data to draw upon to make their case than Student Services have access to. The challenge is not to offer an academically robust correlation in order to demonstrate impact and save a service from cuts; the challenge is to put forward a persuasive argument, with reference to data, which shows how Student Services can make money for the institution and enhance TEF outcomes.
To conclude, the group offered their recommendations for Student Services leaders, based on their experience and learning from the discussion. Key recommendations included the following:
Use data effectively – understand the data Student Services hold, and how that joins up with institutional decision making. Use data to articulate impact on outcomes for students and relate that to institutional TEF outcomes.
TEF puts Student Services back into the heart of the assessment framework, which creates an opportunity to raise the profile of Student Services and articulate value.
Coordinate the range of Student Services to find the best fit for the institution, because a coordinated approach will enable the creation of a consolidated, strong set of metrics to help demonstrate impact.
Develop good stories alongside metrics from evidence-based practice to create a convincing case for investment in Student Services. Show that the deficit model perception is out of date, and outline the benefits of having Student Services embedded in the teaching and learning strategy.
Speak their language – use the language of the institution when articulating the value and impact of Student Services, rather than the discourses of Student Services departments. The concerns of TEF are institution-wide, so talk to that high-level audience.
Be bold and confident about the impact that Student Services can make in terms of student outcomes and TEF ranking.
Develop your institutional community – TEF is not about academics versus services, but an institution-wide approach in which everyone works together to influence student outcomes. Take the opportunity to build relationships.
Don’t be too distracted by the current metrics – also be aware of the context of what is on the horizon. For example, the forthcoming LEO (Longitudinal Educational Outcomes) data will show, more than existing or new DLHE (Destinations of Leavers from Higher Education) data, the reality in cash terms of different subjects in different institutions, and highlight the value of the graduate premium.
Recommendations for how AMOSSHE can help members to meet the challenges of TEF included continuing professional development about how Student Services leaders can articulate value, using data effectively and the right language to influence institutional leadership. Also, how to develop good relationships within the institution to ensure that the case for Student Services is heard by the right people.
