Busting REF myths
3 September 2018
We’ve heard many REF myths in our time, and with the publication in July of the draft guidance and criteria, we’ve been keeping our ears to the ground to stay alert for any that are gathering particular momentum. Myths range from the bizarre to plain misunderstanding – some favourites we’ve come across during this exercise include that the REF team is huge, and we have a call centre of staff to deal with queries (it isn’t, and we don’t!), that impacts on policy can’t be submitted (they certainly can), and that the panels use journal impact factors to assess outputs (the criteria underline more than once that they won’t). Despite our best efforts to dispel many of these points in the guidance, some myths are particularly enduring. So we’d like to focus here in a bit more detail on three of the more persistent ones we’ve encountered…
‘Only journal articles can be submitted’
Equity is an underpinning principle of the REF, and means that the guidance and criteria enable recognition of excellence across all types of research and forms of output, and their assessment on a fair and equal basis. We have provided a glossary defining the full range of output categories for REF 2021, and the criteria make clear that the panels welcome all forms of output meeting the definition of research. Excellence was found across all output types submitted in REF 2014. Feedback from that exercise does suggest there was some uncertainty with the submission requirements for some types of practice research outputs. With this in mind, main panel D (where the largest proportion of these outputs were returned) have set out more guidance on what should be submitted this time round. This is intended to ensure the panel has access to the research dimensions of the output and give confidence to institutions to present their best research in whatever form it has been produced. Views on this are welcomed during the consultation.
‘The discipline-based UOA structure means that interdisciplinary research will be disadvantaged’
There is little evidence that interdisciplinary research was disadvantaged in REF 2014. In fact, analysis of the outcomes indicated that outputs identified by institutions as interdisciplinary were found to be of equal quality to other outputs. However, there is a concern that institutions did not feel confident submitting interdisciplinary research. Working with our Interdisciplinary Research Advisory Panel (IDAP), we have introduced a number of measures intended to reassure institutions that interdisciplinary outputs will continue to be assessed robustly and fairly. At the heart of this is a network of expert interdisciplinary advisers, who will provide guidance to their own sub-panels and liaise with the advisers on other panels to ensure the equitable assessment of interdisciplinary outputs.
‘You can’t have a high-scoring impact case study based on public engagement (PE)’
This is simply not true. A 2017 report by the NCCPE did not find any significant difference in the scores awarded to case studies featuring public engagement in 2014. However, we know that institutions can be nervous about submitting PE-based case studies, which are perceived to be ‘risky’ and difficult to evidence. There is often confusion about where dissemination ends and impact begins – does being on the telly ‘count’ as impact? Engagement can be an important pathway to impact and can play a vital role in creating a compelling narrative. Participation or viewing figures, for example, can provide valuable evidence of the reach of the impact. But the significance of the types of impact most commonly associated with public engagement – impacts on understanding, participation and awareness – can be tricky to evidence. The REF panels are clear that they welcome all types of impact and have included an extensive (but not exhaustive) list of impacts and indicators in the ‘Panel criteria’, including plenty of examples of public engagement!
The changes to REF 2021 following the Stern review are likely to bring with them a new set of myths – in fact, we’d already like to make clear that we’re not expecting to see any half-outputs submitted: rounding should be applied to give a whole number of outputs for return! What we would encourage researchers to do, if feeling bemused, baffled or incensed by any REF myths coming through the grapevine, is to reassure themselves with a rummage through the guidance on our website www.ref.ac.uk, or through a word with their institutional REF contact.