Thinking about the future of research assessment

Thinking about the future of research assessment

4 October 2019

Today Research England has published research, carried out for us by RAND Europe, on developments in the research landscape and their potential impact on research assessment in the next 5-10 years, post-REF 2021.

The report is part of a wider package of research and analysis, focused on different areas of research assessment and the wider research environment, including two other studies that were also published this summer:

  • Technopolis Group’s ‘International Landscape Study of Research and Innovation Systems’ helps us understand how the UK system is currently positioned in the international landscape
     
  • Cardiff University and the University of Sheffield’s joint pilot study the ‘Real-Time REF Review’ provides evidence of current perceptions, opportunities and challenges in the system
     
  • The RAND Europe study published today provides intelligence on the changes we might see in the research system.

These studies have drawn on evidence from over 4,000 people including researchers at different career stages and in different research disciplines, university managers, policymakers, funding bodies, publishers, tech developers, and academics specialising in research policy. Taken together they provide a foundation on which to start building the research assessment landscape following the current REF, which will conclude in 2021.

There are some clear themes that emerge from the three reports:

  • National research assessment continues to be an important feature internationally, and researchers recognise the challenges and opportunities that this brings.
     
  • The UK is set apart by its focus on the broader benefits of research within the evaluation process, but the importance of this area is gaining increasing prominence worldwide.
     
  • The researchers who were surveyed as part of these studies see increasing engagement with external stakeholders and associated openness of research to be features that are incentivised by the current system. These features are also seen to be of increasing importance in the future, with research considered to be becoming more collaborative and interdisciplinary.
     
  • Researchers predict that they will continue to maintain a balance between expanding and enhancing knowledge and understanding alongside delivering benefit and impact. An increasing diversity of research outputs, some of them focused on audiences beyond academia, will be an important part of the mix.

Researchers paint a picture of incremental change, whereas technologists describe the potential of disruptive innovations in the processes of research assessment. And with developments in artificial intelligence and machine learning moving at an increasing pace, it is inevitable that questions will be asked on the role of these technologies in research evaluation. Simple analytical approaches, such as plagiarism detection, are already in widespread use by journals, and many are experimenting with using machines for more sophisticated tasks such as reviewer selection. How long before the suggestion that algorithms can automatically screen the methodological rigour of experimental research, or even help with overall quality judgement?

While it would be wrong to embrace such technological change without critical reflection, to ignore the potential might also miss an opportunity. It is striking that these technological developments did not feature in the views of researchers about the future.

This set of reports gives us much to think about. In the short term, we will be commissioning a second phase of the real time analysis of perceptions of REF 2021, moving beyond the pilot to a more representative sample of researchers and universities. We are also beginning to plan the broader evaluation of REF 2021, to generate evidence of the strengths and weaknesses of the existing process. I think we will also need work to understand the potential role that artificial intelligence might play in the future of research assessment, keeping in mind the key principles of robustness, fairness and transparency. This is not just a simple technological assessment, we also need to start a debate, so that researchers themselves are involved in shaping the use of technology in research assessment.

More than anything, a key conclusion from the package of work is the importance of listening to a diverse range of voices, including the huge diversity of views held by researchers themselves, across a range of disciplines and institutional contexts. As we think about the future of research assessment post REF 2021, maintaining open dialogue will be key.

Download RAND Europe’s report 'The changing research landscape and reflections on national research assessment in the future'.