Evolution not Revolution: Public Engagement and the Future Research Assessment System

25 / 05 / 2022

The NCCPE response to the Future Research Assessment Programme (FRAP) Consultation asks for Evolution not Revolution - seeing value in continuing to run the REF, but arguing for a number of significant enhancements that support a fairer, more equitable and more productive research assessment system in the UK.

What is the FRAP?

As part of the Future Research Assessment Programme (FRAP), the four UK funding bodies (Research England, Scottish Funding Council, Higher Education Funding Council for Wales, Department for the Economy, Northern Ireland) are seeking views on the design of the UK’s future research assessment system. 

They launched a consultation in Spring 2022 to gather evidence, asking for views on:

• The purposes of a future exercise
• The principles that should guide its development
• The assessment criteria and processes

The NCCPE has worked with colleagues and institutions from across the sector, to consider how best to embed considerations about public engagement into the design of future research assessments.

Following a workshop in April, and extensive consultation, we have now published our response. 

Evolution not Revolution

We argue for evolution not revolution. The FRAP consultation invited ‘blue skies’ thinking about radical alternatives to the Research Excellence Framework (REF), but on balance we see value in continuing to run the REF, with a number of significant enhancements:

• We would like to see an evolution in the framing of ‘impact’: the term is a blunt instrument to describe the complex process of realising benefits from research and innovation. We recommend a change to ‘Public Benefit’ or ‘Social Benefit’.
• We argue a case for an additional criterion for assessing impact case studies, to mirror the inclusion of ‘rigour’ in the assessment of research outputs. We think it is important to focus attention on the professional care taken to realise impact, in particular on how Equality, Diversity and Inclusion (EDI) and ethics are taken into account, and how the approach aligns with good practice principles of engagement. We suggest that the criterion ‘sensitivity’ might work well to convey the expectation.
• We suggest that a similar device to the 2014 impact template should be reinstated. For instance, a narrative account of the underpinning approach to support impact, building on lessons learned from the use of narratives in the Public and Community Engagement in the KEF. This would contextualize the submitted impact case studies within the Unit’s wider impact and engagement strategy and operations. 
• We highlight significant issues with how the REF treats EDI in relation to the assessment of impact, and argue that major changes are required to address this, in line with the Public Sector Equality Duty which puts the onus on organisations to positively promote equality, not just avoid discrimination.   
• We introduce a ‘thought experiment’ to consider the potential benefits of decoupling the assessment of outputs and impact, and perhaps sequencing these exercises so that they don’t happen simultaneously. If this were implemented, the assessment of impact could be delivered through a ‘lighter touch’ approach, with perhaps four broad discipline-linked panels and a fifth interdisciplinary panel, each chaired by an impact specialist. 
• We argue that evaluation of the research environment should take more account of the very significant place-based and institutional disparities which are currently hidden from view in the REF’s assessment approach. There isn’t a ‘level playing field’ in terms of the investment and capabilities that different higher education institutions (HEIs) can draw upon to invest in impact-generating activity. We also argue for the inclusion of a separate, data-driven ‘Context’ section to support fairer judgements about what institutions have achieved with the resources at their disposal.
• We question the ongoing value of the existing criteria of ‘vitality’ and ‘sustainability’ for assessing the research environment. We suggest that the enabling principles described in UKRI’s new strategic framework provide more robust and productive focal points, targeting Diversity, Connectivity, Resilience and Engagement. This would have the added benefit of encouraging common purpose and joining up thinking across the Research and Investment system.

If the sophistication and rigour with which the REF assesses impact is enhanced as we have outlined, then we would like to see the weighting for the impact portfolio be raised from 25% to 30%, and Environment from 15% to 20% (and outputs reduced to 50%).

This would recognise the importance of valuing the public benefit achieved through research, and the importance of accelerating enhancements to research culture. This change will focus attention on the vital need for Higher Education Institutions to invest in capability to support impact, including through their investment in public engagement expertise. 

Thanks to everyone who contributed to our response. We’ll continue to engage with the FRAP programme, which is expected to conclude at the end of 2022.