REF 2021 guidance published: what’s the public engagement angle?

The consultation on REF 2021 is over and the final guidance has been published – NCCPE Director Paul Manners explains what has changed and what this means for public engagement.

It was with bated breath that we opened the PDFs of the final guidance for the REF 2021 to see what had changed. The NCCPE had put a lot of work into our response to the consultation, with the help of many people across the sector, and were keen to see what difference this had made.

Therefore it was initially disappointing to see that very little had changed from the draft published back in July 2018.  Sadly our work had little impact on this occasion! 

The NCCPE’s consultation response had identified some critical areas pertinent to public engagement, areas where we thought the guidance could be improved.

Equality and diversity

We argued that the guidance should highlight the importance of considering equality and diversity in relation to impact, not just internal HEI staffing policies. We argued for a clear expectation being set by the panels that equality and diversity be addressed in impact planning (in the ‘environment’), and, where relevant, within impact case studies. We recommended that the 2010 Equality Act and the Public Duty regarding socio-economic inequalities were referenced to provide an appropriate framing of this expectation.

Whilst this hasn’t happened in the way we had hoped, there is a small but significant change in the Environment template, which now includes an additional prompt in Section 4 (Collaboration and Contribution to the Research Base, Economy and Society) inviting information about:

'How the unit engages with diverse communities and publics through its research' (Panel Criteria, paragraph 358)

We were also pleased to see that the Panel Criteria and Working Methods have slightly refined their guidance about impacts arising from public engagement.  Their definition of ‘reach’ now encompasses demography – not just numbers of participants. The additional phrase is highlighted below:

'Panels expect that case studies based on public engagement will demonstrate both reach (for example, through audience or participant figures and demographics) and significance…' (Panel Criteria, paragraph 299)

Both are a step in the right direction, and we shouldn't underestimate how important even small changes in wording can be.

Areas and indicators of impact

We also suggested various ways in which guidance about public engagement could be made more explicit and coherent. We recommended some changes to the framing of the areas and indicators of impact (Annex A of the Draft Criteria and Working Methods) to ensure that impacts arising from public engagement are considered more holistically.  The draft table saw outcomes linked to public engagement largely parked in just one area – ‘Impacts on Understanding, Learning and Participation’.  We argued that public engagement can provide a pathway to impact in all of the other impact areas – for instance by enabling public perspectives to influence policy making and professional practice, or through patient involvement in health. However the table remains virtually the same, and we are concerned that this sends a message that public engagement is all about influencing public understanding – and stops people thinking about other kinds of outcomes it can realise, for instance influencing social structures and social practices.

However, always on the lookout for positives, we were pleased to see the Impact Area ‘Impacts on practitioners and professional services’ reframed as ‘impacts on practitioners and delivery of professional services, enhanced performance and ethical practice’.  We think that this more expansive framing could encourage approaches which actively engage publics as service users and stakeholders.

We were also pleased to see the Guidance on submissions emphasises the value of co-produced research in paragraph 319 c:

'The relationship between research and impact can be indirect or non-linear (for example, co-produced research)'.

Evaluation

Lastly, we did push hard for greater emphasis on the vital role of evaluation in our responseWe recommended that the use of robust evaluation was signalled more explicitly.  That hasn’t happened.  However, there is a welcome softening in Panel A’s position regarding qualitative data and a strong, Panel-wide endorsement of the value of qualitative evidence:

309. The panels anticipate that impact case studies will refer to a wide range of types of evidence, including qualitative, quantitative and tangible or material evidence, as appropriate. Individual case studies may draw on a variety of forms of evidence and indicators. [] A diversity of evidence is welcome, and no type of evidence is inherently preferred over another (Panel Criteria and Working Methods)

There is also clarification of how the panels will approach the inclusion of testimonials:

310. Testimonials should draw on statements of fact and relate specifically to the impact(s) claimed. There may be occasions where opinion-based testimonials are appropriate (for example, where the impact is on public understanding of an issue). 

The right direction of travel

Overall, we are really pleased with the direction of travel from 2014 to 2021.  The guidance lays a solid foundation to encourage people to feature impacts arising from public engagement.  We look forward to working with the sector to seize this opportunity.

Comments

I think these are small but nevertheless significant wins, Paul. And my team will certainly be adding them to our armoury of persuasion and influencing tools. Thank you for your part in helping win them!

Great blog Paul! I agree that it is disappointing that there wasn’t more done to encourage or better evaluate public engagement, but your efforts in the consultation were admirable and have achieved modest but important gains. As you say, it is a move in the right direction!

The guidance does seem to relax the criteria for underpinning research, potentially leaving wiggle room for case studies to draw on much broader bodies of work than the six outputs they can list, presumably describing this wider work in the accompanying underpinning research section. I don’t think it helps the case for public engagement that draws extensively on research beyond the submitting unit, as it is clear that the outputs and wider work must be by the submitting unit. I am regularly asked if I think public engagement case studies that are only tenuously linked to research in the submitting unit will be okay, on the basis that this seems to have been acceptable in the Brian Cox case. My answer is to cite the “Brian Cox effect” that was coined in the case study – just because it worked for Brian Cox doesn’t mean it will work for you. As far as I can see, there’s no major shift (certainly of the scale proposed by Lord Stern) and so cases like this remain high risk (unless your name is Brian).

You can read more of my thoughts on this on my blog here: https://www.fasttrackimpact.com/single-post/2019/01/31/Everything-you-ne...

I agree entirely with Laura. Every little helps, I think particularly changing Panel A's endorsement of testimonials/qualitative evidence should be seen as a major win! Well done!
What still puzzles me is the -in my view- discrepancy between message (impact is important) and rules (REF criteria etc, funding decisions, etc.). Why is there such hesitation to align the rules to the message?

Add new comment

CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.