Ref 2014 Impact Case Study

Impact case studies (REF3b)

The information on this page was published whilst the exercise was being conducted; this material is provided for background information only and will not be updated.

Latest FAQs (September 2013)

Can we complete the templates REF3a, 3b and REF5 without a border?

Yes. From September 2013 templates were made available with and without borders. This follows institutional feedback that the templates with borders were difficult to use when inserting tables and diagrams. The REF team have revised guidance to allow the template to be completed without borders, if preferred. The section headings provided within the 'no borders' templates should not be changed. Changes should not be made to the margins, headers or footers provided in the templates.

(The templates are available within the REF Submission system and on the REF website).

Who should have carried out the research underpinning an impact, in order for it to be eligible as an impact case study?

The research underpinning an impact must have been carried out by staff 'while working in the submitting HEI'. This may include research undertaken by unpaid staff or staff who would have been considered as 'Category C' in a former RAE, on the basis that their research was clearly focussed in the submitting HEI. The individuals need not be working in the submitting HEI on the census date but must have been at the time they carried out the underpinning research. Research undertaken by research students is not considered as having been carried out by staff while working in the submitting HEI.

Can we make any changes to the impact case study template?

The template format and page limits stated at Annex F of the 'guidance on submissions' must be adhered to. Changes should not be made to the margins, headers or footers provided in the template. You may use a template with or without borders (see revised guidance). The template also includes section headings for the impact case study; these may not be removed. However, the text stating the indicative word limits and number of references may be removed from your final submission if you wish.

(Updated September 2013)

Previous FAQs

Can diagrams and other images be included in an impact case study?

Yes. The information provided in an impact case study may be presented in any form the institution considers to be appropriate. This may include tables and non-text content, so long as the guidance on maximum page limits and minimum font size, line spacing and margin widths are adhered to.

How many corroborating statements/individual contacts can be provided in an impact case study?

A total of five individuals to corroborate an impact case study can be provided. This may include any combination of statements already gathered by the institution and individuals who can be contacted by the REF team.

We do not envisage auditing more than five statements for any particular case study, which is why we have set this limit. The corroborating sources listed should focus on the key claims made within the case study. If a larger number of individuals could potentially provide such corroboration, then five should be selected that best represent this larger group.

How can we ensure that case studies will not be seen by panel members or assessors with commercial conflicts of interest?

Guidance on 'REF data management' provides information on the arrangements regarding case studies that include confidential or sensitive data.

The REF submission system contains a field 'Conflicted panel members' which can be used to identify panel members or assessors that should not be able to view the case study, due to commercial conflicts. However, if a member or assessor becomes unable to participate we may replace them, and will update the panel membership on the REF website to reflect such changes. Since the panel membership may change over time, in addition to the guidance above, HEIs can use the 'Conflicted panel members' field to identify companies or organisations that should not have access to the case study. Any impact case study with data in the 'conflicted panel member' field will not be circulated to the whole sub-panel; it will be made available only to the specific individuals who are assessing the case study. When the sub-panel chairs allocate the case studies to individual members/assessors, they will not allocate the case studies to individuals or organisations identified in the 'conflicted panel members' field.

Where an individual moved to the HEI after 1 January 2008, and their research had a continuing impact that started before they joined the HEI, which part of the impact can the HEI claim?

The HEI can claim the impact from the point at which research carried out by staff while working in the HEI made a distinct and material contribution to it. Any impact prior to that point can be described in the case study as context only (to help explain the background the impact), but cannot be claimed as part of the impact underpinned by the submitting HEI's research.

What action should an HEI take if the source to corroborate an impact case study is in a language other than English?

If the corroborating evidence is a pre-existing document not available in English, the HEI should reference the document and state what language it is in. The REF team will use the expertise of specialist advisers with the relevant language skills, if corroboration through these sources is required.

Any corroborating statements provided by users and held on file by the HEI will need to be provided to the REF team in English if requested through an audit. Where necessary the HEI will need to arrange for their translation.

Corroborating contacts should be given only for people who the REF team can communicate with in English.

Do all the outputs referenced in an impact case study need to be of at least two-star quality?

A case study should include references to up to six research outputs that represent the body of research or a research project that was carried out at the submitting institution. These should be key outputs that underpinned the impact, and that best demonstrate the quality of the body of work or project. The sub-panels will not expect each individual output to meet the quality threshold, but will wish to be satisfied that the listed work was predominantly of at least two-star quality.

Can the same impact case study be submitted by more than one submitting unit?

Where more than one submitting unit made a distinct and material research contribution to an impact, each of those submitting units may submit a case study of the impact. However, the case studies should not be identical, because each submitting unit will need to show that its research made a distinct and material contribution to the impact. This applies whether an HEI wishes to submit the same impact in different submissions, or different HEIs wish to submit the same impact.

Can an HEI submit an impact case study in a UOA, even if the individual who conducted the research is returned in a different UOA?

Yes, we recognise that individual researchers may undertake research across multiple disciplines over time and that UOA boundaries are not rigid. Provided the underpinning research is within the scope of the UOA in which it is submitted, a case study may be submitted in a different UOA from the individual.

Does the indicative maximum of 6 references to research in an impact case study refer to a combined total of research references and grant information details?

No. In addition to the maximum of 6 research references, grant information details may also be provided, where appropriate.

Can an impact that had taken place before the output(s) from the underpinning research were published be submitted in a case study?

Yes. It is recognised that research may have had impact prior to the publication of the output(s). The end of the publication period for the underpinning research (31 December 2013) extends beyond the end of the period for the impact (31 July 2013).

Can 'interim' impacts be submitted in case studies, and how will they be assessed?

Submitted impacts may be at any stage of development or maturity so long as some effect, change or benefit meeting the REF definition of impact (at Annex C of 'Assessment framework and guidance on submissions') took place during the assessment period (1 January 2008 to 31 July 2013). Regardless of the stage of maturity, case studies will be assessed in terms of the 'reach and significance' of the impacts that had occurred during this period.

Can an institution submit an impact underpinned by research carried out while the staff were working at the submitting institution, but the output was published after they had left the institution?

Yes. The case study should state what research was carried out at the submitting institution, by whom, and when. As with all information provided in submissions, such information should be capable of verification by the institution.

Can an institution submit an impact that is underpinned by research carried out by staff who were returned (or were eligible) as 'Category C' staff in a former RAE?

Yes, provided that they carried out the underpinning research while working in the submitting HEI.

Can an impact case study be submitted in a different REF UOA to the UOA that the underpinning research had been submitted in a former RAE?

Yes. Impacts must be submitted in an appropriate REF UOA, regardless of where the underpinning research may have been submitted to a former RAE. See paragraph 162 of 'Assessment framework and guidance on submissions'.

If a case study is graded as Unclassified, will the HEI be informed whether this was due to the reach and significance of the impact, or due to some other reason (such as ineligibility or insufficient quality of the underpinning research)?

No. The panels may include general comments on this issue in their overview reports, if applicable.

Can the institution only submit impacts underpinned by research in areas in which it still remains active?

A submitted impact must be underpinned by research that was carried out by staff while working in the submitting HEI. The HEI need not remain active in the area of research.

Note that in reaching their overall judgements about the impact element of submissions, sub-panels will consider the submitted unit's recent activity and strategy for enabling impact from its current research.

If the submitting HEI has merged with another HEI or has taken over another research unit, can it submit impacts from research that was undertaken by the other HEI or research unit before becoming part of the submitting HEI?

Where a submitting HEI is the result of a merger between former HEIs, the submitting HEI can submit impacts from the research undertaken by the former, now merged, HEIs.

Where a submitting HEI has taken over a research unit – whether from another HEI or from elsewhere – the submitting HEI can submit impacts from research that was undertaken by the absorbed unit before it became part of the submitting HEI, with prior agreement from the relevant UK funding body.

Prior agreement must be sought by providing details of the nature of the research unit and of when and how it became part of the submitting HEI, to info@ref.ac.uk, no later than 30 June 2013. The REF team will liaise with the relevant funding body and communicate the decision to the HEI.

In each case the funding bodies will take into consideration whether a distinct unit was absorbed by the submitting HEI in its entirety, and the extent to which there has been genuine structural change.

For clarity, these arrangements do not apply to impacts from research carried out by individuals before they joined the submitting HEI. See paragraph 160a of 'Assessment framework and guidance on submissions'.

By Prof Mark Reed @profmarkreed

8 points for writing an effective impact case study

1. Create a coherent narrative that explains clearly the relationship between the underpinning research and the impact. It may be useful to briefly explain what was original or distinctive about the research that contributed to the impacts. Consider starting from the perspective of the beneficiaries – how did they benefit and why is that benefit so important to them? Be as specific as possible if you want the link between the research and the impact to be credible, including specific details about the names of researchers, their position and dates and locations of the research activity. Choose a strong headline for your case study and ensure your summary starts with a powerful opening sentence that summarises the case study and draws the reader in

2. Be as clear as possible about exactly what the impact was, adding some sort of precise quantification with numbers wherever possible. Quantitative data and indicators need to be meaningful and contextualised to clearly support the case being made, not used as a substitute for a clear narrative. Avoid generalised or exaggerated statements about your impact.

3. Clearly identify specifically who has benefited from the work or which groups/organisations have changed something as a result of the research. Bear in mind that this may include ‘intermediary’ organisations as well as your intended ‘end users’ or audiences.

4. Be concise. A concise case study that pulls out the key points easily for readers has far greater impact than one that is dense and rambling. The UK’s Research Excellence Framework gives people strict page limits, but if you don’t have this constraint, then you could consider imposing some sort of constraint on your case studies.

5. Keep your language simple and direct. If possible, get advice from a science writer or communications specialist, but try and avoid introducing inaccuracies. Remember your audience, and that your audience is probably not other academics. Readers should not have to have in-depth expert or prior knowledge to be able to understand your case study. It is essential that impact case studies avoid academic jargon, so that they are accessible to all your readers. 

6. If you are writing multiple case studies, identify key features of best practice and be consistent about all your case studies covering these aspects. You might for example want to consider a particular list of sub-headings for each case study to follow. However, avoid making all your case studies sound too similar, as though they are all written by the same person, if you want them to also feel authentic.

7. Related to this, make sure you provide detailed, specific and independent evidence to support every claim you make. All material required to make a judgement about the impact needs to be contained within the case study. Avoid anecdotal evidence or evidence that might perceived as such. If possible, link to published evidence that demonstrates the impact of your research. It is possible to commission work to demonstrate impacts yourself, but you will need to consider carefully how this will be published, to ensure that it is perceived to be sufficiently independent and credible. Rather than just listing sources of evidence, explain how each source of evidence supports a specific aspect of the impact that has been claimed.

8. Bring your case studies to life with quotes that illustrate the impact with greater resonance than could otherwise be done with formal language. If these quotes are from people with high profile and relevant job titles, then this adds significant credibility to your case study, as well as some lived experience. Finding quotes from people years after an impact has occurred can be tricky however, so it is recommended to collect these from people as the impact unfolds, and if nothing else, keep track of contact details so that people can be easily contacted later. Bear in mind that key people who could attest to the impact of your work may retire or pass away before you need to write your case study. Usually it is best to aim for a good balance of quantitative data and quotes to support impacts. However, in case studies where there is no quantitative data to corroborate an impact (e.g. a policy which has been developed but not yet implemented), quotes may be the primary source of corroboration to evidence your impact. 

What do we know about impact in REF2020?

We don’t yet know the rules and weighting that will be associated with the assessment of research impact in HEFCE’s Research Excellence Framework in 2020 (REF2020), but we do know four important things:

  • Impact was worth 20% of an institution’s score in 2014, and most commentators expect this to either remain the same or increase slightly to 25% for REF2020
  • Some impact case studies submitted to REF2014 that have continued to generate further impacts during the current REF period are likely to be eligible for re-submission to REF2020. HEFCE are currently consulting on this, to try and get the balance right between incentivising Higher Education Institutes to continue developing and building on existing impacts, whilst having sufficient incentives to develop new impacts
  • It is likely that the eligibility criteria and guidelines for submitting impact case studies in 2020 will be broadly similar to the 2014 assessment, although these are currently under consultation too
  • Assessment criteria may be expanded (e.g. including impacts on public discourse and attitudes) or become more fine-grained (e.g. considering different elements of reach such as number versus the types or geographical spread of people adopting a new technology or behavior) in response to feedback from REF2014 panels, but they are still likely to address aspects of significance and reach.

HEFCE commissioned an assessment of the potential for metrics to inform REF2020, which reported in early 2015. It concluded that metrics could not replace case study narratives for the assessment of impact in 2020. Research commissioned by HEFCE about the impact case studies submitted to REF2014 somewhat uncritically accepted the breadth and depth of impacts captured in the assessment, implying that a similar approach in 2020 could yield similarly authoritative “evidence” of impact. This approach to impact may be criticized as biased towards a narrow set of self-reinforcing perceptions of what constitutes valid and significant impacts, created primarily between different sectors of the research community, rather than in the eyes of beneficiaries. However, although research has been proposed to address these issues, if funded, it will not report until after procedures have been set in place for REF2020. It is therefore reasonable to assume that REF2020 impact case studies will be broadly similar to REF2014.

This is what Universities were required to submit in 2014, which is likely to be similar in 2020:

  • Each Unit of Assessment (UOA) had to submit an impact statement (REF3a, Figure 1)
  • Each UOA also had to submit between 2 and 7 impact case studies, depending on the number of academics submitted in the Research Outputs section of the assessment (REF3b, Figure 1)
  • Underpinning research had to be “excellent” (2* quality or above) and must have been conducted by University members of staff to be eligible.

Underpinning research could include published reports (e.g. final reports to funders) where it was clear that they were of 2* or above quality, but it is highly risky to submit a case study with underpinning research that has not undergone peer-review in an international journal, in case panel members do not deem it to be of sufficient quality for the impact to be eligible. If you are submitting grey literature as the underpinning research, provide copies or links to any assessments that may have been made by the funders as a form of peer-review, and if this isn’t possible then make sure you get it reviewed by a number of experts in the field to ensure the panel is likely to judge it as 2* or above in quality. Depending on how long there is before REF2020, you may be able to turn a report from the grey literature into a peer-reviewed journal article and submit both as underpinning evidence (the original report to show that the research preceded the impact and the peer-reviewed version to demonstrate 2* quality). The underpinning research has to be conducted (not just published) at the submitting institution, so members of staff appointed between now and the 2020 assessment will need to be able to conduct significant new research to further underpin and extend existing impacts if these are to be claimed by their new institution.

Research has to have made a distinct and material contribution to any impacts that are claimed, such that the impact would not have occurred or would have been significantly reduced without the research. If this link cannot be adequately established, then the case study may be graded as “unclassified”. Research from a number of different individuals, UOAs and institutions may have contributed towards a particular impact, and as a result, there are a number of REF2014 case studies that cover the same impact in different ways, based on the specific impacts related to the research conducted by each of the different submitting institutions. If you are in this position however, it is worth considering whether the impacts that you are able to claim linked to your work will be as far-reaching and significant as those claimed by a competing institution who is submitting to the same UOA. Panels assessing the same impact submitted by different institutions will inevitably compare case studies, and you want to make sure that if this happens, your grade doesn’t suffer as a result.

0 comments

Leave a Reply

Your email address will not be published. Required fields are marked *