RAND Europe, Cambridge and 1 Health Economics Research Group, Brunel University, London, UK.
Correspondence to: S. Wooding, RAND Europe, Grafton House, 64 Maids Causeway, Cambridge CB5 8DD, UK. E-mail: wooding{at}rand.org
![]() |
Abstract |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
Methods. The payback framework was applied to 16 case studies of research grants funded in the early 1990s. Case study methodology included bibliometric analysis, literature and archival document review and key informant interviews.
Results. A range of research paybacks was identified from the 16 research grants. The payback included 302 peer-reviewed papers, postgraduate training and career development, including 28 PhD/MDs, research informing recommendations in clinical guidelines, improved quality of life for people with RA and the reduction of the likelihood of recurrent miscarriage for women with antiphospholipid syndrome. The payback arising from project grants appeared to be similar to that arising from other modes of funding that were better resourced.
Conclusions. There is a wide diversity of research payback. Short focused project grants seem to provide value for money.
KEY WORDS: Research evaluation, Arthritis, Research payback, Returns from research
![]() |
Introduction |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
This paper summarizes the results of an evaluation of research grants funded by the Arthritis Research Campaign (ARC) in the early 1990s. We aim to demonstrate how research outputs and outcomes can be systematically surveyed, and go on to illustrate how this information can provide an evidence base for research funding policy. A methodological paper explaining the evaluation framework adopted for the study has been published [6], as has a two-volume report which sets out our approach, results, conclusions and recommendations in considerable detail, along with descriptions of the case studies that were compiled as part of this evaluation [7, 8].
ARC is the UK's fourth largest medical research charity; its mission states that it is Committed to curing arthritis. In pursuit of this mission, ARC spends £22 million per year to support research into the causes, treatment and cure of arthritic conditions [9]. ARC supports a wide portfolio of research, including basic, clinical and allied health professional research; this research is supported primarily in a university and institute context. Definitional issues hinder precise classification, but currently about half of ARC funding supports clinical research, one-third supporting basic science. The remainder supports research in various areas, including the allied health professions and education. In 2002, to mark its 65th anniversary, ARC undertook a strategic review of its activities and impacts. The 5-yr strategic plan, Research into Practice [10], concluded there seems to be a gap between the aspirations of people affected by arthritis and the ability of science and academia to meet those aspirations. The strategy document went on to recommend that ARC needed to instigate a system of rigorous retrospective evaluation on work which has already been completed, with a view to identifying opportunities for development. This recommendation resulted in the study reported here.
![]() |
Methods |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|
|
Using the evaluation framework, we constructed case studies of 16 research grants selected from 556 possible grants awarded by ARC between 1990 and 1994. With the help of the ARC's Development Committee, we identified candidate case studies to allow us to compare the effect of the mode of research support, the type of research and the bibliometric impact of the principal investigators (PIs). Case-study PIs were then contacted to see if they were willing to participate in the study: all were. Our selection of grants contained: six project grants, three programme grants, three fellowships and four institute grants; six basic grants, eight clinical grants and two allied health professional grants (classified according to the qualifications of the PIs); and nine high-impact PIs and seven mid-impact PIs (classified according to bibliometric indicators). [We were interested in whether grants of the most successful researchers produced a spectrum of outputs different from those of less successful colleagues. Success was determined from the PI's total publication record from 1990 to 2002. We identified the PIs' papers in the Science Citation Index and added up the journal impact factor (JIF) for each paper. We then ranked the total JIF score to identify the top and middle 10% of PIs. The top 10% were defined as high impact, and the middle 10% as mid impact.] With sixteen case studies we could not expect them to be representative of all ARC grants in a statistical sense [19]; however, by using a selection matrix we aimed to produce a set of case studies that mirrored the diversity of ARC funding in key dimensions and hence would be a basis for cautious generalization.
Using the information collected from document and literature reviews, semistructured key informant interviews and bibliometric analysis, each of the 16 cases was written up and published as a narrative organized according to the structure provided by the payback framework [8]. Use of a common structure facilitated comparative analysis, allowing us, for example, to identify the factors apparently associated with the successful translation of research [19]. We employed two approaches to compare across cases. The first was based on a qualitative assessment of the case studies. The second involved a novel method of scoring the case studies on the five payback categories.
We carried out the scoring process during a 2-day workshop of the research team, using a process developed for reviewing evidence and testing agreement in the formulation of clinical guidelines [20]. Team members were asked to individually score the level of payback from each case study in each category. Scoring of each of five categories was undertaken on a scale of 19 (where 1 was the lowest and 9 the highest score). The scores from the first round were analysed and sheets showing their distribution were circulated to the team. Following this, there was discussion about scoring discrepancies in order to resolve any misunderstandings between the team about the evidence on the outputs from the case. The team members then had the opportunity to rescore all of the case studies for that payback category, so providing a set of second-round scores. The median of these second-round scores was used for the analysis presented here.
![]() |
Results |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|
The largest number of papers were produced by case studies E and L, with over 40 papers each. The highest levels of citation were achieved by case studies C, H and L; the collection of papers from these case studies all received over 100 citations per year. In considering measures of knowledge production based on citation, it is important to remember the disadvantages suffered by the allied health professional field of research. Of the papers attributed to these grants, case studies A and I, only 53% were published in journals for which citation information existed in the Science Citation Index (SCI); furthermore, many of the citations of those articles that are included will be missed, as the citations will have been in journals not covered by the SCI.
Research targeting and capacity building
The majority of case studies recorded significant paybacks in terms of research targeting and capacity building. A major element was the career development of either the PIs or postgraduate and postdoctoral research assistants who were employed on the research grants. For example, 24 scientists were awarded either a PhD or an MD as a result of the research undertaken on the 16 grants; for case studies I and J, these were awarded to the PI. A number of the PIs were subsequently appointed to senior academic positions, including research chairs, clinical consultants, heads of Medical Research Council (MRC) units, and the dean of a medical school. It is inappropriate to attribute wholly such outcomes directly to the ARC funding, but many of the PIs suggested it had been a contributory factor.
A further element of research capacity was the transfer of technological know-howin case study O, this involved genetic mapping from a group acknowledged as leader in the field, which hosted the PI for a 6-month training fellowship before she returned to her host institution. Similarly, in case study P, a number of interviewees commented that the establishment of the Norfolk Arthritis Register increased (arthritis) research capacity locally.
In terms of research targeting, case studies A, B, C, H, K and L seem to have the major impact. As summarized in Table 2, this included: informing a >£2 million MRC-sponsored clinical trial (case study A); pharmaceutical companies' interest in recombinant TIMPs (tissue inhibitors of metalloproteinases) (case study C); a Japanese study on occupational lifting (case study B); subsequent research on the effect of antiphospholipid antibodies on in vitro fertilization (case study K); and a significant, possibly paradigm-changing, shift in the use of biologicals as therapeutic agents (case study L).
Finally, it is worth noting the development of reagents (case study C) and a mouse model (case study D) as further examples of research payback. In the former, although records were not kept, the PI estimated that the reagents produced during the grant were sent to over 100 other research groups.
Informing policy and product development
As with the previous two categories, we identified a range of different types of payback that informed either policy development or product development. These include: being cited on systematic reviews; being cited on clinical guidelines and other forms of policy guidance; and the development of specific clinical tests.
Four papers that were attributed to case studies A and B were cited subsequently in systematic reviews. Moreover, in both of these examples, the quality of the original papers was assessed by the authors of the systematic reviews and the studies scored well against the criteria used. Seven papers from the case studies were cited on clinical guidelines or technology appraisals relating to arthritis. Case study P had the highest level of citation on clinical guidelines, with three papers cited on arthritis-related guidelines and one paper cited by the non-arthritis-related guideline. Case study L, which relates to the use of anti-TNF therapy in RA, had two papers cited in a National Institute for Clinical Excellence technology appraisal. Case studies E and K both had one paper that was cited by arthritis-related guidelines. For case study E, this was a European guideline. For case study K, the findings were an important element in the British Royal College of Obstetricians and Gynaecologists guidelines recommending a combination therapy of aspirin plus heparin to prevent recurrent miscarriages for women who have antiphospholipid syndrome (APS).
An especially interesting example of payback arose from case study B. This grant looked at the role of occupational activity and hip osteoarthritis (hip OA) and concluded that agricultural workers were at an increased risk of hip OA. This work was cited in the IIAC assessment of whether hip OA in farmers should be a prescribed disease (and thus receive Industrial Injuries Benefit); the Council's subsequent advice to the Secretary of State was that this was justified. In addition to the paper, one of the lead PIs gave a presentation to the Council.
In terms of informing product development, three case studies identified potential drug targets: case study C stimulated industrial research in the area of matrix metalloproteinase inhibitors; case study H indirectly informed the development of a class of drugs used outside the arthritis field to treat deep-vein thrombosis; and case study L tested the hypothesis that RA could be treated by using targeted TNF- inhibitors.
Finally, two case studies, F and G, identified specific diagnostic tests, although these were for very rare conditions. The first was for specific mutations in collagen X and is used to test for chondrodysplasia type Schmid. The second enabled the measurement of antibodies to C1q in patients with systemic lupus erythematosus (SLE).
Health and health sector benefits
Three grants have clearly had significant potential health and health sector benefits. Case study K concluded that a combined treatment of aspirin and heparin for women with APS, which affects about 15 in 10 000 women, reduces the risk of recurrent miscarriages and provides these women with a far higher chance of a successful pregnancy.
Case study L provided some approximate estimates of the numbers of RA patients benefiting, in terms of improved symptom relief and physical functioning. To date, take-up has been limited, due to the high cost of treatment and doubts about the drugs' cost-effectiveness except in restricted groups of patients [21]. The prevalence of RA in Western populations is estimated at 0.8% and there is an uptake of approximately 2% of patients in the UK, 6% in Scandinavia and 15% in the USA, this suggests that approaching 10 000 patients in the UK and 350 000 patients in the USA are receiving anti-TNF treatment. Results of trials suggest that between 29% [22] and around 70% [23, 24] may enjoy a health improvement due to the anti-TNF treatment.
Although case study P concerned the epidemiology and prevalence of RA, it was, nevertheless, seen to be providing health benefits by its identification of other risk factors, such as heart disease, for RA patients; this information was improving patient care.
For the more basic research, it is still to early too judge what the total benefits may be. For example, for case study C, there are still active drug development programmes that were stimulated by the research. In other cases, such as case studies E and F, improved understanding of the disease processes of OA may lead to future improvements in treatment; similar benefits may come out of case study H for RA patients. Finally, there are case studies where the technology and techniques developed have laid the groundwork for current basic research that eventually may lead to patient benefit: case study O has led to work in pharmacogenomics that may benefit RA patients.
Broader economic benefits
For cases A, B, J and K, there are unquantified returns in the reduction of days off work and the value of production gained from having a more healthy workforce. In addition, for case study L, there are economic returns to those companies that have licensed drugs. However, we were unable to identify evidence to quantify the broader economic returns arising from this ARC-sponsored research.
Cross case analysis: the effectiveness of different modes of funding
In order to assess the strengths and weakness of different modes of funding, we undertook comparative analyses of the case studies. The cross-case analyses were based on a qualitative examination of common types of payback (Table 3), supported by the quantitative assessment of scores summarized in Fig. 2.
|
|
Analysis of Table 3 and Fig. 2 suggests that, despite lower costs and shorter duration, at a median size of grant of £90 000 over 2.5 yr, project grants yield considerable payback over a range of categories, and do not appear to underperform the other three funding modes (programme grants at a median cost of approximately £480 000 over 5 yr, fellowships at approximately £250 000 over 5 yr, institute grants at approximately £450 000 over 4 yr). Funding provided to institutes resulted in the widest range of levels of payback and also encompassed the highest-scoring study. Project grants and institute funding include the case studies with the broadest health benefits (case studies K and L, respectively). By comparison, programme grants (roughly comparable to institute grants in median funding amounts and periods) perform poorly, with the payback recorded as limited to two PhDs, a guideline citation and a clinical test.
![]() |
Conclusions and policy implications |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
The first caveat is the use, selection of and generalization from case studies. Given the approach that we adopted and the budget, we had to restrict the number of case studies to 16. This raises the question of whether the results of this study can be generalized to all ARC research or indeed beyond that. This, in itself, then becomes a debate on the legitimacy of qualitative research and the use of case studies in general. Yin, who has written a seminal text on case study research, argues that the case study is but one of several ways of doing social science research ... [each with] peculiar advantages and disadvantages [19, p.1]. In terms of scientific generalization, Yin makes the point that scientific facts are rarely based on single experiments and that the investigator's goal [with case study research] is to expand and generalise theories (analytic generalisation) and not to enumerate frequencies (statistical generalisation) [19, p.10].
A second issue is the attribution of payback to ARC-funded work. It is undoubtedly true that there is a range of unquantified and (largely) unspecified non-ARC inputs into the research evaluated in the 16 case studies. This issue was explored in some depth at a 1999 international workshop on research evaluation [25], and it was concluded that there are many complications in identifying the impact of specific funding. How far it is desirable to attempt this in detail depends partly on the purposes for which the evaluation is being conducted.
A third issues relates to the robustness and consistency of the scoring system that we developed for the payback profiles (Fig. 2). Although we believe that this system is a major step towards being able to operationalize and embed a multidimensional research monitoring and evaluation system, we would stress that it was developmental. We need to have a better understanding of the reliability of the scoring system, a tighter definition of the scoring scale and an understanding of the quantity and type of information necessary to produce informed and consistent scores. All of these issues are future research topics.
Finally, there is the issue of elapsed time. In deciding on a time window for research evaluation, a compromise needed to be found between the quality of records, the ability of researchers to recall their activities and allowing enough time for the outcomes of research to develop [26]. In this study we used a 10-year lag, which seemed to be an optimal trade-off between these two requirements. However, it was not long enough to allow a fair comparison between basic and clinical research. Previous work examining how long it takes for basic research to inform clinical guidelines suggests that even after four generations of citation, taking around 17 yr, less than 20% of the cited research is basic [27]. And it is worth noting that the anti-TNF work, case study L, would have been basic research in the mid-1980s whilst case study O has resulted in further pharmacogenetics work that may benefit patients with RA in the future.
Diversity of payback
There is strong evidence from our analysis that there is a wide range of research paybacks and that many of these would not have been readily identified without the structured in-depth case study approach that was employed in this evaluation. In Table 2 we summarize the various types of payback identified from the 16 case studies. While this is an impressive list, it should be remembered that our sample of 16 case studies was biased towards more successful investigators, because we selected high- and mid-impact investigators rather than a full spread. Nevertheless, these returns come from only 16 (or 3%) of 556 possible grants, and, even if cautiously extrapolated, illustrate a diverse and significant return to ARC's investment in research.
Given this diversity of outcome, it is important that the ARC and other research funders use an assessment method that captures a broad range of payback. The payback categories developed in this study provide a structured basis for any further monitoring or evaluation of the outputs and outcomes of research funded by the ARC. This wide-ranging assessment is important, as there is a tendency in research evaluation to rely heavily on bibliometric analysis and, as shown in this study, this can seriously underestimate payback.
The importance of a multifaceted approach to research evaluation was confirmed by Cronin and Normand [28] in their review of the literature. First, they concluded that there was consensus that all criteria needed a qualitative and quantitative information base; and secondly that a number of different criteria should be used. This was best articulated by Martin and Irvine [29], who proposed the concept of converging partial indicators; that is, a series of incomplete indicators that indicate the reliability of the assessment when they are all pointing in the same direction.
The relative value of project, programme and institute grants
The cross-case analysis presented in Fig. 2 and Table 3 shows a significant and diverse range of payback arising from the six project grants that we examined. This included, for example, 62 papers, four PhD/MDs, informing four clinical guidelines or policy documents, reduced risks in miscarriages for women with APS, and unquantified reductions in days of work. By comparison, the three programme grants resulted in 82 papers, two PhDs, one citation in a clinical guideline and the development of a clinical test for chondrodysplasia type Schmid. This does not appear to show the much higher returns that might have been expected from programme grants. However, it should be borne in mind that: (i) programme grants, by their very nature, might be expected to be more speculative and long term; and (ii) we only examined three such grants. [We are also subsequently told that the award system for such grants has become far more rigorous and competitive since the early 1990s, when these grants were first awarded.]
Of all the observations we have made from our analysis, this was the most unexpected and surprising. There is a widespread view in science policy that long-term stable funding is preferential, although all types of support mechanisms have been shown to contribute significantly to 13 clinical advances in cancer research [30]. However, there are a number of potential confounding factors that mean the conclusion regarding project grants needs to be treated with caution. For example, four of the six project grants were patient-focused, and three of the project grants involved a randomized controlled trial. This suggests perhaps that the projects were more mission-oriented and therefore likely from the outset to lead to faster payback. Conversely, the fact that patient-focused, randomized controlled trial-type work is being funded through projects illustrates the importance of maintaining a mechanism for funding short-term focused research of this nature. This unexpected finding needs to be tested elsewhere. It was also notable that the institute case studies showed the widest range in outcomes, including the most successful case study. This could imply that institute funding allows the pursuit of speculative and innovative research that, when successful, has large payback.
For a number of decades, research funding organizations have been interested in assessing the payback from their research. In response to this demand, analysts have developed a number of methods to evaluate research. In this study we have combined a number of different approaches to assess the long-term payback from ARC-funded research. We would like to conclude by noting that the rationale for supporting this study was well founded. Research funding agencies such as the ARC need a firm evidence base to support policy and decision-making. Indeed, other research funding agencies should follow ARC's lead and accept the need for a firm evidence basis for their policies through supporting studies such as these [5].
|
![]() |
Acknowledgments |
---|
This work was carried out for and funded by the Arthritis Research Campaign. As such we acted as consultants for the Arthritis Research Campaign. All the authors continue to seek financial support to undertake studies researching and evaluating the effectiveness of research funding organizations.
![]() |
References |
---|
![]() ![]() ![]() ![]() ![]() ![]() ![]() |
---|
|