Systematic review of available evidence on 11 high-priced inpatient orphan drugs
Orphanet Journal of Rare Diseases volume 8, Article number: 124 (2013)
Attention for Evidence Based Medicine (EBM) is growing, but evidence for orphan drugs is argued to be limited and inferior. This study systematically reviews the available evidence on clinical effectiveness, cost-effectiveness and budget impact for orphan drugs.
A systematic review was performed in PubMed, Embase, NHS EED and HTA databases for 11 inpatient orphan drugs listed on the Dutch policy rule on orphan drugs. For included studies, we determined the type of study and various study characteristics.
A total of 338 studies met all inclusion criteria. Almost all studies (96%) focused on clinical effectiveness of the drug. Of these studies, most studies were case studies (41%) or observational studies (39%). However, for all orphan diseases at least one experimental or quasi-experimental study was found, and a randomized clinical trial was available for 60% of the orphan drugs. Eight studies described the cost-effectiveness of an orphan drug; an equal number described an orphan drug’s budget impact.
Despite the often heard claim that RCTs are not feasible for orphan drugs, we found that an RCT was available in 60% of orphan drugs investigated. Cost-effectiveness and budget impact analyses for orphan drugs are seldom published.
Evidence based medicine (EBM) stresses the importance of evidence stemming from clinical studies next to clinical experience, rather than making treatment decisions on intuition and clinical experience alone . Attention for EBM in reimbursement decisions on drugs has grown over the years, also for policy decisions concerning drugs for rare diseases (i.e. orphan drugs). Compared to the evidence base for common drugs, however, the available evidence on effectiveness, cost-effectiveness and budget impact of most orphan drugs is limited. The evidence that is available, often does not meet traditional quality standards. Small number of patients and heterogeneity of the diseases hampers enrollment of sufficient patients in trials [2, 3]. Randomized controlled trials (RCTs) are therefore often claimed to be unsuitable for orphan diseases [4, 5] and thus inferior evidence is deemed satisfactory by decision makers [6, 7].
In the Netherlands, from 2006 till 2012 a policy rule enabled temporary reimbursement for orphan drugs. For these drugs, treatment centers are required to collect data on effectiveness and cost-effectiveness over a period of 4 years. The Dutch policy rule on orphan drugs lists 11 inpatient therapies for 10 orphan diseases (Table 1); two therapies are listed on the policy rule for one indication (Fabry disease). Table 1 also provides the number of patients deemed eligible for treatment in the Netherlands as well as the estimated costs per patient and total budget impact of the treatment and the year for which these figures were estimated. Finally, Table 1 provides information about whether the EMA provided marketing authorization under exceptional circumstances (i.e. authorization was granted despite an incomplete dossier with respect to safety and/or efficacy). These estimates were based on the Dutch temporary reimbursement dossiers.
In this study, we examine what evidence on effectiveness, cost-effectiveness and budget impact is available for these drugs. For this purpose, a systematic review was conducted. The findings of this review provide insight into what type of studies (with respect to study design) were performed in orphan diseases despite small numbers of patients.
The PRISMA guidelines for systematic reviews were followed . A literature search was conducted in PubMed and Embase for all orphan drugs on the policy rule. The literature searches were performed in August 2012. The search terms included the disease and treatment (brand name and generic name; Table 1). For instance the following search string was performed for acute lymphoid leukemia: Acute Lymphoid Leukemia [OR] Acute Lymphatic Leukemia [AND] Clofarabine [OR] Evoltra. Similar search strings were used for the other orphan drugs. In both PubMed and Embase, the search was limited to English results. No additional search terms or limitations were used to ensure all relevant studies were found. A supplementary search was performed in the NHS Economic Evaluation Database (NHS EED) and HTA database to identify literature on cost-effectiveness (both databases were accessed through http://www.crd.york.ac.uk/CRDWeb/HomePage.asp). Only treatment (brand name and generic name) was used as a search term. The searches in the grey literature were performed in September 2012.
Eligibility criteria were stated before assessment of the studies. Two researchers (TK & CdS) subsequently decided independently on selection of the studies. In the first selection round, studies from PubMed and Embase were screened on the basis of titles and abstracts. In a consensus meeting differences were discussed. The same reviewers then assessed the full-text articles of the remaining studies. Studies from grey literature were assessed in the same way.
The following eligibility criteria were used for all drugs investigated (in this order):
Relevance to disease of interest: only studies on the disease of interest were included (e.g. exclude studies on the treatment of other diseases, orphan diseases in general, etc.)
Primary focus on disease of interest: only studies focusing primarily on the disease of interest were included (e.g. exclude studies on multiple diseases one of which is the disease of interest)
Treatment: only studies on the effectiveness, cost-effectiveness and budget impact of the treatment of interest were included (e.g. exclude studies on alternative treatment options)
Treatment of humans: only studies on the treatment of humans were included
Format: only research articles were included (e.g. exclude conference proceedings, editorials, comments, letters, conference abstracts, supplements etc.)
Summaries: Only original research articles were included (e.g. exclude summaries and reviews)
Duplicates: Multiple publications on the same study were identified in the assessment of full-texts. These studies were included in the analyses only once, to avoid double counting.
Accessibility: papers for which no abstract and no full-text were available were excluded.
For Acute Lymphoid Leukemia (ALL) and Chronic Lymphoid Leukemia (CLL), additional exclusion criteria were defined. When reviewing the evidence for ALL, studies in adult patient populations were excluded, as the treatment is only registered for the use in children and adolescents in the Netherlands. For CLL, only studies on patients who were refractory to fludarabine and alemtuzumab were included; studies on ofatumumab in previously untreated patients were also excluded to comply with the Dutch indication for of atumumab. When the drug was provided as part of a combination therapy, the record was excluded on account of the third exclusion criterion (i.e. the focus was not on the treatment of interest).
Analyses of included studies
Two researchers (TK and CdS) carried out the data extraction independently. Included studies were classified into three groups with respect to subject: 1) effectiveness; 2) budget impact; 3) cost-effectiveness. Studies could potentially be classified into more than one group.
Studies on clinical effectiveness were examined using the hierarchical level of the study design . Interventional studies (experimental or quasi-experimental designs) were defined as studies in which the investigator determined treatment regimen. In contrast, in observational studies treatment was decided on the basis of clinical characteristics of individual patients. Studies were labeled case studies when patients were individually described. Extension studies of clinical trials were considered as observational studies when all patients were treated with the new drug.
In addition, several key elements of study design were identified: control, randomization, blinding, follow-up duration and number of patients enrolled in the study. Controls, if any, could be placebo, historical, healthy and untreated patients or patients receiving supportive care. Randomization could have been done with respect to treatment, dosage, route of administration or no randomization. Blinding options were double blind, investigator blind, patient blind or open label. For interventional studies the follow-up duration of the study was also assessed.
Articles were considered as describing budget impact when total costs on a societal or health care level were provided; the sole mention of treatment cost per patients was not considered as a budget impact analysis. The relationship between disease prevalence and available evidence was tested using a Spearman correlation.
Figure 1 shows that Embase yielded the most results from the literature. Searches in grey health economic literature resulted through the HTA and NHS EED databases in another 44 studies. After removal of duplicates, 3,612 unique studies remained. A total of 2,062 studies were excluded after screening titles and abstracts. Consequently, 1,550 (42.9% of total studies assessed) full-text articles were assessed, of which 338 met the inclusion criteria. More than one third of the 1,550 articles (39.4%) were excluded on the basis of format and a similar number (37.5%) was excluded due to being summary articles.
Table 2 shows that almost all of the 338 included studies described effectiveness of therapy. Only eight studies focused on cost-effectiveness, two of which were found in published papers and six in the grey literature (i.e. reports from various national health technology assessment agencies). Eight studies described the budget impact of an orphan drug. Budget impact for agalsidase α and agalsidase β was described in the same article. Six reports from the grey literature described both cost-effectiveness and budget impact of an orphan drug.
Case studies made up the largest share (42%) of all studies on effectiveness (see Tables 3 and 4). A further 39% included studies described observational studies. Although only a limited number of interventional studies were described (in total 19%), Table 3 shows that interventional studies were available for all drugs. For most interventional studies, study designs were adapted, particularly with respect to use of a placebo control group and blinding. Only a minor share of interventional studies (n = 14; 22%) involved a randomized, placebo-controlled, double blinded clinical trial. For four orphan drugs, no such study was available.
In 33% of the interventional studies and 28% of the observational studies a control group was used. In most studies healthy controls and historical controls were used as a comparison group; in only 21% (12/58) of the studies the control group received placebo treatment.
Interventional studies often applied multicenter and international study designs to enlarge sample sizes. Nevertheless, the size of the study population in these interventional studies averaged 48 patients (range 7-270). A total of 16% of observational studies (21/132) were based on registry data; the majority stemming from the registry set up for agalsidase α in Fabry disease.
No significant correlation was found between the disease prevalence and the number of included studies on effectiveness (rho = 0.373; p = 0.258). There appeared to be fewer studies available for orphan drugs with market authorization under exceptional circumstances compared to drugs authorized without exceptional circumstances (average 25 versus 36 studies, respectively). However, the limited number of drugs under study limited the statistical power to detect any important differences.
Cost-effectiveness analyses and budget impact studies
Only eight studies described a cost-effectiveness analysis of an orphan drug. All cost-effectiveness analyses made use of a health economic model. Seven studies presented a cost-utility analysis, in which incremental costs per QALY (quality adjusted life years) ratio was calculated, whereas one study described a cost per life year gained. All studies included survival gains in their analyses.
Eight studies described the budget impact of an orphan drug. Most studies (n = 6) only described the costs of treatment itself, but the budget impact study on agalsidase α and agalsidase β as published in the same paper examined the impact of the interventions on the health care budget as a whole, including (substitution of) costs of alternative treatments and other health care usage.
This study reviewed the available evidence for 11 orphan drugs listed on the Dutch policy rule on orphan drugs. In total, over 3,612 studies on these orphan drugs were reviewed, of which 3,274 (91%) were excluded; 338 studies remained. The vast majority of these studies focused on drug effectiveness. These results resemble the results of a recent study in oncologic orphan drugs, which also showed the remarkable absence of pharmaco-economic evidence in these drugs .
A notable finding in this review was that interventional studies were available for all of the 11 orphan treatments investigated. In fact, a placebo-controlled double blind randomized trial was available for seven of these drugs. These findings correspond with the results of two earlier studies examining EMA drug approval dossiers, which found that about 60% of orphan drugs were assessed using an RCT [12, 13]. While this seems to contrast with the claim that randomized controlled trials are not feasible for orphan drugs [4, 5], the current review also showed that the design of these trials differed compared to ordinary RCTs, with a particular focus on multi-center, international studies. Earlier studies comparing orphan and non-orphan cancer and neurological drugs also showed adaptations in study design in orphan drug studies [14, 15]. Studies in various therapeutic areas have shown that the size of study population seen in orphan drug RCTs is also significantly smaller than those for non-orphan drugs [14, 16].
An adequate study design in studies assessing orphan drugs is not only important because enlisting patients in badly designed studies is unethical , but also, due to the small number of patients, there is generally only ‘one shot to do it right’; no treatment-naïve patients might be found after an initial study has been performed. Evaluation of the included studies on effectiveness showed that adaptations to study design were not simply made because of the small number of patients, but also because of other aspects. For instance, in the absence of an alternative treatment it can be considered unethical to withhold treatment from patients . This was especially seen as a hurdle against setting up a placebo-controlled trial for children and infantile patients with high mortality risks (for instance ). In addition, when the drug was already commercially available it proved to be difficult to initiate a placebo-controlled trial.
A large part of the studies that were assessed were observational studies. In reading this review, it must be considered that observational studies, when adequately set up and performed, need not always be inferior to randomized clinical trials or systematically overestimate treatment effects [11, 20]. Although an RCT is the most appropriate study design to prevent the treatment effect from being biased, in some cases, RCTs might not be a feasible method to study effectiveness of therapy . In this sense, for example, problems relating to the feasibility of blinding or randomization, and ethical considerations also determine whether an RCT is the most suitable study design.
A large share (38%) of full-text publications were excluded because they only provided a summary of available evidence and did not contribute any new knowledge on the particular orphan drug. This high frequency of summaries might be explained by the orphan status of the diseases. Due to the rarity of the diseases, most clinicians might encounter only one or two patients with a particular orphan disease during their careers . This can lead to delays in diagnoses and even misdiagnosis [23, 24]. To educate other clinicians about the disease, experts on particular orphan diseases might frequently summarize the current state of knowledge in summary articles. Any delay in making the right diagnosis is unwanted, especially because for some orphan diseases effectiveness of treatment is dependent on timely initiation [25–28]. In addition, a substantial proportion of studies assessed (39%), was excluded because these were not classified as research articles, but had another format (predominantly conference abstracts). In this sense, conference presentations are also hypothesized to be used to disseminate knowledge to other physicians about new therapies and educate other clinicians about orphan diseases.
Limitations of the study
As with any systematic review, the existence of a publication bias cannot be ruled out – studies without significant therapeutic effects may not be published. Publication bias might especially be prominent for cost-effectiveness analyses. Because of high costs, orphan drugs are rarely cost-effective under conventional thresholds  and this might decrease pharmaceutical companies’ incentives to publish such analyses. Publications describing budget impact might be less prone to publication bias, since budget impact is generally small for an orphan drug due to the small numbers of patients treated. However, for all orphan drugs combined, the budget impact is substantial . A publication bias for budget impact analyses can therefore not be ruled out.
The Dutch policy rule was specifically designed to provide early access to high-priced orphan drugs (with an expected budget impact of at least €600,000) for diseases with high unmet medical need. The drugs listed on the policy rule might not be representative for other orphan drugs, which might limit the generalizability of our results to other orphan drugs. Furthermore, transferability of the findings for these drugs to outpatient drugs needs to be studied in future research. However, this review covered 11 inpatient orphan drugs from a broad spectrum of disease areas. The overall conclusions could therefore be considered to represent the situation with orphan drugs in general, and not simply the situation with a specific disease.
In assessing the effectiveness of orphan drugs, policy makers can expect an international interventional study to be available. However, policy makers should not expect that country-specific RCTs have been carried out. In assessing the cost-effectiveness and budget impact of treatments, policy makers should not expect to find large body of evidence in the literature. Instead, they are restricted to evidence submitted by pharmaceutical companies or to coverage-with-evidence-development schemes.
Whether the available evidence is considered to be sufficient fully depends on the role of evidence based medicine in reimbursement decisions. Further research is needed to examine the relation between available evidence and positive reimbursement decisions.
The results of this review showed that at least one interventional study was conducted for all orphan drugs and that at least one randomized placebo-controlled, double-blind, clinical trial was available for over 60% of orphan drugs. The claim that RCTs are not feasible in orphan diseases therefore does not seem to hold for all orphan diseases. Cost-effectiveness and budget impact analyses for orphan drugs are seldom published.
Sackett DL, Rosenberg WM, Gray J, Haynes RB, Richardson WS: Evidence based medicine: what it is and what it isn't. BMJ. 1996, 312: 71-72. 10.1136/bmj.312.7023.71.
Buckley BM: Clinical trials of orphan medicines. Lancet. 2008, 371: 2051-2055. 10.1016/S0140-6736(08)60876-4.
Luisetti M, Balfour-Lynn IM, Johnson SR, Miravitlles M, Strange C, Trapnell BC, Van Bronswijk H, Vogelmeier C: Perspectives for improving the evaluation and access of therapies for rare lung diseases in Europe. Respir Med. 2012, 106: 759-768. 10.1016/j.rmed.2012.02.016.
Hughes DA, Tunnage B, Yeo ST: Drugs for exceptionally rare diseases: do they deserve special status for funding?. QJM. 2005, 98: 829-836. 10.1093/qjmed/hci128.
Wilcken B: Rare diseases and the assessment of intervention: what sorts of clinical trials can we use?. J Inherit Metab Dis. 2001, 24: 291-298. 10.1023/A:1010387522195.
Dupont AG, Van Wilder PB: Access to orphan drugs despite poor quality of clinical evidence. Br J Clin Pharmacol. 2011, 71: 488-496. 10.1111/j.1365-2125.2010.03877.x.
Cheng MM, Ramsey SD, Devine EB, Garrison LP, Bresnahan BW, Veenstra DL: Systematic review of comparative effectiveness data for oncology orphan drugs. Am J Manag Care. 2012, 18: 47-62.
Orphanet: Prevalence of rare diseases. Paris: Bibliographic data; 2011.
Toker O, Hashkes PJ: Critical appraisal of canakinumab in the treatment of adults and children with cryopyrin-associated periodic syndrome (CAPS). Biologics. 2010, 4: 131-138.
Moher D, Liberati A, Tetzlaff J, Altman DG: Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009, 151: 264-269. 10.7326/0003-4819-151-4-200908180-00135.
Concato J, Shah N, Horwitz RI: Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000, 342: 1887-1892. 10.1056/NEJM200006223422507.
Joppi R, Bertele’ V, Garattini S: Orphan drugs, orphan diseases. The first decade of orphan drug legislation in the EU. Eur J Clin Pharmacol. 2012, 69: 1006-1024.
Putzeist M, Heemstra HE, Garcia JL, Mantel-Teeuwisse AK, Gispen-De Wied CC, Hoes AW, Leufkens HG: Determinants for successful marketing authorisation of orphan medicinal products in the EU. Drug Discov Today. 2012, 17: 352-358. 10.1016/j.drudis.2011.10.027.
Kesselheim AS, Myers JA, Avorn J: Characteristics of clinical trials to support approval of orphan vs nonorphan drugs for cancer. JAMA. 2011, 305: 2320-2326. 10.1001/jama.2011.769.
Mitsumoto J, Dorsey ER, Beck CA, Kieburtz K, Griggs RC: Pivotal studies of orphan drugs approved for neurological diseases. Ann Neurol. 2009, 66: 184-190. 10.1002/ana.21676.
Orfali M, Feldman L, Bhattacharjee V, Harkins P, Kadam S, Lo C, Ravi M, Shringarpure D, Mardekian J, Cassino C, Coté T: Raising orphans: how clinical development programs of drugs for rare and common diseases are different. Clin Pharmacol Ther. 2012, 92: 262-264. 10.1038/clpt.2012.87.
Halpern SD, Karlawish JH, Berlin JA: The continuing unethical conduct of underpowered clinical trials. JAMA. 2002, 288: 358-362. 10.1001/jama.288.3.358.
Rothman KJ, Michels KB: The continuing unethical use of placebo controls. N Engl J Med. 1994, 331: 394-398. 10.1056/NEJM199408113310611.
Kishnani PS, Corzo D, Nicolino M, Byrne B, Mandel H, Hwu WL, Leslie N, Levine J, Spencer C, McDonald M: Recombinant human acid α-glucosidase: major clinical benefits in infantile-onset Pompe disease. Neurology. 2007, 68: 99-109. 10.1212/01.wnl.0000251268.41188.04.
Benson K, Hartz AJ: A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000, 342: 1878-1886. 10.1056/NEJM200006223422506.
Black N: Why we need observational studies to evaluate the effectiveness of health care. BMJ. 1996, 312: 1215-1218. 10.1136/bmj.312.7040.1215.
Luisetti M, Campo I, Scabini R, Zorzetto M, Kadija Z, Mariani F, Ferrarotti I: The problems of clinical trials and registries in rare diseases. Respir Med. 2010, 104: S42-S44.
Stolk P, Willemen MJC, Leufkens HGM: Rare essentials: drugs for rare diseases as essential medicines. Bull World Health Organ. 2006, 84: 745-751. 10.2471/BLT.06.031518.
Rinaldi A: Adopting an orphan. EMBO Rep. 2005, 6: 507-510. 10.1038/sj.embor.7400450.
Clarke LA, Wraith JE, Beck M, Kolodny EH, Pastores GM, Muenzer J, Rapoport DM, Berger KI, Sidman M, Kakkis ED: Long-term efficacy and safety of laronidase in the treatment of mucopolysaccharidosis I. Pediatrics. 2009, 123: 229-240. 10.1542/peds.2007-3847.
Banikazemi M, Bultas J, Waldek S, Wilcox WR, Whitley CB, McDonald M, Finkel R, Packman S, Bichet DG, Warnock DG: Agalsidase-beta therapy for advanced Fabry disease. Ann Intern Med. 2007, 146: 77-86. 10.7326/0003-4819-146-2-200701160-00148.
Van den Hout JMP, Kamphoven JHJ, Winkel LPF, Arts WFM, De Klerk JBC, Loonen MCB, Vulto AG, Cromme-Dijkhuis A, Weisglas-Kuperus N, Hop W, Van Hirtum H, Van Diggelen OP, Boer M, Kroos MA, Van Doorn PA, Sibbles B, Van Corven EJJM, Brakenhoff JPJ, Van Hove J, Smeitink JAM, De Jong G, Reuser AJJ, Van der Ploeg AT: Long-term intravenous treatment of Pompe disease with recombinant human α-glucosidase from milk. Pediatrics. 2004, 113: e448-e457. 10.1542/peds.113.5.e448.
Wraith JE, Scarpa M, Beck M, Bodamer OA, De Meirleir L, Guffon N, Meldgaard Lund A, Malm G, Van der Ploeg AT, Zeman J: Mucopolysaccharidosis type II (Hunter syndrome): a clinical review and recommendations for treatment in the era of enzyme replacement therapy. Eur J Pediatr. 2008, 167: 267-277. 10.1007/s00431-007-0635-4.
Drummond MF, Wilson DA, Kanavos P, Ubel P, Rovira J: Assessing the economic challenges posed by orphan drugs. Int J Technol Assess Health Care. 2007, 23: 36-42.
Schey C, Milanova T, Hutchings A: Estimating the budget impact of orphan medicines in Europe: 2010-2020. Orphanet J Rare Dis. 2011, 6: 62-10.1186/1750-1172-6-62.
This study was financially supported by the Netherlands Organization for Health Research and Development (ZonMw; grant no. 152002045). The funding source had no role in the design, analyses, reporting of the study or in the decision to submit the manuscript for publication.
The authors declare that they have no competing interests.
TK performed the literature searches. TK and CdS performed the inclusion and exclusion of studies. All authors were involved in the design of the study and have been involved in writing and revising the manuscript. All authors have given final approval of the final version of the manuscript.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
About this article
Cite this article
Kanters, T.A., de Sonneville-Koedoot, C., Redekop, W.K. et al. Systematic review of available evidence on 11 high-priced inpatient orphan drugs. Orphanet J Rare Dis 8, 124 (2013). https://doi.org/10.1186/1750-1172-8-124