Table 2 Summary of findings according to each research paradigm

Explanatory evidence generation

- internal validity for conventional RCTs in rare diseases is threatened because of the small and often heterogeneous patient population - with small numbers of participants, there is potential for unbalanced confounders across groups despite randomization, and poor statistical power to detect treatment effects- modifications to conventional RCTs have been proposed to help maximize internal validity (e.g., adaptive trials, application of Bayesian statistics)

- lack of patient/family/clinician acceptance to the possibility of being randomized to a placebo group, study designs that make participation more appealing by maximizing time spent on- or guaranteeing provision of- the new treatment have been suggested (e.g., crossover trials, N-of-1 studies)

- challenging to evaluate evidence with small sample sizes and short duration of follow-up

“I do find it quite difficult when the clinical trials are very short, very small numbers, and the endpoints are something like the six minute walk test in regards to really being confident that that is going to be an effective treatment for the patients that I’m seeing.” – Physician 4

“Well the ex-[profession] in me looks at things like, you know, the size of the study, well MPS is [laughter around the table], okay that’s not going to happen. You know, so you have to, it’s hard when you’re looking at MPS because the things that you would normally look for in a good study aren’t going to be there because of the size of the sample…” – Patient/caregiver 3

Comparative effectiveness/pragmatic evidence generation

- external validity is threatened in rare disease research by the desire to enroll homogenous groups of participants for explanatory trials (patient population is often inherently heterogeneous)

- there are also substantial differences between trial protocols and real-world clinical practice, reducing the external validity of some studies (i.e., need to account for co-interventions, less frequent follow-up visits, etc.)

- several study designs have been suggested that may compromise internal validity to some extent in order to address real-world effectiveness (e.g., pragmatic trials, registry studies, hybrid study designs)

- need to address heterogeneity in treatment effect across clinical spectrum

“…the way that the trials are designed, very sub, select populations with the actual disease of concern, which is already a narrow disease as it is. It makes it very difficult for us to know where and when these therapies are going to work. And so, when we’re talking about rare diseases, it really has to be linked to not just research, but effectiveness research about natural history and epidemiology. And given the wide degree of heterogeneity with the diseases that we’re dealing with, we’re going into this with a huge degree of uncertainty about whether or not there really is any evidence to support that these therapies are going to work.” – Policy advisor 2

Patient-oriented evidence generation

- explanatory study designs tend to rely on short-term, surrogate outcomes that are not necessarily clinically meaningful or relevant to patients and/or families

- important to engage with patients and/or families throughout the research process to define important outcomes and determine minimal clinically important difference- there has been a shift/push to improve use of more patient-oriented outcomes in clinical evaluative research for rare diseases (e.g., quality of life measures, activities of daily living)- more difficult to use patient-oriented outcomes because tools have not been standardized in rare disease populations

- very important to measure outcomes that are clinically relevant

“For me I think one of the big issues is the outcome measures that we’re trying to document. For instance, with the lysosomal storage diseases, what is the relevance of a 6 min walk test? What is the clinical relevance of this type of test?” – Physician 5

“…because yes, scientific research is important too, but it’s this push-pull dichotomy between the happiness, the living life, just the simple moments, you know, going outside, sitting in the sun, that type of, going down to the beach, those things need to be equally measured…” – Patient/caregiver 4