Measuring what matters to rare disease patients – reflections on the work by the IRDiRC taskforce on patient-centered outcome measures

Challenges to outcome measurement in rare diseases

Heterogeneity and variability are the two hallmarks of rare diseases. Even within defined conditions, health outcomes (i.e. symptoms and signs of the disease and its overall effect on patient function) can be very different across individuals and these can change as the condition progresses. This is often confounded by the difficulty in distinguishing between the symptoms associated with the rare condition and those due to comorbidities. The main challenge rests in the difficulty to clearly determine and agree on what outcome ought to be measured. This is all the more true as there may not be any discrete outcome that is measurable across the whole patient population.

Take the example of Alström syndrome, a rare complex genetic disorder, associated with a wide variety of symptoms affecting multiple organ systems of the body. Generally characterized by a progressive loss of vision and hearing, it may also involve obesity in childhood, insulin resistance, type 2 diabetes mellitus, dilated cardiomyopathy (i.e. weakening and expansion of the heart) and slowly progressive kidney dysfunction. Additional symptoms including pulmonary, hepatic, renal, and endocrine dysfunction can also occur. This challenge to identify the right outcome is magnified by the small size and geographical spread of the patient population (e.g. only some 1200 individuals affected by Alström syndrome have been identified worldwide to date) and also because half of the individuals affected by rare diseases are children (which raises the question of the reliability of self-reports by young children and the relevance of proxy reports vs. observer reports) [24–26].

Moreover, rarity and poorly understood natural history can sometimes lead to misconceptions about what affects patients at different stages of disease. Thus, clinical information about the disease reported in medical literature may be misleading. And, at times, the outcomes considered critical by clinicians are not the same as the priorities of patients. In chronic, debilitating diseases such as most rare diseases, disease stabilisation ‘is’ improvement and may thus be considered as a meaningful outcome to patients [27, 28]. Another hurdle to accurate outcome measurement relates to a phenomenon known as ‘response shift’, which in this case would refer to situations where rare disease patients adapt to their impairment leading to a ‘new normal’; their self-reported health status becomes ‘fine’ [29].

Patients as partners to understand disease burden

Of course, other data stemming from literature reviews, disease burden surveys, preference studies, real-world/registry evidence, or placebo and standard-of-care control arms of past clinical trials [31] provide invaluable complementary information. In particular, literature reviews can inform upon qualitative and quantitative research, concepts, and existing instruments and core outcome sets (the latter of which can then be evaluated to determine whether they fully capture patient experience). Databases such as COMET or COSMIN [32, 33] may serve as a useful starting point to track these existing instruments. And of note, the use of internet and social media (e.g. RareConnect) can be an effective leverage to overcome patients’ geographical spread. For example, analysis of internet discussion groups posted by rare disease patients can provide access to unabridged peer-to-peer discussion, free from researcher intervention. This may also enable us to access hard-to-reach subgroups.

From clinical concepts to measurement: The power of mixed methods psychometric research in rare diseases

Once the outcomes of interest are identified, an instrument which reflects those outcomes can either be selected, adapted or developed. Whatever the route one may opt for (discussed further below and illustrated in Fig. 2), the parameters remain the same: study samples are going to be small and patient populations are going to be heterogeneous. Therefore, in rare disease research, pragmatism and creativity are required, while maintaining high-standards research practices. This involves moving away from sole dependence on traditional hard psychometric statistics and criteria, and fully integrating what rare disease patients have voiced. As such, the traditional psychometric data-driven approach to PCOM is inherently inappropriate in rare disease because, by definition, there are limited available data to drive the decisions. And thus, there is no inherent constraint on the intelligence we could use in a rare disease context, so we should put the emphasis on the hypotheses and critical thinking to base our decisions, rather than the data. To this end, we believe that mixed methods psychometric research is the best fit in rare diseases. This methodology brings together qualitative and quantitative research methods in tandem with the explicit aim to efficiently utilise data from small samples [42]. The goals of this marriage focus on maximising clinical interpretability, increasing our understanding of the concepts under study, and avoiding potential measurement problems early.

It is fundamental that appropriate conceptual models and definitions are developed in the first instance in the construction of any PCOM. This provides the substantive foundation of any PCOM instrument. The extent to which a conceptual model can then be converted into a list of items (i.e. questions with response options which are assigned numbers), to which the responses of patients or clinicians can be summed to form a total score, requires detailed examination. Of all the measurement properties, ‘content validity’ is sine qua non. Any subsequent quantitative analysis (known as psychometrics) is at best limited, and at worst misleading (or meaningless), without this substantive patient-driven clinically-anchored framework. Adequate and appropriate clinically meaningful interpretation of PCOM data rests heavily on this a priori articulation.

Once we have a draft instrument (or are evaluating an existing instrument), three main psychometric approaches can be used to assess the measurement properties of PCOM: Classical Test Theory (CTT), Item Response Theory (IRT), or Rasch Measurement Theory (RMT). Despite having apparent common goals, these approaches differ methodologically, ideologically, and practically. As in any research setting, the psychometric approach needs to be clearly justified after taking into consideration the context of use and concept of interest. All three psychometric approaches can be used in developing PCOMs. But it is widely acknowledged that IRT or RMT include more sophisticated methods than CTT, and that IRT (in the instances where modelling involves multiple parameters) requires large samples to ensure stable estimates. Therefore, RMT provides the most appropriate and scientifically defensible psychometric methods for use in small sample mixed methods research [43–53]. We expand on the central issues supporting Mixed Methods Research and Rasch Measurement Theory in Additional file 2.

Taken together the key properties of RMT speak directly to the core of the utility of rare disease PCOMs: the ability to detect treatment benefit. When sample sizes are necessarily limited, high instrument responsiveness (i.e., the ability to detect all important effects, even if small) is particularly important. This entails appropriately capturing and quantifying those proximal (core) symptoms and their direct impact, which we would expect to be affected by the treatment, to provide the best chance of picking up a treatment benefit signal. Good examples of the use of RMT in rare disease include the improvement of Hammersmith Functional Motor Scale (HFMS) in spinal muscular atrophy [54, 55], the de novo development of the upper limb PCOMs for DMD [34, 35], and strategies to measure clinical change using the North Star Ambulatory Assessment in DMD receiving different corticosteroid regimens [56].

The routes to PCOMs in rare diseases

Few disease-specific PCOMs are available for use in rare disease. Many commonly used traditional instruments do not relate specifically enough to the disease containing a mix of conceptually different items, some of which are irrelevant or non-applicable. This introduces ‘noise’ rather than ‘signals of patient benefit’ when being used to evaluate a health intervention. That being said, it is practically impossible to develop different specific outcome measures for every rare disease. Therefore, consideration of recycling existing instruments from one context of use to another is worth exploring. Of particular relevance would be considering concept-specific instruments, which may be applicable across a ‘family of rare diseases’ (e.g., autoimmune diseases that share chronic fatigue as a chief complaint, or neuromuscular diseases associated with loss of ambulation) [57].

The relevance of a PCOM relies on its ability to accurately capture the impact of a treatment and communicate those results to patients and other stakeholders. Of particular importance is the degree to which an observed change in a PCOM can be interpreted as clinically meaningful to patients and indicates a treatment benefit. What does a 2-point change on a 0- to 10- point scale mean? Is a 2-point change from 10 to 8 saying the same as a change from 4 to 2? Here again, patients can help researchers define ‘meaningful change’. In a rare disease setting where patients are few, emerging novel methods such as the qualitative exit-interview method whereby patients who recently completed a clinical trial are interviewed to provide insight into how observed changes in the measure compare to their own perception of treatment benefit are worth further investigation [59].

Finally, there may be situations where it is unrealistic to aim to accurately measure – with a metric – the impact of a treatment in patients. Such situations include ultra rare diseases that affect a handful of individuals only. Since flexibility and creativity need to prevail in rare disease research, other ‘out-of-the-box’ routes to outcome measurement may be worth a go (illustrated as the Grey line of Fig. 2). Among them is the idea to implement a series of consecutive patient interviews throughout a clinical study to explore with patients the evolution of their treatment experience and perception of treatment benefit. To which extent such qualitative insights may then be converted into quantitative data is a matter of controversy. Patient diaries, a record of the patient perspective of their day-to-day experiences and thoughts relating to their health and medical condition, are another option [60].

Collaboration

PCOM research is a time-consuming and resource-intensive process. Developing PCOMs for a rare disease should be considered as a non-competitive activity where expertise and resources are pooled. Inherent to rare diseases is the reality of a ‘rare disease community’, where for many rare conditions a network of patients, patient groups, health professionals, researchers and drug developers work together to promote scientific knowledge to win the fight against the rare condition. This collaborative spirit should be extended to PCOM research. Research and development efforts to ‘crack’ a rare disease never happen out of the blue: they result in the slow accumulation of scientific knowledge that gradually crystallise into clusters of drug development programmes. We suggest that, as clusters of ‘drugable’ targets/pathways emerge and are identified, stakeholders of the targeted rare condition should be encouraged (for instance through ‘Community Advisory Boards’ set up by patient advocacy organisations, as recently proposed by Eurordis [62]) to join forces in early development stages to fund research on what outcomes ought to be measured. Since half of drug developers are small-and-medium-sized companies that don’t necessarily have PCOM expertise, such a collaborative model would boost creative PCOM thinking and result in sharing costs. It would also prevent multiple sponsors developing instruments for the same purpose, which creates inefficiencies. An example of multi-stakeholder collaboration, including patients as leading strategic partners, comes from the haemophilia community. Recent success in gene therapy raises the question whether outcomes that have been used for past clinical trials are suitable for future evaluations of potentially curative technologies. Working in a pre-competitive environment, the CoreHEM project [63–65] seeks to address this question through a multi-stakeholder consensus process, with the goal of agreeing on a standard approach to consistent collection and reporting of relevant and well-specified outcomes, with an emphasis on outcomes deemed most important by the haemophilia community. The project includes a balanced mix of patients, clinicians, haemophilia researchers, US and international payers and health technology assessment groups, governmental entities, and pharmaceutical companies that are currently developing gene therapies for haemophilia.

Alignment

Since in rare diseases overall patient outcome is the integration of impacts on different domains that the traditional single primary endpoint design is ill-suited to capture well [30], we consider that investigational research should move towards the use of multiple clinical endpoints to ascertain patient benefit. From that end, PCOMs have a central role to play. As clinical development plans are being designed, we believe that the case of patient-relevant outcomes and endpoints should be discussed proactively with/by regulatory agencies, HTAs and payers, for instance in the context of joint scientific advice meetings or a qualification procedure [39], so that optimal evidence generation plans are designed and agreed on. As Facey et al. have pointed out ‘there is a need to gain international agreement on the evidentiary requirements for clinical effectiveness assessments of rare diseases that is accepted by all stakeholders’ [66]. Of note, there are a number of ongoing initiatives globally (e.g., ICHOM, COMET [67–70]) to develop and agree on standardized collections of outcomes – known as core outcome sets (COS) – to be measured and reported, as a minimum, in all research for a specific clinical area. Their overall aim is to facilitate comparative effectiveness research and evidence synthesis. The Advisory Panel on Rare Disease at the Patient-Centered Outcomes Research Institute (PCORI), for instance, has expressed an interest in developing COS for rare diseases, with a particular focus on paediatric populations [71]. We argue that patients should be heavily involved in the COS development process, so that patient evidence guides the adoption of genuine patient-centered COS.

Integration

Value frameworks and other operational mechanisms to ascertain the value and evidentiary uncertainty of new treatments for rare diseases (such as managed entry agreements, MEAs) are on the rise [72, 73]. However, existing value frameworks largely fall short of consistently measuring outcomes that matter to patients [74]. In order to achieve the overall aim of patient-centered care, we believe that value assessments must be informed by criteria that matter to patients. Thus, we encourage the greater integration of PCOMs into endpoints across both outcomes-based (performance) agreements and patient registries more broadly. Many registries have limited patient involvement in their design, oversight and operations and the information generated may only partly reflect what matters to patients [75] – this can be reversed by a greater use of PCOMs. Likewise, the outcome measures used to guide start/stop criteria across MEAs to allow patients receive reimbursed treatment are often surrogate or generic measures that may only remotely relate to patients’ disease and treatment experience (as recently exemplified by the English MEA on mucopolysaccharidosis type IVa [76]). Incorporating fit-for-purpose instruments that focus on outcomes that truly matter to patients would enhance patients’ acceptance of the burden brought by managed entry agreements on their daily life. In addition, it would also support the legitimacy of ‘stop decisions’ by health authorities to take patients off treatment.

Innovation

Where patient evidence suggests that novel PCOMs or the adaptation of existing outcome measures to make them more patient-relevant are warranted, regulatory and HTA agencies should be open to the prospect of innovative measures or methodologies (such as individualised outcome measures) to capture patient benefit. As our knowledge on natural history for every single rare disease keeps increasing, we must be ready to challenge the established order and any ‘clinically validated’ endpoint altogether – rare diseases require a dynamic model of PCOMs. Of note, we believe that any rare disease PCOM strategy in the context of investigational trials should be mindful of and compatible with novel trial designs to encourage accelerated development of rare disease therapies. Of note, mobile health technologies (wearables, wireless medical sensors, apps etc.) can add to PCOM creativity and hold the promise for improving the quality of patient care and clinical outcomes [77]. Wearables that can unobtrusively measure physiological performance (e.g., movement, vital signs, seizures) offer the opportunity for continuous monitoring and may enhance the ability of researchers to understand the effect of new drugs. Remote assessment (in which the evaluator and individual being evaluated are not located in the same physical space) may also offer significant advantages in the field of rare diseases as they can reduce burdensome travel for patients and families and increase patient access to research studies (while reducing overall costs of collecting patient data). Patients should be involved in the design and operationalisation of such devices.

Communication

For a PCOM virtuous circle to come into play, open communication is key. We are aware of very interesting ongoing PCOM research initiatives in rare diseases – sadly, their outputs are too often under-published, including their foundational patient qualitative works. This is regrettable since awareness of best practices can only further promote the cause of PCOMs and also help avoid duplication of efforts. Scientific publications and presentations to both learned audience and a wider public should occur all along as the evidence on a PCOM is building on. Likewise, training materials (for instance in the context of summer schools for patient representatives) should be developed to empower patients and patient advocacy groups steer PCOM research in future.