Building a trusted framework for uncertainty assessment in rare diseases: suggestions for improvement (Response to “TRUST4RD: tool for reducing uncertainties in the evidence generation for specialised treatments for rare diseases”)

Abstract

The aim of this letter to the editor is to provide a comprehensive summary of uncertainty assessment in Health Technology Assessment, with a focus on transferability to the setting of rare diseases. The authors of “TRUST4RD: tool for reducing uncertainties in the evidence generation for specialised treatments for rare diseases” presented recommendations for reducing uncertainty in rare diseases. Their article is of great importance but unfortunately suffers from a lack of references to the wider uncertainty in Health Technology Assessment and research prioritisation literature and consequently fails to provide a trusted framework for decision-making in rare diseases. In this letter to the editor we critique the authors’ tool and provide pointers as to how their proposal can be strengthened. We present references to the literature, including our own tool for uncertainty assessment (TRUST; unrelated to the authors’ research), and provide a broader perspective on uncertainty and risk management in rare diseases, including a detailed research agenda.

Background

In their position paper entitled “TRUST4RD: tool for reducing uncertainties in the evidence generation for specialised treatments for rare diseases”, Annemans & Makady present recommendations for reducing uncertainty in rare diseases. We very much welcomed this article, which recognises the importance of uncertainty and its implications for Health Technology Assessments (HTA) decision-making, with a particular focus on rare diseases. HTA in the area of rare diseases may suffer from more uncertainty than in other conditions and it is vital that this is dealt with transparently. This article therefore tackles an important topic and shows that more research is needed into the identification, assessment and management of uncertainty in general, as well as in rare diseases.

Our group of authors have developed a tool for identifying, assessing and reporting uncertainty, the TRansparent Uncertainty ASsessmenT (TRUST) tool [1], independently of the research presented by Annemans & Makady and published earlier in 2020. Based on our own research in this area, we think that the article could have benefitted from more detail, clarification and placement in the quickly developing literature in this field. Our aim of this letter to the editor is to provide a comprehensive summary of uncertainty assessment in Health Technology Assessment, with a focus on transferability to the setting of rare diseases. To do this, we critically review the TRUST4RD tool, make suggestions for improvement and reference the wider literature, we demonstrate the use of TRUST in two assessments of orphan drugs in rare diseases, and, finally, a present a broader perspective on uncertainty and risk management and propose a research agenda.

TRUST4RD by Annemans & Makady: suggestions for improvement

Annemans & Makady set out broad considerations around reducing uncertainty in HTA of rare diseases. These considerations consist of three “building blocks”: 1) evidence gaps, 2) data sources and design, and 3) the presence of an iterative dialogue. The authors then move on to describe a process for finding a common solution. We present a critique as well as suggestions for improvement and references to the wider literature for each of these. It may be worth mentioning that the acronym TRUST4RD is not defined by Annemans & Makady.

For 1), the authors claim a new taxonomy of evidence gaps, distinguishing four main types of uncertainties important in rare diseases, according to which uncertainties should be identified early on in the HTA:

• uncertainties related to the size and characteristics of the population;

• uncertainties related to the natural history of the disease and its current management;

• uncertainties related to the new treatment;

• and uncertainties related to the health ecosystem.

We are concerned that these four uncertain aspects of an HTA may unnecessarily restrict stakeholders in their uncertainty identification process. Whilst we acknowledge that these four aspects may indeed be important in rare diseases, we find these to lack specificity. It is unclear to us and may be unclear to other users, whether certain aspects of a HTA are included in these four aspects. For example, health-related quality of life estimates can be a significant source of uncertainty in rare diseases. It could be argued that “uncertainties related to the natural history of the disease and its current management” include health-related quality of life aspects. If this were the case, then it is difficult to see how these four aspects are a reduction of all aspects in a HTA. Furthermore, we would argue to the contrary, that, in order to ensure explicit and transparent presentation of uncertainties, a less condensed view of aspects would be beneficial. Assessments of rare diseases are likely heterogeneous and sources of uncertainty could affect all aspects, so we urge caution against blinding ourselves to aspects with a lower prevalence. The TRUST tool, which was based on a review of the literature and HTA report guidelines and validated using stakeholder interviews, workshops and case studies, distinguishes between the following aspects of an assessment that may be subject to uncertainty: context or scope—including population, intervention, comparator, outcomes, time horizon and perspective; selection of evidence; model structure; model inputs—including transition probabilities, time-to-event, or accuracy estimates; relative effectiveness estimates; adverse events; utilities; resource use and costs; and model implementation. We consider that, even in non-model-based assessments, these distinctions may be helpful and allow for a more comprehensive identification process and more specific assessment and reporting.

The claim to a taxonomy may furthermore be misplaced. Multiple taxonomies of uncertainty exist and they tend to have in common that aspects or locations (such as those proposed by the authors and the assessment aspects considered in TRUST as discussed in the previous paragraph) are only one dimension of uncertainty. The other dimensions include the nature of uncertainty, so whether it is epistemic or aleatory; and sources of uncertainty, sometimes also referred to as “levels” [2]. In our TRUST tool we synthesized the latter based on the literature [3,4,5,6,7,8] as follows: transparency issues, methodological issues, imprecision, indirectness and unavailability. These distinctions may help in study design as well as in making decisions about how to explore and analyse this uncertainty.

We agree with Annemans & Makady that not all uncertainties are worth researching. How to achieve research prioritisation within HTA, however, has been much debated in the past and methods are available to help with this, namely value of information analysis [4, 9,10,11,12]. The authors’ solution is multi-stakeholder involvement; and potential impact scoring of uncertainties. Unfortunately, no methods to derive such an impact score are provided in the article. A part of the TRUST tool enables researchers and analysts to assess the impact of uncertainties on outcomes. In interviews, meetings and case study applications at the Dutch Health Care Institute conducted for the development of TRUST, we found that numerically scoring uncertainties for their impact was not only difficult for stakeholders, it also resulted in substantial inter-rater variability and could evoke the impression of quantified uncertainty, whereas scores were really based on subjective perception. These impact scores were therefore abandoned. We think that, if such scores were to be introduced for HTAs, further research would need to examine how this could work in practice. However, based on our research findings, we consider the benefit of this to be questionable. Instead, in the TRUST tool we proposed a simple assessment of whether the impact would be “likely high”, “likely low”, or “likely no impact”. Annemans & Makady go on to propose “what-if” scenarios to explore the impact of uncertainties, which we consider important, too. However, other commonly used methods [4] such as one-way sensitivity analysis, scenario analyses including threshold analysis, probabilistic sensitivity analyses, model averaging [13] and value of information methods should not be disregarded.

For 2), the authors recommend that different study designs be explored and matched with the evidence gap. There is no specific guidance on exactly what study designs would be beneficial in what settings, although the authors do provide some helpful examples. We think that the TRUST tool’s different sources of uncertainty may be informative in devising study designs. Furthermore, we consider that the impact of uncertainties alone cannot dictate research priorities. As argued by many scholars, the cost of the research has to be weighed up against the value of research and this assessment should be undertaken for different research targets and study designs [14, 15]. Methods like value of information and real options analyses enable such assessments. There are, of course, caveats: not all assessments are based on models that include all uncertainties. Qualitative judgements will have to be made in practice, but we should be clear on the “gold standard”, being an open and transparent consideration of the value of reducing uncertainties and the costs, as well as practical and ethical concerns associated with the potential research [16].

Research should not be considered in isolation when addressing risk in HTA decision-making. Uncertainties that have an impact on the decision at the originally proposed price may no longer do so at a reduced price [17]. Furthermore, research can be coupled with other schemes that target an effective price reduction, such as in pay-for-performance schemes. Consideration of all Managed Entry Agreement options is therefore vital [18,19,20].

For 3), we cannot argue with the importance of an iterative dialogue and collaboration throughout the process. The authors recommend early dialogue between manufacturers and regulatory bodies and also advocate for the application of the principled compromise concept in finding a common solution to addressing uncertainties. We think this is an excellent suggestion and would like to see an application of this following a real-world prospective case study in the future. In the meantime, we suggest the use of TRUST for supporting decision-makers in setting up a reflexive assessment that allows for transparent debate of all uncertainties and their best possible management.

Uncertainty management in rare diseases going forward

Assessments in rare diseases will always be subject to uncertainty. It is all the more important that uncertainty management receives due consideration. In this paper we showed that identification of uncertainties in rare diseases can be done systematically, transparently and comprehensively. However, there are challenges for the future. To provide a research agenda for this topic, we suggest breaking down the rather large topic of uncertainty management into five steps (Fig. 1).

• (1) A clear overview of uncertainties necessitates identification of all uncertainties relevant in a decision problem. Assessment of impact (based on quantitative analysis if available or subjective opinion) can help in prioritising those uncertainties that should be analysed further. This step can be aided by tools such as the TRUST tool [1].

• (2) Ideally, all uncertainties are quantified in a decision model and incorporated in the probabilistic analysis [4]. Uncertainties lead to a risk of making a “wrong” decision, such as making a recommendation for reimbursement, when the technology is, in fact, not cost effective, or vice versa. Risk is composed of the outcomes of a decision and the probability of occurrence [21]. Risk analysis can be performed using value of information methods [9, 22]. Quantitative analyses may not always be feasible due to time and resource constraints. In such cases, the assessment of impactful uncertainties may be used to inform the appraisal. Further research should be performed on exploring uncertainties that are difficult to quantify [23].

• (3) Two features are of interest in the appraisal for a reimbursement decision: the expected net benefit (or expected cost effectiveness) of the new intervention and risk [14, 20]. It depends on these two whether, and if so what type of, risk-sharing arrangements may be indicated. In orphan diseases, per patient risk is typically high. The risk of making a wrong decision for the whole health system or society, on the other hand, is probably often low due to the population being small. For example, even if a new technology turned out not to be cost-effective, cost to society would be limited (population risk). However, the uncertainty about adverse events may result in a high risk for each patient that receives the new treatment (per patient risk). Further research into risk appraisal in orphan diseases may be worthwhile.

• (4) Risk-sharing arrangements can be used to manage the risk. They are broadly composed of financial schemes (outcome-based or simple, not outcome-based) and data collection schemes (such as trials or collection of observational data [20]), or hybrid schemes that make use of both options. First, consideration has to be given to potentially appropriate risk-sharing arrangements. Risk-sharing arrangements can then be assessed quantitatively where possible, for example using the HTA risk analysis framework [17, 24], which requires value of information analysis [25,26,27,28]. Further research could inform a more qualitative approach to identifying research targets, study designs and negotiating an acceptable agreement with manufacturers to ensure appropriate pricing. Furthermore, it may be worth thinking about whether further data collection should not always be considered in the appraisal of orphan drugs.

• (5) Follow-up is important: new data may necessitate a reaction and this needs to be planned for. Management of any risk-sharing arrangements and planning of future re-appraisals as well as monitoring are crucial. It would be informative to study experiences from risk-sharing arrangements and how they have been managed.

Fig. 1

Five steps of uncertainty management in rare diseases

As proposed by Annemans & Makady, an iterative dialogue amongst stakeholders with their multiple diverse perspectives is key. Ideally, this will be upheld throughout all the steps from early assessment to follow-up. Further research may focus on how this could be implemented in existing HTA processes.

Conclusion

We provided a comprehensive summary of uncertainty assessment in Health Technology Assessment, with a focus on transferability to the setting of rare diseases. We made suggestions for improving TRUST4RD in the context of the rapidly emerging uncertainty literature. We defined five steps to uncertainty management and proposed a research agenda specific to these in the field of rare diseases.