Incomplete description of the current body of evidence of the health economics of Duchenne muscular dystrophy

Recently, Ryder et al. [1] published results from a systematic literature review of the burden, epidemiology, costs, and treatment of Duchenne muscular dystrophy (DMD), a rare, terminal, neuromuscular disease for which several molecules currently are being tested in trials. The review provides a somewhat peculiar perspective of the current body of evidence of these aspects of DMD, in particular the economic burden and patient quality of life, which in our opinion may result in misconceptions of prevailing data gaps concerning evidence employed in health technology assessments. In addition, there are several components of the synthesis and reporting of the identified evidence which warrant clarification to avoid confusion. The purpose of this commentary is to highlight some of the shortcomings in the search strategy and synthesis of cost and quality of life data reported by Ryder et al. that ultimately result in a description that we find is at odds with our understanding of the health economic context of DMD.

First, the systematic literature review by Ryder et al. only considered records published up until June 2015. As a consequence, several recent key publications on the economic burden and patient quality of life were not identified. Examples include our own work [2, 3], as well as the work of others [4,5,6,7,8,9]. In addition, several studies of costs and quality of life in DMD published between 2005 and June 2015 are not included in the review (e.g. [10,11,12,13,14,15,16]). For these reasons, the outcomes of the review provide an incomplete description of the current evidence base.

Second, Ryder et al. claim they only included studies of patients with DMD, not mixed populations of e.g. patients with muscular dystrophy. This is a reasonable criterion for robust evidence synthesis and meaningful comparison, as e.g. Becker muscular dystrophy and DMD have very different pathophysiology and thus impact in terms of burden, costs, and quality of life. However, of the three identified cost studies, one [17] is indeed a study of a mixed population of patients with hereditary progressive muscular dystrophy, which includes diagnosis of DMD but also other forms of muscular dystrophy.

Third, in their synthesis of identified cost data, Ryder et al. employ ambiguous labels for reported subgroups and make several incorrect observations when comparing the data reported in the identified publications. For example, in Table 5 in their article, which according to the table title provides a summary of direct healthcare costs (which also is incorrect as some of the reported data contains direct non-medical costs), country-specific labels for the first stratum from Landfeldt et al. [18] read “DMD Age 9 to 17” which most readers would interpret as the range (i.e. the minimum and maximum patient age) of the sample, when it in fact refers to the inter-quartile range (the range was 5–34 years). Moreover, when contrasting cost evidence from different studies, the authors provide little guidance to help readers understand if the different estimates are indeed comparable (with respect to e.g. included resources and valuation techniques). The authors also employ a categorisation of costs (i.e. direct healthcare costs, indirect costs, social care costs, and out-of-pocket expenses) which is not very helpful, as the identified studies estimate and combine these cost components differently.

Fourth, concerning evidence of patient quality of life, Ryder et al. state that parent proxy scores were similar to patient self-assessments of quality of life. However, no explanation is provided for this claim, which in our experience is not entirely accurate. In fact, we have reported as part of our own research [2] that the level of agreement between patient self-assessments and caregiver proxy-assessments to be poor to fair [10, 19], poor to fair to moderate [11], moderate [15], and moderate to good agreement [2, 20]. With respect to the synthesis of the quality of life data, it is unfortunate that the authors devote little effort to contextualize current evidence with respect to e.g. evidence of coping mechanism, self-perceived versus objective quality of life (i.e. as assessed through general population preferences), issues concerning the measurement of quality of life in paediatric populations (which is particular relevant to a genetic childhood disease such as DMD where a non-trivial proportion also suffer from cognitive impairments), and issues concerning precision and generalizability.

Fifth, and last, Ryder et al. claim to assess the quality of the identified records using the STROBE criteria, however, no explanation on how this assessment was conducted is provided. We would suggest authors of future reviews to instead assess the validity of the evidence to facilitate an understanding of e.g. generalizability of results and potential biases, as subjective evaluations based on what can be regarded as “adequate” or “representative” in our experience easily result in misunderstandings.

We hope that this commentary will help readers interpret the review by Ryder et al. and hopefully avoid misguided research initiatives into aspects of the health economics of DMD already described. That being said, there are obviously data gaps concerning costs and quality of life in DMD that warrant further study, although these are unfortunately not correctly mapped out in work by Ryder et al.

Author’s response

• S. Ryder,
• R. M. Leadley,
• N. Armstrong,
• M. Westwood,
• S. de Kock,
• T. Butt &
• J. Kleijnen 

Whilst the commentary by Landfeldt et al. does highlight a couple of deficiencies in the original review, most of the comment is unwarranted or inaccurate. The contents of the commentary do, however, suggest that evidence is rapidly changing and that the period covered by the review up until June 2015 is perhaps not reflective of the findings of more contemporary research including that published by the authors of the letter.

Each of Landfeldt’s five points is addressed in turn as follows:

Missing studies

Authors of the review explicitly state “The main potential limitation of our approach was in its restriction to published and unpublished evidence from 2011 to 2015”. In this sense Landfeldt and colleagues are agreeing with the authors of the paper. Aside from evidence produced after June 2015, Landfeldt and colleagues suggest that seven other studies have been missed. However, this is factually inaccurate in that four of these studies [10, 11, 13, 16] were published earlier than the time frame mentioned above i.e. 2011 to 2015. There is a misleading phrase in the Ryder et al. [1] manuscript which suggests “we decided to focus on records from 2010 onwards” this could have perhaps been better written “all records between 2011 and June 2015 were the focus of the review”. Landfeldt et al. do point to three additional studies [12, 14, 15]: which in their view should have been included. Reasons for exclusion could be further researched if necessary but ultimately no review process, particularly one which considers both published and unpublished research can claim to be 100% infallible.

Study with an incorrect population

Landfeldt et al. suggest that one of the included studies [17] relates to a population which is not restricted to DMD only (it is described as a proxy for DMD). This point is correct. However, the study does, it could be argued, provide useful international comparative data albeit on a contentious definition of DMD. The study was assessed as low quality in the Ryder et al. review which suggests caution of interpretation in any case.

Ambiguous subgroups and incorrect observations

The Landfeldt letter suggests that Ryder et al. use ambiguous labels for subgroups and incorrect observations. The example they suggest is Table 5 which they suggest is incorrect because direct healthcare costs include direct non-medical costs. This is merely opinion. Healthcare is broader than just medical. For example, physiotherapy and occupational therapy is not “medical” expenditure but is “healthcare” expenditure. We would agree that there are different categorisations of cost and that researchers estimate and combine costs differently but this is very much a feature of the literature and not just the review by Ryder et al. Landfeldt et al. will be well aware of differences in cost categorisation used by different authors and that it is difficult/impossible to assess comparability (particularly between countries). It would be great to be more prescriptive but this may not be possible. Landfeldt points out that the range referred to in Landfeldt et al. 2014 is an IQR (and indeed this could have been made more explicit in the paper by Ryder et al. [1]).

Parent proxy and self-assessment

The comments in the Landfeldt letter largely relate to evidence outside of the period studied in Ryder et al. [1] (2011 and June 2015) and so may be reflective of more current research but not that reviewed. Comments made by Ryder et al. [1] are reflective of those made by Lim [20] which indicated good to moderate agreement between parent proxy and self-assessment.

Quality assessment / reporting guidelines

STROBE was used and details are provided as to how this assessment was conducted. STROBE is not a quality assessment tool, but more accurately, a set of reporting guidelines. We have modified them and used them as a checklist to assess quality in the absence of something more suitable. At the time of the review we did not know of a better tool.

Conclusions

The letter by Landfeldt suggests there are shortcomings in the search strategy but then fails to substantiate what these are. Most of the criticism seems to relate to the fact that evidence outside of the time period of review is different to that within the period of review. This may be a very valid observation but unreasonably used as a criticism of the review by Ryder et al. [1].