The goal of this research was to develop and validate a new tool COMPASS to assess the quality of OMPs clinical evidence presented at the time of marketing authorization in the EU. The COMPASS tool (Additional file 1) does not attempt to score or rank the quality of clinical evidence, but rather to give an outline of various, key elements with respect to quality of clinical evidence, as seen by experts. Ultimately, it is up to the evaluator to define minimum conditions of quality for an individual OMP or a set of similar OMPs. Ideally, these conditions should be defined without taking into account unmet need and disease severity, as these are not determining factors for quality of clinical evidence.
This COMPASS tool can be applied to assess the quality of evidence of an OMP based on the information in the registrations dossiers, for example by local reimbursement agencies for the review of clinical evidence or by pharmacists and clinicians upon considering a (new) treatment. Registration and reimbursement bodies currently acknowledge the data limitations for OMPs by providing them special considerations . However, over time they are likely to become more sensitive to data requirements . For example, non-binding recommendations for the approval of cancer drugs and biologics were issued by the Food and Drug Administration (FDA) several years ago . Nowadays, deviations from the guidelines, stated in a similar European document, should be thoroughly justified .
To improve the reliability of results, data extraction should be performed independently by at least two raters. The reliability of results has shown to be independent of blinding, for that reason data extraction should not necessarily be blinded . An exception on the number of raters can be made for the first part of the tool, as it collects more general and descriptive information about the OMP. Yet, minor differences between the raters can occur depending on the information source. To reduce variability, the source of information was pre-specified for some queries (ie specified the source of prevalence data to EPAR and Orphanet). There are more subjective questions in part two of the tool, emphasizing the need for independent raters.
Study quality is dependent, not only on methodological quality, but also on the quality of reporting . Indeed, shortcomings in the reporting can complicate the interpretation of the methodological quality. Correct and complete information should provide the reader with the ability to make informed judgements about the validity of a study . In practice, it was (nearly) impossible to assess the methodological quality of those (few) studies that were only very briefly discussed in the EPAR. Additionally, the interpretation was also complicated for those EPARs that consisted only of literature studies. To address the issue of poor reporting, part three of the tool focuses on quality of reporting. Additionally, for some questions, an additional check box was provided with 'Not reported’. Whilst in some cases the difference between answering 'No’ or 'Not reported’ may be vague, the subtle difference influences the evaluation of the quality of reporting.
The COMPASS tool has several strengths and weaknesses. The tool was developed after iterative rounds of expert consultation and underwent several pilot rounds to increase its validity. The tool assesses the level of clinical evidence that is presented in the pivotal studies at the time of marketing authorization. As such, it does not take into consideration the evidence in any of the supporting studies or evidence generated as part of post-marketing commitments. Adding data from publications would allow for quality control of the EPAR data, but was considered outside the scope of this study. Also, the tool is dependent on the quality of reporting in the EPAR and/or SD documents. Finally, a general medical knowledge of the rater is advisable to complete the tool. Additional check boxes ('Don’t know’ and 'Not reported’) were provided, to account for possible problems related to these last issues.