Patient involvement in rare diseases research: a scoping review of the literature and mixed method evaluation of Norwegian researchers’ experiences and perceptions

Background Patients’ involvement (PI) in research is recognized as a valuable strategy for increasing the quality, developing more targeted research and to speed up more innovative research dissemination. Nevertheless, patient involvement in rare diseases research (PI-RDR) is scarce. The aims were: To study the Norwegian researchers` experiences and perceptions of PI-RDR and review the literature on PI-RDR. Methods 1. A systematic scoping review of the literature on PI-RDR. 2. A cross-sectional questionnaire study with close-ended and open-ended questions to investigate the researchers` experiences. Results In the scoping review 608 articles read in full-text and 13 articles (one review and twelve primary studies) were included. The heterogeneity of the design, methodology and results was large. Most studies described several benefits of PI, but few described methods for measuring impacts and effectiveness of PI-RDR. In the cross sectional part of this study, 145 of 251 employees working in the nine Norwegian Centers on Rare Diseases participated, of these 69 were researchers. Most (95%) of the researchers claimed that rare diseases research is more challenging than for the more common diseases. The majority (95%) argued that PI-RDR may increase the quality of the studies and the relevance, and most (89%) agreed that PI-RDR in dissemination may increase the awareness and public interest for rare diseases. In the open-ended questions several researchers also claimed challenges related to PI-RDR, and many had proposal for improving PI and promotion of rare disease research. Conclusion Both the literature and researchers emphasized that PI-RDR is important for improving research quality and increase the public attention on rare diseases, but what constitutes effective PI-RDR still remain unclear. More research on the design, methodology and assessment for measuring the impact of PI-RDR is warranted. Supplementary Information The online version contains supplementary material available at 10.1186/s13023-022-02357-y.

diseases [5]. RDs are associated with limited published research to inform interdisciplinary clinical practice, thus limited evidence-based practice, and barriers for developing clinical guidelines [3]. This scarcity of RDs knowledge is challenging for professionals who want to provide evidence-based health care to patients with RDS.
Modern clinical research increasingly recognize patient engagement (PI) as a valuable part of RDs research [3,6]. Involving patients with RDs in the planning, conducting and dissemination of research may be a valuable approach for addressing evidence gaps for management of rare diseases. PI in research may promote research that evaluates health outcomes that are both relevant to patients with RDs and useful for decision making [3,[7][8][9]. Several Norwegian associations as well as most research funds in general have begun to require a statement of PI in their call for research proposals. According to INVOLVE [10] PI refers to the inclusion and activation of patients as partners in various stages of the research process, or as "research being carried out "with" or "by" patients rather than "about" or "for" them. Involving patients with rare diseases is emphasized as particularly important due to the low incidence and the unique challenges they face associated to the rarity of their diseases.
Studies [8,[11][12][13] indicate that Patient Involvement in Rare Diseases Research (PI-RDR) can improve the relevance of research questions, study design, methodology, recruitment rare, interpretation of data and financing. Thereby, higher likelihood of translation and adoption of research results in everyday practice as well as more effective communication findings. All of this may lead to improve clinical practice and better outcomes in patients [14]. Despite the growth of PI in rare diseases research and in general research, and that most researchers recognize the potential value of PI, achieving this potential in practice may involve many challenges. While literature is growing on different methods of PI, the conceptual meaning behind PI still in unclear. Lack of clarity of the definition and that the terminology of the concept varies greatly [15] may be a barrier to fully implement PI in rare diseases research. In a newly published review [15] of the definition of "patient involvement" or "patient engagement", a horizon scan of related terms identified 24 terms each with multiple definitions across the health care sector. This review [15] found that one of the most common term used was patient involvement (PI).
Although there are anecdotal indication that PI-RDR increase the relevance and quality of studies, a challenge is the lack of systematic evidence to demonstrate the impact [8], and the limited amount of systematic sets of measurement methods for assessing the impact [14]. A systematic review from 2014 [8] of PI-RDR, including both articles and grey literature, found that most studies reported perceived impacts of PI-RDR that was not measured or confirmed. This review [8] also reported that studies mainly emphasized how PI could facilitate relevant clinical questions and patient centered outcomes, and few studies addressed PI-RDR in dissemination process. Improvement of communications around RDs may be essential to improve the quality of life for people with RDs [16]. The European Union Committee of Experts on Rare Diseases (EUCERD) recommendations for Centers` of Expertise (CoE) underscore the importance of collaboration with patient organizations in research and provide information that is accessible and adapted to the patients' needs [17,18].
Although authors report benefits of PI in the research process in general, the knowledge about approaches, impact and effectiveness of PI-RDR is limited. The existing PI framework have not addressed the specific considerations surrounding RDs [3]. Therefore, an overview of the existing research and more systematic information about the researchers` experiences and perceptions of PI-RDR seems warranted.
Therefore the aims of the study were:

Aim 1: Review of literature on patient involvement in rare diseases research Study design
A scoping review methodology was applied because this is a suitable method for mapping findings from a research area that is heterogeneous in methods or disciplines. Scoping reviews are also suitable for examining the extent, variety, range and characteristics on a topic such as Patient Involvement in Rare Diseases Research, and also for identifying gaps [19,20]. The method is suitable for applying broad review questions, also when very diverse findings make systematic review with critical appraisal and meta-analysis difficult [20,21]. This scoping review was performed according to recommendations from the Johanna Briggs Institute and Collaboration Centers guidance for conduction scoping reviews [22], and the PRISMA Extension for Scoping Reviews (PRISMA ScR) [21] (shown in Additional file 1). However, we have made one adjustment by looking into the findings of the included studies, and have synthesized the overall results on experiences, benefits and challenges of PI in rare disease research. This review was guided by the question: "What is the characteristics and extent of research on Patient Involvement in Rare Diseases Research".
1. What is the extent of secondary research articles (i.e. reviews) versus primary articles describing PI-RDR? 2. What is the characteristics of the study population (types of diseases, number of participants) and when and where have the studies been carried out (i.e. publication years and country)? 3. What types of PI approaches were used? 4. What was the reported impact of PI (on design, conduct, relevance, dissemination and impact on the participants) and effectiveness of PI (long-term outcome, cost-benefits etc.)? 5. How was the impact and/or effectiveness of PI measured (which methods or tools were used for assessments)?

Eligibility criteria
The review included secondary and primary research aiming to investigate and present results on Patient Involvement in Rare Diseases Research (adult patients, patient representatives and representatives for rare diseases organizations). Studies of PI on pediatric participants were not included due to the particularly methodology used for involving children in research. We included articles that stated that they aimed to investigate PI in Rare Diseases Research. Table 1 present inclusion and exclusion criteria.

Search strategy
Systematic searches were conducted until 20th September 2021, in the following databases PubMed, AMED, CINAHL (EBSCO), Embrase (OVID), Eric, Google Scholar and Web of Science. We searches for a combination of subject heading (where applicable) and text words for rare diseases or rare disorders. We also searched for a combination of subject heading (where applicable) and text words for patient involvement and related terms with equivalent meaning (see Table 2). A three stage search strategy was utilized, including search words shown in Table 2

Selection of publications
All review steps were performed by two authors (TB/GV). Endnote software was used for selection and data extraction. Following the removal of duplicates, citations were screened independently by two reviewers (GV, TB) based on title and abstract (level 1 screening) and full-text articles (level 2 screening). The studies were assessed against the inclusion and exclusion criteria. When agreement was not reached, conflicts were resolved through discussion with a third author (TH), using the inclusion and exclusion criteria ( Table 1). The screening process was done in two steps. In the first step abstracts and titles were reviewed and all articles clearly not meeting inclusion criteria were excluded. In second step full-text was collected and reviewed for the remaining articles, those that did not meet the eligibility criteria were excluded.

Data extraction
A data-extraction form was developed to provide a standardized and transparent method for mapping the methodology and findings from the studies [19,20]. This was piloted on a subset of relevant papers and refined to ensure the extraction template met the specific objectives of the review. One researcher (GV) extracted data into a priori form, and another (TB) checked the accuracy. The following data were collected from each article. Bibliographic data, nationality/country of the study, study aim, participants data (number, diagnoses, age, status (patients, patient representatives, patient organization representatives or other stakeholders), objectives with PI-RDR, PI approaches and methods, reported impact of PI, measurement methods for measuring the impact (benefits and disadvantages) and effectiveness (long term utility, cost-benefits) of PI. The findings were summarized and synthesized [23] and presented in a table, and research dissemination was specified in a separate column due to the special methodology this entails.

Aim 2: Norwegian rare disease researchers' experiences and perceptions of patient involvement in rare diseases research Organizational context and participants
All employees working in the nine Centers of Rare Diseases (CRDs) in Norway were invited to participate in this survey in November 2019. The nine CRDs are organized under the umbrella organization the National Advisory Unit on Rare Disorders in Norway (NKSD) [24]. NKSD is a governmental entity with a national responsibility to provide patients, medical health professionals and the general public with updated information on RDs. The CRDs play a central role in generating medical and psychosocial scientific knowledge on RDs Information about NKSD and the nine CRDs in Norway are available at website [24].

Study design and research questions
This cross sectional survey is a part of a larger study about patients' involvement in research on rare diseases research, approved by the Data Protection Authority at Sunnaas Rehabilitation hospital (03.02.2019). The survey is conducted in accordance with the STROBE guidelines for observational studies [25]. A web-based survey was chosen to facilitate access and maintain anonymity of the participant.
The survey was guided by the question: "What was the researchers' experiences and perceptions with Patient Involvement in Rare Diseases Research?".
Our specific questions were: 1. Is there particular challenges related to rare diseases research? 2. What is the extent and experiences with PI-RDR? 3. What is the researchers` perceptions and experiences of benefits and challenges with PI-RDR.

Mixed method design
A study specific web-based questionnaire was developed due to lack of relevant validated instruments. The questionnaire had a mixed-method approach, combining closed-ended and open-ended questions strategically placed throughout the questionnaire. Mixed method is recommended as a powerful tool for collecting more detailed and specific responses from the respondents [26,27].
The use of open-ended questions enabled us to give the respondents "another" response option, and to explore, explain or reconfirm the closedended questions. Analyses of the respondents' verbatim responses gave important insight, not only into respondents' substantive answers, but also in how they understood the questions and thereby establishing the validity of the questionnaire (26.27).

The questionnaire
The questionnaire was constructed in order to capture the employees and researchers experiences and perceptions of PI-RDR. The literature review was used to inform the choices of questions. The questionnaire was piloted by researchers in the Research Department at Sunnaas Rehabilitation Hospital, which led to valuable feedback and revision of three questions. The questionnaire was divided into items of questions to all employees and specific questions to researchers only.
Questions to all employees: • Questions on demographic data: Gender, age, educational level, profession, work place (rare disease center), length of employment history in the field of rare diseases. • Two close-ended about their experiences with rare disease research. • Two close-ended and three open-ended questions about the need for more knowledge about PI-RDR and proposal for improving PI-RDR and research dissemination.

Specific questions to researchers:
• Five close-ended and three open-ended questions to elicit the researchers' perception and view of rare diseases research, and collect information of their suggestion for improving rare disease research. • Three close-ended and three open-ended questions about experiences, perception and proposal for improving PI-RDR.
• Five questions including a 10 points Likert scale of the researchers experiences and perception of benefits and challenges related to PI-RDR and dissemination. • Two close-ended and two open-ended questions about the researchers' experiences and perception of benefits and challenges related to PI-RDR in dissemination and implementation of research results.

Analyses
A digital self-reported questionnaire Questback platform was established for data collection and analyzes [28], in addition to Excel. Questback software [28] is appropriate for capturing anonymized questionnaire responses digitally, facilitating automatic transcription and computer assisted coding. The quantitative data were analyzed and presented with descriptive statistics (mean, median and range or number and frequency, as appropriate). Thematic template analyses [29] were used for the open-ended questions, including a five steps process: (i) Get the data into the template by using excel, (ii) Identify response categories, (iii) Record the individual responses, (iv) Organizing the categories, (v) Present the data visually (using figures and quotations). Two researchers (GV/ TH) independently conducted the initiating steps of the analytic process, of identifying categories and recording the individual responses into the categories. Discrepancy or disagreement were discussed until agreement with the third author (TB).

Ethical considerations
Ethical issues were considered in all stages of the study [30]. Invitation to participate was send to each of the employees` e-mail with information about the study and that participation was voluntarily, including a Web-link to the survey. The participants consented by voluntary returning completed questionnaire. Due to that the participants were employees from different CRDs in Norway and easily identifiable in the context, a detailed description of the sample and advanced statistical analyses were not conducted.

Aim 1: Review of literature on patient involvement in rare disease research
Of the 625 potentially relevant papers read in full text, 13 articles; one secondary (review) and twelve primary research articles met the eligibility criteria and were included in this review. Figure 1 shows a flowchart of the screening and inclusion process.
Data extraction from the thirteen included articles are shown in Table 3.

Secondary studies (review)
Only one review [8] of PI in research of rare diseases was identified, published in 2014. This was a systematic review, but no quality assessment of the included articles was conducted. This review included all types of publications (n = 35) (narrative reports, web-sites, qualitative studies, survey studies and grey literature), on patients, parents, professional and other stakeholders involvement. The authors of the review claimed that although nearly all included articles reported benefits of PI, the methods for assessment of the impact and effectiveness of PI were lacking. Only two [31,37] of the 35 publications in this review fulfilled our inclusion criteria. The reason for the exclusion was that most of these publications were not peer reviewed articles and/or not dealt with rare diseases.
The main objectives with most of the included articles were to create high quality research by using PI, to examine different methods of PI, and/or to share their experiences of the process, methods and perceived impact of PI and to address gaps.

Reported impact of patient involvement in rare diseases research
All studies reported some positive impact of PI-RDR either on the process or the products, or both. Several studies emphasized the importance of including patients in the early stage and setting the research agenda, influencing new avenues for research, improving the study design and the research protocol and by addressing what really matters for patients [12, 33-35, 38, 39]. It was reported that PI-RDR is of great importance for improving the relevance and the utility of the study [11, 12, 33-35, 38, 39]. Three studies [12,33,38] also reported benefits of PI on recruitment rate by promoting research projects through patient organizations and web-sites. PI-RDR was also reported improving the compliance and commitment for both researchers and patients involved in the research project [32,38]. Four studies [12,32,37,38] also emphasized the benefits of mutually learning through interaction and new understanding and new perspectives as benefits of PI-RDR. One study [37] reported that researchers and clinicians made several discoveries that had transformed their understanding of the diseases and treatment, significant for diagnosing and counselling patients. Increased self-efficacy and confidence for the patients involved were also reported [12,13,37,39]. Three studies [11,34,38] claimed that PI-RDR in the dissemination of research increased the end-use accessibility, understandability, readability and action ability. They indicated that PI-RDR in the translation and dissemination meant more effectivity in conveying the message and reaching the target audience. One study [33] reported that PI-RDR created more awareness and engagement in society and increased financial support. Another study [39] indicated that PI-RDR contributed to valuable insight into the difficulties or small organizations trying to do research.
Few studies described challenges and disadvantages with PI-RDR, but some [12,13] described that PI requires extra time and efforts for patients and researchers. Descriptions of the effectiveness such as long-time utility or cost-benefits of PI were almost absent.

How was the impact and effectiveness of patient involvement in rare diseases research measured?
The studies stated different methods for measuring the impact of PI, but the measurement and methodology were very limited described in most studies [13,31,[37][38][39]. Some [11,34,36] did not include any information about the methodology used for measuring the impact of PI-RDR. The majority of the studies mainly offered and account of experiences with PI in research, elaborated and presented jointly be academic researchers and patients, claiming that empowerment was an important purpose in each instance. One study [35] used a questionnaire; the Standard Data on Initiatives (STADIT) to researchers and patients about their experiences and perceptions of the process and impacts. Another [32] used the "Ladder of participation tool" for assessing the levels of involvement. Only one study [12] included comprehensive description of methods for assessing the process and impacts of PI-RDR, combining interviews, observation noted, web-conferences and telephone interview. Very few studies described in detailed what worked and which strategies generated desired outcomes, or rigorously assessed the research activities.

Aim 2: Results from the mixed-method study of the researchers' experiences and perceptions
Of 251 employees from nine Norwegian Center for Rare Diseases, 145 participated (response rate 58%). Of these; 77% were women, 54% > 50 years old, 62% had higher education (master or Ph.D), and 47% had worked more than 11 years with RDs. Forty-eight percent (N = 69) had conducted research on RDs (see Table 4).
The results are presented sequentially by the themes from the questionnaire, including the categories that emerged from the open-ended questions for each of the themes. The four themes were: (1) Challenges with rare diseases research, (2) The extent of experiences with PI-RDR and perceived benefits and challenges. The themes including the subcategories, are illustrated by researchers' citations (shown in Additional file 2).
In the open-ended questions researchers reported that rare diseases research poses unique challenges, and that many barriers exists in advancing knowledge and recruitment options for RDs. The most obvious challenge mentioned for conducting rigorous research was the small sample sizes, but also the heterogeneity between and within RDs.
Some challenges are unique to rare diseases research, one is the small populations, another the heterogeneity, including the different subtypes of the diagnoses (researcher).
Some researchers also claimed that methodological and data constraints may limit the ability to generate evidence. Others emphasized the analytic challenges related to small samples, whether there is sufficient (statistical) power to draw definitive conclusion, included the extent to which available data can be viewed as representative for the entire population of patients within the condition.
We need particular design for rare diseases. It is difficult to do statistical analyses and there is a need for specific methods for rare diseases research (researcher).
Many of the researchers had proposals for improving rare diseases research. Some mentioned that it is important to identify innovative approaches that may overcome the methodological challenges inherent in the study of rare diseases. Other proposed international research collaboration on specific disease populations and particularly on the ultra-rare diseases, to increase the study population. Others stated a need for more collaboration within the rare society for developing more rigorous methodology, and illuminate the particular challenges related to living with rare diseases.
Together we are stronger-we need to work together, to find common solutions in the rare society (researcher)

The researchers' experiences and perceptions of patient involvement in rare diseases research
Fifty-one percent of the researchers reported that they had conducted PI-RDR, and 54% stated that PI is of particular importance in rare diseases research. Their understanding of the concept of PI-RDR and dissemination varied, 32% defined PI as "that patients are informed about the project", 30% that "the patients are involved and can give feedback", and the rest that "the patients are actively involved in all phases of this research process".
On a-scale from 1 to 10 very few agreed that PI-RDR was useless, but several stated that it is difficult and demanding (Fig. 3).
In the open-ended questions most researchers were positive to PI-RDR. Several emphasized that is was important to involve patients in the early stage of the research process for identifying research topic that are relevant and address patients' needs.

Collaboration with patients is crucial for high quality research. They know what they need so we have to listen (researcher).
Most of the researchers recommended PI throughout the research circle. Several mentioned the particular benefits of PI on the recruitment rate, due to small populations. They claimed that involving patients' organizations may increase the legitimacy and interest of the study when patients' organization promote information about research projects on their web-sites or meetings. Others stated that involving patients in the analyses and interpretation of results could provide new dimensions and understanding of the results. Some claimed that PI could help promote collaboration networks and provide financial support for research infrastructure. Others stated that it is important to plan levels and types of PI, appropriate for the different projects. Some stated that in some research projects PI may not be of great value, and that it is important to avoid PI as tokenism or just being "politically correct. The PI must be fruitful and meaningful for both the patients and the researchers.
Despite that PI is politically correct, the collaboration must be realistic compared to what we achieve. In some type of research projects PI may be meaningless.
Several researchers' also stated challenges related to PI, such as difficulties in educating patients for equal research participation, power imbalance between researcher and patients and the importance of facilitating and paying attention to the patients' health problems when they are involved in research process. Some also claimed that PI is resource-intensive, time-consuming and logistically demanding, therefore it is a necessary to assess the effectiveness and long-term utility impact of PI-RDR.
Forty-one of the researchers reported that they had conducted PI-RDR in the dissemination process. Most, 89% agreed that PI can be a useful tool for increasing more relevant and targeted knowledge translation, and increase the use of more creative and innovative communications channels. Most (79%) claimed that the research dissemination in RDs is more difficult than for the more common diseases, due to lack of researchers and societal interest (see Fig. 4).
In the open-ended questions several researchers claimed that patients can assist in translating and communicating the research in a more accessible language, thus reaching out to a wider community more efficiently. Several stated that more appropriate communication can increase the understanding and awareness of the patients` needs, which in turn can lead to more relevant research projects on rare diseases.
When involving the patients, the research have more impact, as the researcher is grounded in an understanding and prioritization of patients need (researchers)

Aim 1: The review of pertinent literature The characteristics of the studies
Despite that PI-RDR is recognized as a valuable methods for increasing the quality and relevance of research, the studies on PI-RDR is very limited. Only 13 articles were included in this review, indicating that RDs still is much less studied than common diseases [3]. Most of the included studies were from the Western Countries, and more than half have been published the last three years. The cultural differences in involving patients in the research process may be of interest for further studies. All included articles emphasized the importance of examining and sharing their experiences of PI-RDR, its impact and address gaps. Most studies reported several positive benefits of PI-RDR, but the heterogeneity of PImethods and methods for assessment of the impact of PI were large, and in most articles poorly described.

The reported impact of PI
In all the included articles both researchers and patients had cited positive impacts of PI; on the research itself, with the design, executing and dissemination as more applicable to those populations the research were intended to serve. Commonly reported impact of PI were more relevant research questions, improved design, methodology and increased recruitment rates, similar to reported in studies of more common diseases [40][41][42]. Some also described the empowerment aspects of PI-RDR, such as transformational learning outcomes both for researchers and patients, aligning with studies of more common diseases [43]. Indicating that transformational learning may occur through the process of involving patients in research.
Some also emphasized that PI-RDR may be a method to better deal with the particular challenges related to rare diseases research such as lack of interest, financing, awareness and policies support. When the number and size of research studies are small, as in RDs research, targeted outcomes and dissemination seems of particular importance [44][45][46][47][48]. To overcome this barriers Gagnon et al. [3] have introduced the Knowledge Mining methodology Framework (RKMMF) to improve the development of knowledge translation products and dissemination in RDs for developing clinical practice guidelines and evidence based practice. This methodological framework emphasize the patients` experiences as an important source of information, and can be a rigorous and flexible framework that can be adapted to the specific context of many rare diseases [3]. The authors [3] suggest that involving patients and end-users as early as possible may improve the dissemination products and meet the endusers need and expectations. This in accordance with some of the included studies that indicate that PI-RDR in dissemination may increase the acceptability, utility and relevance of the study results, and may increase the public awareness and policy support of RDs.

Methods for assessing the impact of patient involvement
Even though the international rare disease research network emphasize the importance of involving patients in research, [12,34,49,50], our results indicate that the Fig. 4 The utility of PI in research dissemination scientific proof of the impact and efficiency of involving patients in research is limited. The included studies did not go into detail in analyzing the nature of the process of PI that leads to impact. Few studies described strategies that generated desired or reported outcome. The transferability, strength of impact and relevance of the findings were often not extensively discussed. Preconceptions and founding values frequently seems to be confirmed, reappearing as results without substantial discussion and methodological reflection. Only one study [35] used a validated tool for measuring the process and impact of PI, most studies used reflection and discussion methodology and included minimal description of on how the assessments were conducted. The quality of the involvement process is a key [14,51]. There is only with rigorous evidence about the impact of PI-RDR the researchers can makes strides towards exploring the best recipes for PI-RDR. Appropriate methodology for impact assessment seems to be lacking [14,43,51]. Therefore, more validated instruments and methods for assessing the process and impact of PI seems warranted. More systematic assessment of the different conceptual frameworks of PI-RDR could provide opportunities to evaluate and compare across a range of study design.

Aim 2: The researchers' perceptions and experiences with patient involvement Experiences and challenges with rare diseases research
The results from our cross-sectional mixed methods study showed that nearly all researchers reported that research on RDs is more challenging than for the more common diseases. This align with other studies [3,8,37,46], indicating that research on RDs poses unique challenges related to choice of study design, analyses, recruitment and assessment of the representativeness for the entire population. In RDs research, the most obvious challenge to conduct rigorous research is the small number of eligible participants [52,53]. Other challenges reported were the heterogeneity between and within rare diseases, the geographical dispersion of patients, lack of knowledge about clinical course and lack of appropriate treatments [52,53]. Several of the researchers had proposals for improving the study quality by increasing the patient population by international collaboration. The European Union (EU) has put much effort into funding rare diseases research, encouraging national funding organizations to collaborate together in the E-Rare program, setting up European References Networks for rare diseases, and initiating the International Rare Diseases Research Consortium (IRDiRC) together with the National Institutes of health in the USA [54]. The international collaboration in RDs may be the key to improve the life of people with RDs [54].

Experiences and perception of patient involvement in rare diseases research
Many of the researchers had experiences with PI-RDR, and nearly all emphasized the particular importance of PI in rare diseases research. They reported benefits such as: more relevant research questions and meaningful research, which may lead to improved study design, more effective methods, increased recruitment, higher quality of data, improved interpretation and higher likelihood of translation and implementation of the research into everyday clinical practice. This is similar to reported in the articles included in the review part of this article. Nevertheless, the researchers emphasized that there are also several challenges related to PI-RDR. Some reported difficulties in educating patients for equal research participation, this may create power imbalance between researchers and patients. Power imbalance or lack of real involvement may result in tokenism, indicating only making perfunctory or symbolic effort to involve the patients [55]. Some of the researchers also specified that PI is resource-intensive, time-consuming and logistically demanding, therefore the needs to assess the effectiveness and long-term utility impact of PI-RDR also was emphasized.
When asking the researchers to define PI, their perception of the concept greatly varied. Only one third of the researchers perceived PI as actively involving patient in all phases of the research and dissemination process, similar to the definition of INVOLVE [10]. This may indicate that PI is a complex phenomenon lacking a clear theory and definition, and impose problems because many fail to define PI. In a study of Boaz et al. [56] researchers were asked about their attitude towards PI. This study found that attitudes ranged from positive expectation that PI improves research, to PI being something that needs to be done to comply with a formal demand. Even if the patients are willing to get involved, the effective realization of PI can be hampered by different ideas of PI, or misinterpretation of aims and expectations among and between patients and researchers [56]. A mismatch can inhibit successful involvement. Therefore a common understanding of the concept of PI-RDR is important. Another aspect emphasized by the researchers in our study was that PI is not suitable for all types of studies, and it is important that PI-RDR is meaningful both for researches and the patients. Therefore planning PI in an early stage of the research process and finding appropriate PI-RDR methods were emphasized of several researchers in our study.

Implication for further research
Despite that both the literature and the researchers claim several benefits of PI-RDR, our study indicate that there are some challenges related to PI that need further research. One challenge is that the concept of PI-RDR is complex and lacking a clear definition and theory. This lack of a clear definition may impose a problem, because many studies fail to define PI. Another challenge is the heterogeneity of PI methodology. PI is not a simple activity and it takes many forms and operate on many levels, and are highly contextual. It is very hard to standardize a PI-RDR method, and it is not a single simple intervention testable. Nevertheless, many studies describe that they have involved patients in all stages of the research circle, and experienced great benefits of PI-RDR. It had been advantageous if the methodology and approaches in these studies had been described in more detail. A third challenge is that validated measurements methods for assessing the impact of PI seems to be lacking. Most studies included in our review described positive benefits of PI-RDR, but very few described how they had measured these impacts. This may indicate that existing measures and approaches to evaluate the impacts of PI on the conduct of research and particularly on research outcome are limited. It is only by careful measurement of the impact of PI in research we truly can identify whether it ultimately benefits rare disease patients and improve the health care. Therefore more research on approaches and methodology of PI and PI-measurement in research is warranted, and particularly on PI in rare diseases research.

Limitations and strengths
Our scoping review inclusion criteria, restricted to English, French, German and Nordic languages may have lost important references, however the broad search in several databases may have compensated somewhat for this. In addition, PI is not a well-defined concept and we may have missed some studies using other terminology than included in our extensive search terms. The PRISMA-ScR was followed, but the included articles were not assessed regarding risk of bias, and thereby the results should be interpreted with caution.
The cross-sectional survey of the researchers` experiences may have several potential biases. We were unable to do analysis on responders versus non-responders, due to lack of information in ensuring the anonymity of the participants. There may also be a possibility that we have not reached all employees working in the nine Center of Rare Diseases, however, the response rate was high. Study-specific questionnaires may imply possibilities for misunderstanding the questions. Nevertheless, the construction of a questionnaire was deemed necessary, as no validated questionnaire existed for the purpose of this study. The pilot testing of the questionnaire in the Research Department of Sunnaas Rehabilitation hospital has probably decreased potential biases. Retrospective questions might also have introduced recall bias to the results. The use of Questback online survey program made it possible to feed the responses into the data analysis software automatically, and this avoids associated data transcription error. A strength was the combination of close-ended and open-ended questions, but the mixed method analyses and presentation of result are complex and require thoroughly methodological knowledge, something we believe the authors possess.

Conclusion
This study revealed that the research on PI in rare disease research is limited, but both the studies researchers seems to emphasize that PI is particularly important in rare diseases research due to the unique challenges related to research on small heterogeneous populations. The results from the included articles in the review and researchers experiences poses many positive impact of PI such as improving the quality of the research, and improved clinical relevance, designs, acceptability, legitimation and dissemination. However, appropriate methodology for PI seems not clear, and needs further definition and research.
Despite this challenges, the positive attitudes and experiences of patients, researcher and society indicate that PI in rare diseases has great positive impact. We therefore believe the benefits of this way of doing research outweigh the challenges.