Developing and evaluating rare disease educational materials co-created by expert clinicians and patients: the paradigm of congenital hypogonadotropic hypogonadism
- COST Action BM1105,
- Corin Badiu1,
- Marco Bonomi2, 3,
- Ivan Borshchevsky4,
- Martine Cools5,
- Margarita Craen5,
- Cristina Ghervan6,
- Michael Hauschild7,
- Eli Hershkovitz8,
- Erik Hrabovszky9,
- Anders Juul10,
- Soo-Hyun Kim11,
- Phillip Kumanov12,
- Beatriz Lecumberri13,
- Manuel C. Lemos14,
- Vassos Neocleous15,
- Marek Niedziela16,
- Sandra Pekic Djurdjevic17,
- Luca Persani3, 18, 19,
- Franziska Phan-Hug7,
- Duarte Pignatelli20,
- Nelly Pitteloud21,
- Vera Popovic22,
- Richard Quinton23,
- Nicos Skordis24, 25,
- Neil Smith26,
- Magdalena Avbelj Stefanija27,
- Cheng Xu21,
- Jacques Young28 and
- Andrew A. Dwyer29Email author
© The Author(s). 2017
Received: 14 December 2016
Accepted: 6 March 2017
Published: 20 March 2017
Patients with rare diseases face health disparities and are often challenged to find accurate information about their condition. We aimed to use the best available evidence and community partnerships to produce patient education materials for congenital hypogonadotropic hypogonadism (CHH) and the olfacto-genital (Kallmann) syndrome (i.e., CHH and defective sense of smell), and to evaluate end-user acceptability. Expert clinicians, researchers and patients co-created the materials in a multi-step process. Six validated algorithms were used to assess reading level of the final product. Comprehensibility and actionability were measured using the Patient Education Materials Assessment Tool via web-based data collection. Descriptive statistics were employed to summarize data and thematic analysis for analyzing open-ended responses. Subsequently, translation and cultural adaption were conducted by clinicians and patients who are native speakers.
Co-created patient education materials reached the target 6th grade reading level according to 2/6 (33%) algorithms (range: grade 5.9–9.7). The online survey received 164 hits in 2 months and 63/159 (40%) of eligible patients completed the evaluation. Patients ranged in age from 18 to 66 years (median 36, mean 39 ± 11) and 52/63 (83%), had adequate health literacy. Patients scored understandability at 94.2% and actionability at 90.5%. The patient education materials were culturally adapted and translated into 20 languages (available in Additional file 1).
Partnering with patients enabled us to create patient education materials that met patient- identified needs as evidenced by high end-user acceptability, understandability and actionability. The web-based evaluation was effective for reaching dispersed rare disease patients. Combining dissemination via traditional healthcare professional platforms as well as patient-centric sites can facilitate broad uptake of culturally adapted translations. This process may serve as a roadmap for creating patient education materials for other rare diseases.
KeywordsCongenital hypogonadotropic hypogonadism Kallmann syndrome Rare diseases E-health Community based participatory research Patient education Patient participation Patient-centered care Nursing
The landmark 2009 report from the European Organization for Rare Diseases (EURORDIS) brought to light the many challenges faced by patients with rare diseases . Delays in diagnosis, difficulty finding information about their condition and inadequate access to expert care are frequent patient experiences. Indeed, some have posited that living with a rare disease places one in the realm of health disparities . Physical and psychological morbidity can be significant and feelings of isolation and powerlessness can further undermine quality of life . Importantly, potential means to overcome these challenges include using the internet to connect dispersed patients with expert care and community engagement to help empower patients who feel marginalized by the healthcare system [4–6].
One such rare disorder is congenital hypogonadotropic hypogonadism (CHH, ORPHA174590). Based on a study of French conscripts, CHH occurs in approximately one in 4,000-10,000 . It is clinically characterized by incomplete (or absent) puberty and infertility resulting from insufficient secretion or action of gonadotropin releasing hormone (GnRH) - the master hormone of the reproductive axis . Genetic defects that affect olfactory axon and olfactory bulb development, thus leading to absent or defective sense of smell (anosmia or hyposmia), usually also affect the migration of neuroendocrine GnRH cells from the nose to the brain during embryonic life, thus leading to CHH. This condition is known as the olfacto-genital syndrome or Kallmann syndrome (ORPHA478). Some genetic and phenotypic overlap may exist between isolated CHH and Kallmann syndrome, yet patients with this syndrome are much more likely to exhibit additional non-reproductive phenotypes (i.e., skeletal defects, renal agenesis, cleft lip/palate, deafness, etc) than normosmic CHH patients. In the context of an international network of leading clinicians/geneticists/researchers focused on CHH , we have previously developed patient partnerships and conducted a needs assessment that leveraged engagement with patient support groups, social media and online data collection . In parallel, we developed a web-based platform  with resources for patients to find expert clinicians and peer-to-peer support. Additionally, consensus guidelines for the diagnosis and treatment of CHH were created using an evidence-based approach .
Engagement and co-creation have been effectively used in diverse fields including business, design and computer science (i.e. user-centered design) as a means to spur innovation, adoption and foster sustainability [11, 12]. Therefore, the aim of the present study was to partner clinical experts and patients to co-create high-quality patient education materials (PEM) that respond to the issues and questions most important and relevant to patients. Secondary aims included evaluating the readability of the PEM and end-user acceptability (i.e. understandability and actionability) as well as to disseminate these materials widely across different countries and cultures.
To assess reading level of the produced PEM, we subjected the final version to several validated measures evaluating readability: Flesch Reading Ease Formula (evaluates sentence length and number of syllables per word), Flesch Kincaid Grade Level (converts the Flesch reading ease formula to a grade level), Gunning Fox Index (calculates a weighted average of the number of words per sentence and long words to determine grade level), Coleman Liau Index (uses number of characters rather than syllables to determine grade level), Simple Measure of Gobbledygook (SMOG, a modification of the Gunning-Fog Index it calculates grade level based on the number of words with 3 or more syllables) and the Automated Readability Index (ratio of difficult words and sentences to provide an estimated age range and grade level) .
To evaluate end-user perspectives of adults with CHH (18 years and older), we used an online data collection (SurveyGizmo™) and recruited a convenience sample of patients via postings on closed/private CHH social media group (Facebook™), as well as notifications in patient support group meetings and RareConnect . This social media approach has been previously shown to be an effective means of recruitment for this rare disease patient population . The survey included questions on patient demographics, past healthcare interactions and a brief assessment of healthcare literacy that has been validated against longer gold-standards metrics [16, 17].
After reviewing a pdf of the PEM, participants were asked to complete the Patient Education Materials Assessment Tool (PEMAT). This instrument was developed and validated by the U.S. Department of Health & Human Services Agency for Health Research & Quality to evaluate print and audiovisual educational materials . The unique aspect of the PEMAT is that it incorporates other additional elements that are not assessed in traditional readability formulas. Patients select agree, disagree or not-applicable for 17 items relating to understandability (the ability to process key messages) and 7 items on actionabilty (the ability to identify what one can do to manage their condition). Items rated as agree are given a score of 1, disagree 0 and cumulative scores are expressed as a percentage (total score/possible total X 100). Initial psychometric evaluation of the PEMAT has demonstrated strong internal consistency, good reliability, and initial evidence of construct validity . Survey respondents were also given an opportunity to provide free text comments (i.e. critiques and suggestions) after completing the PEMAT questions.
The survey was alpha tested by patients in two rounds to identify and correct any bugs prior to online launch. Descriptive statistics were used to report summary findings. To assess for potential response bias, Student’s t test and Chi square test were used to compare demographic characteristics of patients who completed the evaluation with those who did not (partial completion). Thematic analysis  was employed to codify and analyze open-text responses NVivo11 (QSR International PSY Ltd., Melbourne Australia). The study was reviewed and approved by the ethics committee of the University of Lausanne and participants provided opt-in online consent.
The final step of this process was to disseminate the PEM to reach the broadest possible audience. This included using native speakers (i.e. expert clinicians, medical translators) from across the European network to provide versions in multiple languages. Particular attention was given to finding appropriate terms and examples for the translated PEM to make them culturally sensitive and not simply verbatim translations. The final materials will be distributed via traditional means to reach healthcare professionals (i.e. peer-review publication, professional meetings, individual providers’ websites, and via the COST Action website ). In parallel PEMs will be distributed via patient support groups including online social media (Facebook™, Twitter™, patient blogs) and publicized on internet platforms targeting the rare disease community including the EURORDIS initiative RareConnect .
Patient partnerships were used to identify key topics and to target issues most important to patients as well as to contribute content. A working group of the network (Patient Advocacy Working Group) created a topic list based on the consensus statement guidelines . Additional items were drawn from focus group discussions with patients held in the context of patient support meetings (organized with patient leaders) as part of the prior needs assessment  (Fig. 1A). Patient collaborators also contributed lists of “frequently asked questions” as well as topics that were recurrent in social media threads and chat room discussions. Common questions include: what causes CHH? Why didn’t I go through puberty? Why can’t I smell? Is it curable? Can I have children? Will my children have CHH? (see Additional file 1). The drafted material version 1.0 (V1.0) went through two subsequent revisions to refine language, wording and selection of images via email and the PEM was finalized in a face-to-face meeting prior to vetting with the broader network (Fig. 1D). During this development process it was sometimes challenging to balance the input and feedback from clinicians and patients to find the right balance and depth of information provided. Indeed, the face-to-face meeting was valuable for arriving at consensus as opinions were conflicting at times during the process and this was not easy to reconcile via email.
Summary of readability of co-created patient education material
Summary statistics (5 pages, 203 sentences + images)
Word count = 1,654
Character count = 8,251
Complex words (≥3 syllables) = 268
Average characters per word = 5
Average words per sentence = 8.1
Flesch Reading Ease
Flesch-Kincaid Reading Level
Gunning Fog Index
Coleman Liau Index
Automated Readability Index
Patient characteristics (n = 63)
Seen at academic center
Had genetic testing
Received genetic counseling
PEMAT Understandability and actionability of co-created materials (n = 63)
The material makes its purpose completely evident.
The material does not include information or content that distracts from its purpose.
Word choice & style
The material uses common, everyday language.
Medical terms are used only to familiarize the audience with the terms.
When used, medical terms are defined.
The material uses the active voice (e.g. action verbs).
Use of numbers
Numbers appearing in the material are clear and easy to understand.
The material does not expect the user to perform calculations.
The material breaks or “chunks” information into short sections.
The material’s sections have informative headers.
The material presents information in a logical sequence.
The material provides a summary.
Layout & design
The material uses visual cues to draw attention to key points.
The material uses visual aids whenever they could make content more easily understood.
The material’s visual aids reinforce rather than distract from the content.
The material’s visual aids have clear titles or captions.
The material uses illustrations and photographs that are clear and uncluttered.
The material uses simple tables with short and clear row and column headings.
The material clearly identifies at least one action the user can take.
The material addresses the user directly when describing actions.
The material breaks down any action into manageable, explicit steps.
The material provides a tangible tool (e.g. checklists) whenever it could help the user take action.
The material explains how to use the charts, graphs, tables, or diagrams to take actions
The material uses visual aids whenever they could make it easier to act on the instructions.
Patient comments (n = 52) regarding the co-created materials
n = 19
• “I was very impressed and I think my friends and family will find it easy to understand”
• “I am glad that there is a clear male/female explanation. Often materials I find focus predominantly on the males”
n = 11
• “Elaborate on infertility and treatment”
• “I think you may be under discussing the life-long emotional and psychological impact”
n = 10
• “It felt like the pages were a bit full but I can appreciate it must have been hard to provide all the information necessary in only 5 pages”
• “I think if there was a way to click on each section for more detailed info that might help”
n = 9
• “Other rare conditions that can also be evident in patients with KS/CHH… …Explain that patients can have additional illnesses besides KS/CHH”
• “It might be helpful to state that anosmia is permanent. Of course it is most important to focus on the hormonal component, but there are definitely considerations in dealing with anosmia as well”
n = 3
• “It would be great if you could translate into several languages to reach more people”
Native speakers from across the European network made culturally adapted translations. In some instances local patients contributed to this translation and adaption process. The translated PEMs required cultural adaptation in some instances to help make them more relevant for the target audience. For instance, a small cherry was used to describe the size of the pituitary gland in the Hungarian version, the Chinese version was altered as “what you should know” was not culturally appropriate, and terms describing depression were adapted in the Polish version to enhance comprehension by the lay public. Every effort was made to keep the entire content of the PEM in the translated versions. When text length expanded the images were adjusted accordingly to maintain a 5-page document. PEM are now available in 20 languages: English, Bulgarian, Chinese, Danish, Dutch, French, German, Greek, Hebrew, Hungarian, Italian, Korean, Polish, Portuguese, Romanian, Russian, Serbian, Slovenian, Spanish, and Turkish (Additional file 2). Dissemination plans will target healthcare professionals and patient-centric avenues such as social media and patient support sites.
The aim of this study was to engage patients and co-create PEM that respond to what matters most to patients. Subsequently, we evaluated the readability and end-user acceptability of the PEM and sought to widely disseminate the translated PEM across different countries and cultures. Patients living with a rare disease face health disparities  and patient engagement has been identified as potential means to empower this patient population [4–6]. Interestingly, patient engagement has recently been gaining attention in the context of orphan drug development . However, the extent of patient engagement varies widely. A 2014 systematic review of patient engagement for research on rare diseases found engagement is typically unidirectional - involving patients in consultative roles and rarely in creative aspects or in terms of dissemination . The present study is unique in that we used a participatory process to co-create PEM with patients; we then evaluated the PEM produced by this collaboration, and worked with patient groups to facilitate dissemination to the largest possible audience.
We previously partnered with online patient community leaders to identify the unmet health and informational needs of patients with congenital hypogonadotropic hypogonadism (CHH) and Kallmann syndrome . In the present study, the partnership was more clearly bi-directional as patients were not simply providers of opinions; rather they contributed directly in co-creating the PEM in an iterative process. Notably, patient knowledge and expertise emerges from the day-to-day experiences of living and coping with a rare condition and therefore is inherently different from the expertise of healthcare professionals . Recently, a study examining online exchanges among patients with rare adrenal disorders found that information and support were central elements in peer-to-peer exchanges . Moreover, the authors noted that patient-centered care could be enhanced by better integrating patient knowledge with the care provided by professionals. In the present study, developing the PEM was a true partnership that recognized patient expertise as unique and complementary to expert clinician knowledge. We believe that this co-creation contributed to the high acceptability ratings by patients.
This evaluation process of the co-created PEM has limitations. The evaluation was only conducted on the English version. As such, the findings are not completely transferable to the other translated versions despite the inclusion of patients in developing some of the translations. Moreover, the additional validation step of back translating the other versions was not conducted and this could be viewed as a limitation. We only assessed readability once the materials had been finalized, not during the development process. In future studies, this testing could be incorporated earlier in the development process to improve the reading level of developed PEM. While the evaluation was overwhelmingly positive and a fairly sizeable sample was reached (for a rare disease population), the patients completing the evaluation were quite well-educated and exhibited high levels of health literacy. Accordingly, our ability to draw inferences to a broader population of lower literacy patients is limited. This may reflect a bias of using a web-based survey - as perhaps those using the web may have higher literacy levels. However, recruiting sufficient numbers of patients for rare disease studies has been a long-standing challenge [27, 28]. Therefore, we used a web-based approach to overcome this barrier but note that such an approach entails a potential risk of bias.
The Pew Foundation’s published report on health and the internet indicates that patients living with a rare disease are internet power users who are most likely to seek information about their condition online and find support from other patients using social media . Based on our previous success combining patient partnerships and social media for the online needs assessment , we employed a similar approach in the present study to reach a relatively large sample (n = 63) over 8-weeks. These experiences suggest that web-based platforms are an effective means to reach and connect rare disease patients. Thus, the opportunities afforded by the internet and social media may provide novel avenues for crowdsourcing solutions as well as offering a shared venue for either clinician- or patient-led collaborations to improve quality and add value to the healthcare system [5, 30]. The European Union Committee of Experts on Rare Diseases (EUCERD) recommendations for Centers of Expertise underscore the importance of collaboration with patient organizations to provide information that is at once accessible and adapted to patient needs . For many rare diseases, such as CHH/Kallmann syndrome, formal organized patient support organizations do not exist. As such, web-based approaches and social media provide a critical means to broadly reach patients, identify priorities and incorporate their perspectives and knowledge into care. This may be particularly advantageous in light of the movement to form European Reference Networks for rare diseases [32, 33].
The final step in this co-creation process was to engage in bi-directional dissemination. This has been identified as a shortcoming in much of the patient engagement research conducted in the context of rare diseases . Through the work of members of the Network and patients alike, materials were adapted and translated into 20 languages by native speakers. This collaborative process is essential for ensuring that information provided to patients is culturally adapted and sensitive – a key element for Centers of Expertise . In parallel to traditional healthcare professional outlets (e.g. scientific meetings, peer-review publication) patient participants are distributing materials directly to other patients via social media and postings on centralized patients sites . The co-created PEM (in multiple languages) is a critical component of the list of patient resources available on the website of the European network comprising a virtual empowerment toolkit for patients and families . Available information includes listings of international specialized referral centers, genetic testing labs, clinical trials, and peer-to-peer support as well as a portal for a patient registry. We are utilizing both professional-oriented avenues and more patient-oriented social media outlets to hopefully reach unprecedented numbers of patients and clinicians and overcome traditional roadblocks of implementation into practice [34–36].
Partnering with patients enabled co-creation of high-quality PEM while social media and web-based data collection facilitated timely evaluation by a dispersed patient population. We believe that partnering with expert patients was an empowering experience and provides valuable contributions for developing patient-centered approaches to care. We envision this work will serve as a roadmap for those wishing to engage in a co-creation process and will help inform projects aimed at improving care for patients living with a rare disease.
Congenital hypogonadotropic hypogonadism
European Union Committee of Experts on Rare Diseases
European Organization for Rare Diseases
European Cooperation in Science and Technology
Patient education materials
Patient education materials assessment tool
Simple measure of Gobbledygook
The authors wish to thank the patients who generously gave their time in helping to develop the patient education materials and we acknowledge the Working Group members of COST Action BM1105 for their contributions. Special thanks to Ms. Virginia Hughes for her critical input during the development process, Prof. Linan Cheng of the Shanghai Institute of Planned Parenthood Research for her validation of the Chinese translations and Ms. Özlem Firat for her help in coordinating the translated materials. This work was supported by the Rare Disease Foundation and the BC Children’s Hospital Foundation (# 17-26) and COST Action BM1105.
This work was supported by the COST Action BM1105 and received funding from the Rare Disease Foundation and the BC Children’s Hospital Foundation (# 17-26).
Availability of data and materials
Data sharing not applicable to this article as no datasets were generated or analysed during the current study.
AD conceived the study and participated in study design, conduct, analyses and drafted the manuscript. RQ, NP and NS helped draft the educational materials with patients. All authors helped revise the materials and contributed to developing culturally relevant translations. All authors read and approved the final manuscript.
The authors have no financial or non-financial competing interests to declare.
Consent for publication
Ethics approval and consent to participate
This study (protocol #233/13) was reviewed and approved by the Commission Cantonale d’ethique de la recherche sur l’être humain which is the institutional review board (Ethics Committee) associated with the University of Lausanne. All survey participants provided opt-in electronic consent prior to completing the online evaluation.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- EURORDIS. The voice of 12,000 patients: Experiences and expectations of rare disease patients on diagnosis and care in Europe. France: Boulogne-Billancourt; 2009.
- Holtzclaw Williams P. Policy framework for rare disease health disparities. Policy Polit Nurs Pract. 2011;12:114–8.View ArticlePubMedGoogle Scholar
- Cohen JS, Biesecker BB. Quality of life in rare genetic conditions: a systematic review of the literature. Am J Med Genet A. 2010;152A:1136–56.View ArticlePubMedPubMed CentralGoogle Scholar
- Ayme S, Kole A, Groft S. Empowerment of patients: lessons from the rare diseases community. Lancet. 2008;371:2048–51.View ArticlePubMedGoogle Scholar
- Vicari S, Cappai F. Health activism and the logic of connective action. A case study of rare disease patient organisations. Inf Commun Soc. 2016;19:1653–71.View ArticlePubMedPubMed CentralGoogle Scholar
- Sofolahan-Oladeinde Y, Mullins CD, Baquet CR. Using community-based participatory research in patient-centered outcomes research to address health disparities in under-represented communities. J Comp Effectiveness Res. 2015;4:515–23.View ArticleGoogle Scholar
- Fromantin M, Gineste J, Didier A, Rouvier J. Impuberism and hypogonadism at induction into military service. Statistical study. Problemes Actuels d’Endocrinologie et de Nutr. 1973;16:179–99.Google Scholar
- Boehm U, Bouloux PM, Dattani MT, de Roux N, Dode C, Dunkel L, Dwyer AA, Giacobini P, Hardelin JP, Juul A, et al. Expert consensus document: European Consensus Statement on congenital hypogonadotropic hypogonadism--pathogenesis, diagnosis and treatment. Nat Rev Endocrinol. 2015;11:547–64.PubMedGoogle Scholar
- COST Action BM1105 [http://www.gnrhnetwork.eu/]. Accessed 14 Mar 2017.
- Dwyer AA, Quinton R, Morin D, Pitteloud N. Identifying the unmet health needs of patients with congenital hypogonadotropic hypogonadism using a web-based needs assessment: implications for online interventions and peer-to-peer support. Orphanet J Rare Dis. 2014;9:83.View ArticlePubMedPubMed CentralGoogle Scholar
- Crawford MJ, Rutter D, Manley C, Weaver T, Bhui K, Fulop N, Tyrer P. Systematic review of involving patients in the planning and development of health care. BMJ. 2002;325:1263.View ArticlePubMedPubMed CentralGoogle Scholar
- Greenhalgh T, Jackson C, Shaw S, Janamian T. Achieving research impact through co-creation in community-based health services: literature review and case study. Milbank Q. 2016;94:392–429.View ArticlePubMedPubMed CentralGoogle Scholar
- Wallerstein N, Duran B. Community-based participatory research contributions to intervention research: the intersection of science and practice to improve health equity. Am J Public Health. 2010;100 Suppl 1:S40–46.View ArticlePubMedPubMed CentralGoogle Scholar
- CheckText [http://www.readabilityformulas.com]. Accessed 14 Mar 2017.
- RareConnect [https://www.rareconnect.org]. Accessed 14 Mar 2017.
- Chew LD, Bradley KA, Boyko EJ. Brief questions to identify patients with inadequate health literacy. Fam Med. 2004;36:588–94.PubMedGoogle Scholar
- Chew LD, Griffin JM, Partin MR, Noorbaloochi S, Grill JP, Snyder A, Bradley KA, Nugent SM, Baines AD, Vanryn M. Validation of screening questions for limited health literacy in a large VA outpatient population. J Gen Intern Med. 2008;23:561–6.View ArticlePubMedPubMed CentralGoogle Scholar
- Shoemaker SJ, Wolf MS, Brach C, (Prepared by Abt Associates IuCNHI: The Patient Education Materials Assessment Tool (PEMAT) and User’s Guide. In Book The Patient Education Materials Assessment Tool (PEMAT) and User’s Guide (Editor ed.^eds.). Maryland: Agency for Healthcare Research and Quality; 2013.
- Shoemaker SJ, Wolf MS, Brach C. Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns. 2014;96:395–403.View ArticlePubMedPubMed CentralGoogle Scholar
- Saldana J. Coding manual for qualitative researchers. Thousand Oaks, CA: Sage; 2009.Google Scholar
- Safeer RS, Keenan J. Health literacy: the gap between physicians and patients. Am Fam Physician. 2005;72:463–8.PubMedGoogle Scholar
- Doak CC, Doak LG, Root JH. Teaching Patients with Low Literacy Skills. 2nd ed. Philadelphia, PA: J.B. Lippincott Company; 1996.Google Scholar
- Bronstein MG, Kakkis ED. Patients as key partners in rare disease drug development. Nat Rev Drug Discov. 2016;15:731–2.View ArticlePubMedGoogle Scholar
- Forsythe LP, Szydlowski V, Murad MH, Ip S, Wang Z, Elraiyah TA, Fleurence R, Hickam DH. A systematic review of approaches for engaging patients for research on rare diseases. J Gen Intern Med. 2014;29 Suppl 3:S788–800.View ArticlePubMedGoogle Scholar
- Hartzler A, Pratt W. Managing the personal side of health: how patient expertise differs from the expertise of clinicians. J Med Internet Res. 2011;13:e62.View ArticlePubMedPubMed CentralGoogle Scholar
- Kauw D, Repping-Wuts H, Noordzij A, Stikkelbroeck N, Hermus A, Faber M. The contribution of online peer-to-peer communication among patients with adrenal disease to patient-centered care. J Med Internet Res. 2015;17:e54.View ArticlePubMedPubMed CentralGoogle Scholar
- Kruer MC, Steiner RD. The role of evidence-based medicine and clinical trials in rare genetic disorders. Clin Genet. 2008;74:197–207.View ArticlePubMedGoogle Scholar
- Watson MS, Epstein C, Howell RR, Jones MC, Korf BR, McCabe ER, Simpson JL. Developing a national collaborative study system for rare genetic diseases. Genet Med. 2008;10:325–9.View ArticlePubMedGoogle Scholar
- Fox S. Peer-to-peer healthcare: many people - especially those living with chronic or rare diseases - use online connections to supplement professional medical advice. Washington, D.C.: Pew Internet; 2011.Google Scholar
- Nambisan P, Nambisan S. Models of consumer value cocreation in health care. Health Care Manag Rev. 2009;34:344–54.View ArticleGoogle Scholar
- (EUCERD) EUCoEoRD: Recommendations on quality criteria for centres of expertise for rare diseases in member states. In: Book Recommendations on Quality Criteria for Centres of Expertise for Rare Diseases in Member States (Editor ed.^eds.). City; 2011.
- Diseases EUCoEoR. Recommendations to the European Commission and the Member States on European Reference Networks for Rare Diseases. 2013.Google Scholar
- European Reference Network [http://ec.europa.eu/health/rare_diseases/european_reference_networks/erf/index_en.htm]. Accessed 14 Mar 2017.
- Sung NS, Crowley Jr WF, Genel M, Salber P, Sandy L, Sherwood LM, Johnson SB, Catanese V, Tilson H, Getz K, et al. Central challenges facing the national clinical research enterprise. JAMA. 2003;289:1278–87.View ArticlePubMedGoogle Scholar
- Thornicroft G, Lempp H, Tansella M. The place of implementation science in the translational medicine continuum. Psychol Med. 2011;41:2015–21.View ArticlePubMedGoogle Scholar
- Bellgarda MI, Sleemanc MW, Guerrerod FD, Fletcher S, Baynam G, Goldblatt J, Rubinstein Y, Bell C, Groft S, Barreroa R, et al. Rare disease research roadmap: navigating the bioinformatics and translational challenges for improved patient health outcomes. Health Policy Technol. 2014;3:325–35.View ArticleGoogle Scholar