Skip to main content

Developing and evaluating rare disease educational materials co-created by expert clinicians and patients: the paradigm of congenital hypogonadotropic hypogonadism



Patients with rare diseases face health disparities and are often challenged to find accurate information about their condition. We aimed to use the best available evidence and community partnerships to produce patient education materials for congenital hypogonadotropic hypogonadism (CHH) and the olfacto-genital (Kallmann) syndrome (i.e., CHH and defective sense of smell), and to evaluate end-user acceptability. Expert clinicians, researchers and patients co-created the materials in a multi-step process. Six validated algorithms were used to assess reading level of the final product. Comprehensibility and actionability were measured using the Patient Education Materials Assessment Tool via web-based data collection. Descriptive statistics were employed to summarize data and thematic analysis for analyzing open-ended responses. Subsequently, translation and cultural adaption were conducted by clinicians and patients who are native speakers.


Co-created patient education materials reached the target 6th grade reading level according to 2/6 (33%) algorithms (range: grade 5.9–9.7). The online survey received 164 hits in 2 months and 63/159 (40%) of eligible patients completed the evaluation. Patients ranged in age from 18 to 66 years (median 36, mean 39 ± 11) and 52/63 (83%), had adequate health literacy. Patients scored understandability at 94.2% and actionability at 90.5%. The patient education materials were culturally adapted and translated into 20 languages (available in Additional file 1).


Partnering with patients enabled us to create patient education materials that met patient- identified needs as evidenced by high end-user acceptability, understandability and actionability. The web-based evaluation was effective for reaching dispersed rare disease patients. Combining dissemination via traditional healthcare professional platforms as well as patient-centric sites can facilitate broad uptake of culturally adapted translations. This process may serve as a roadmap for creating patient education materials for other rare diseases.


The landmark 2009 report from the European Organization for Rare Diseases (EURORDIS) brought to light the many challenges faced by patients with rare diseases [1]. Delays in diagnosis, difficulty finding information about their condition and inadequate access to expert care are frequent patient experiences. Indeed, some have posited that living with a rare disease places one in the realm of health disparities [2]. Physical and psychological morbidity can be significant and feelings of isolation and powerlessness can further undermine quality of life [3]. Importantly, potential means to overcome these challenges include using the internet to connect dispersed patients with expert care and community engagement to help empower patients who feel marginalized by the healthcare system [46].

One such rare disorder is congenital hypogonadotropic hypogonadism (CHH, ORPHA174590). Based on a study of French conscripts, CHH occurs in approximately one in 4,000-10,000 [7]. It is clinically characterized by incomplete (or absent) puberty and infertility resulting from insufficient secretion or action of gonadotropin releasing hormone (GnRH) - the master hormone of the reproductive axis [8]. Genetic defects that affect olfactory axon and olfactory bulb development, thus leading to absent or defective sense of smell (anosmia or hyposmia), usually also affect the migration of neuroendocrine GnRH cells from the nose to the brain during embryonic life, thus leading to CHH. This condition is known as the olfacto-genital syndrome or Kallmann syndrome (ORPHA478). Some genetic and phenotypic overlap may exist between isolated CHH and Kallmann syndrome, yet patients with this syndrome are much more likely to exhibit additional non-reproductive phenotypes (i.e., skeletal defects, renal agenesis, cleft lip/palate, deafness, etc) than normosmic CHH patients. In the context of an international network of leading clinicians/geneticists/researchers focused on CHH [9], we have previously developed patient partnerships and conducted a needs assessment that leveraged engagement with patient support groups, social media and online data collection [10]. In parallel, we developed a web-based platform [9] with resources for patients to find expert clinicians and peer-to-peer support. Additionally, consensus guidelines for the diagnosis and treatment of CHH were created using an evidence-based approach [8].

Engagement and co-creation have been effectively used in diverse fields including business, design and computer science (i.e. user-centered design) as a means to spur innovation, adoption and foster sustainability [11, 12]. Therefore, the aim of the present study was to partner clinical experts and patients to co-create high-quality patient education materials (PEM) that respond to the issues and questions most important and relevant to patients. Secondary aims included evaluating the readability of the PEM and end-user acceptability (i.e. understandability and actionability) as well as to disseminate these materials widely across different countries and cultures.


PEM development

A community based participatory research framework was selected to guide the development of the patient education material (PEM) for its relevance to patient empowerment and health disparities [13]. The Patient advocacy Working group of the European network focused on CHH (COST Action BM1105, “GnRH Deficiency: Elucidation of the neuroendocrine control of human reproduction”) [9] worked closely with online patient community leaders (i.e. moderators of online patient support sites) to identify key PEM content areas and topics based on the most frequently asked questions on social media sites (Additional file 1) as well as from a previously conducted patient needs assessment [10]. Clinical information was drawn from the evidence-based consensus statement on the approach to diagnosis and treatment of CHH [8]. The PEM development was an iterative process (Fig. 1) involving multiple stakeholders including patients, patient support groups, clinicians and researchers spanning the fields of endocrinology, andrology, nursing and genetics. At each step, input and feedback were used to refine and modify the PEM.

Fig. 1
figure 1

Study Schema. PEM were co-created in a multi-step process. (a) Three main sources were used for PEM development. (b) Members of the Patient Advocacy Working Group and patient collaborators identified topics for the PEM in an iterative process. (c) The initial draft was created and revised based on patient input. (d) PEM (V2.0) was circulated to the Clinical Working Group and Genetics Working Group members for comment and revised accordingly with patient validation in two rounds. (e) PEM (V4.0) were evaluated by patients recruited via social media (private/closed Facebook groups), patient support meetings and via RareConnect [12]. (f) Following evaluation materials were culturally adapted and translated to 20 languages and distributed in avenues targeting healthcare professionals and patients. PEM: patient education materials, V: version

Readability assessment

To assess reading level of the produced PEM, we subjected the final version to several validated measures evaluating readability: Flesch Reading Ease Formula (evaluates sentence length and number of syllables per word), Flesch Kincaid Grade Level (converts the Flesch reading ease formula to a grade level), Gunning Fox Index (calculates a weighted average of the number of words per sentence and long words to determine grade level), Coleman Liau Index (uses number of characters rather than syllables to determine grade level), Simple Measure of Gobbledygook (SMOG, a modification of the Gunning-Fog Index it calculates grade level based on the number of words with 3 or more syllables) and the Automated Readability Index (ratio of difficult words and sentences to provide an estimated age range and grade level) [14].

End-user acceptability

To evaluate end-user perspectives of adults with CHH (18 years and older), we used an online data collection (SurveyGizmo™) and recruited a convenience sample of patients via postings on closed/private CHH social media group (Facebook™), as well as notifications in patient support group meetings and RareConnect [15]. This social media approach has been previously shown to be an effective means of recruitment for this rare disease patient population [10]. The survey included questions on patient demographics, past healthcare interactions and a brief assessment of healthcare literacy that has been validated against longer gold-standards metrics [16, 17].

After reviewing a pdf of the PEM, participants were asked to complete the Patient Education Materials Assessment Tool (PEMAT). This instrument was developed and validated by the U.S. Department of Health & Human Services Agency for Health Research & Quality to evaluate print and audiovisual educational materials [18]. The unique aspect of the PEMAT is that it incorporates other additional elements that are not assessed in traditional readability formulas. Patients select agree, disagree or not-applicable for 17 items relating to understandability (the ability to process key messages) and 7 items on actionabilty (the ability to identify what one can do to manage their condition). Items rated as agree are given a score of 1, disagree 0 and cumulative scores are expressed as a percentage (total score/possible total X 100). Initial psychometric evaluation of the PEMAT has demonstrated strong internal consistency, good reliability, and initial evidence of construct validity [19]. Survey respondents were also given an opportunity to provide free text comments (i.e. critiques and suggestions) after completing the PEMAT questions.

Statistical analyses

The survey was alpha tested by patients in two rounds to identify and correct any bugs prior to online launch. Descriptive statistics were used to report summary findings. To assess for potential response bias, Student’s t test and Chi square test were used to compare demographic characteristics of patients who completed the evaluation with those who did not (partial completion). Thematic analysis [20] was employed to codify and analyze open-text responses NVivo11 (QSR International PSY Ltd., Melbourne Australia). The study was reviewed and approved by the ethics committee of the University of Lausanne and participants provided opt-in online consent.


The final step of this process was to disseminate the PEM to reach the broadest possible audience. This included using native speakers (i.e. expert clinicians, medical translators) from across the European network to provide versions in multiple languages. Particular attention was given to finding appropriate terms and examples for the translated PEM to make them culturally sensitive and not simply verbatim translations. The final materials will be distributed via traditional means to reach healthcare professionals (i.e. peer-review publication, professional meetings, individual providers’ websites, and via the COST Action website [9]). In parallel PEMs will be distributed via patient support groups including online social media (Facebook™, Twitter™, patient blogs) and publicized on internet platforms targeting the rare disease community including the EURORDIS initiative RareConnect [15].


PEM development

Patient partnerships were used to identify key topics and to target issues most important to patients as well as to contribute content. A working group of the network (Patient Advocacy Working Group) created a topic list based on the consensus statement guidelines [8]. Additional items were drawn from focus group discussions with patients held in the context of patient support meetings (organized with patient leaders) as part of the prior needs assessment [10] (Fig. 1A). Patient collaborators also contributed lists of “frequently asked questions” as well as topics that were recurrent in social media threads and chat room discussions. Common questions include: what causes CHH? Why didn’t I go through puberty? Why can’t I smell? Is it curable? Can I have children? Will my children have CHH? (see Additional file 1). The drafted material version 1.0 (V1.0) went through two subsequent revisions to refine language, wording and selection of images via email and the PEM was finalized in a face-to-face meeting prior to vetting with the broader network (Fig. 1D). During this development process it was sometimes challenging to balance the input and feedback from clinicians and patients to find the right balance and depth of information provided. Indeed, the face-to-face meeting was valuable for arriving at consensus as opinions were conflicting at times during the process and this was not easy to reconcile via email.

Readability levels

Readability was evaluated using 6 different validated algorithms that are widely used to assess reading level (Table 1). These employ different formulas that use word length and complexity (i.e. the number of characters or syllables in words, sentence length) to calculate an average grade level needed to understand the material. Most patients read at an 8-9th grade reading level [21]. However, expert recommendation has identified the target reading level at 6th grade (i.e. an 11 year-old child) [22]. Two of the six algorithms scored the PEM at the target grade level (Flesch Kincaid Grade Level: 5.9, Automated Readability Index: 6.1). The mean grade level across instruments was 8.0 (Table 1), indicating that more work could be done to enhance readability. However, one challenge in doing this is the number of complex words (i.e. hypogonadism, cryptorchidism, infertility, etc) that were deemed important by clinicians and patients alike to include and define in lay terms.

Table 1 Summary of readability of co-created patient education material


Following patient alpha testing to identify and correct bugs in the online evaluation, the survey was launched and remained open for 8-weeks. During this period, 164 hits were registered. In total, 38 (23%) were “one-click” entries who passed the opt-in consent but did not enter demographic information. Responses of five participants were excluded (age <18 years). More than a third of respondents (58/164, 35%) partially completed the evaluation (i.e. demographics up to viewing the PEM) and 63 (38%) completed the entire PEM evaluation (Fig. 1E). Characteristics of survey respondents are depicted in Table 2. Notably, the predominance of male responders (2:1) is keeping with the striking sexual discordance in CHH [8]. Overall, patients were well-educated (46/63, 73% achieving university or higher) and by-and-large exhibited adequate health literacy (52/63, 82%). Notably, the mean age of diagnosis was 20.9 ± 6.4 years (range: 10–40, median 19) suggesting that many patients are diagnosed quite late. In terms of prior healthcare interactions, more than half (39/63, 62%) had either a consultation or had received care at a specialized academic center. In total, 36/63 (56%) had undergone genetic testing yet only 12/63 (19%) reported having had genetic counseling. We found no significant differences between those who completed the evaluation and the partial completers in terms of age (p = 0.30), sex (p = 0.37), education (p = 0.94), health literacy (p = 0.15), or being seen at an academic center (p = 0.09).

Table 2 Patient characteristics (n = 63)

End-user acceptability

Patients gave the co-created PEM high scores on understandability (range: 88.9–97.5%, total mean: 94.2%) which includes content, word choice/style, use of numbers, organization, layout/design and visual aids (Table 3). The lowest rating (88.9%) was linked with being uncluttered which was commented on in the free text field by three patients (i.e. having more white space). Similarly, patients gave high scores on actionability (overall mean: 90.5%). The lowest score was assigned to explaining how to use charts, graphs, or diagrams to take action and manage the condition. Together the high scores on both understandability and actionability indicate high end-user acceptability.

Table 3 PEMAT Understandability and actionability of co-created materials (n = 63)

Overall we received comments from 45/63 (71.4%) patients. Comments were coded according to themes and sorted into categories. In total, 52 concepts were identified from the 45 comments clustering into five categories (Table 4). The most frequent sentiments were expressions of thanks/approval (n = 19, 37%) followed by content (i.e. treatment, infertility, and psychological aspects) n = 11 (21%), format (i.e. use of simple language, spacing) n = 10 (19%), personal concerns (n = 9, 17%) and three comments underscored the importance of translating the PEM to make it available to more patients.

Table 4 Patient comments (n = 52) regarding the co-created materials

Broad dissemination

Native speakers from across the European network made culturally adapted translations. In some instances local patients contributed to this translation and adaption process. The translated PEMs required cultural adaptation in some instances to help make them more relevant for the target audience. For instance, a small cherry was used to describe the size of the pituitary gland in the Hungarian version, the Chinese version was altered as “what you should know” was not culturally appropriate, and terms describing depression were adapted in the Polish version to enhance comprehension by the lay public. Every effort was made to keep the entire content of the PEM in the translated versions. When text length expanded the images were adjusted accordingly to maintain a 5-page document. PEM are now available in 20 languages: English, Bulgarian, Chinese, Danish, Dutch, French, German, Greek, Hebrew, Hungarian, Italian, Korean, Polish, Portuguese, Romanian, Russian, Serbian, Slovenian, Spanish, and Turkish (Additional file 2). Dissemination plans will target healthcare professionals and patient-centric avenues such as social media and patient support sites.


The aim of this study was to engage patients and co-create PEM that respond to what matters most to patients. Subsequently, we evaluated the readability and end-user acceptability of the PEM and sought to widely disseminate the translated PEM across different countries and cultures. Patients living with a rare disease face health disparities [2] and patient engagement has been identified as potential means to empower this patient population [46]. Interestingly, patient engagement has recently been gaining attention in the context of orphan drug development [23]. However, the extent of patient engagement varies widely. A 2014 systematic review of patient engagement for research on rare diseases found engagement is typically unidirectional - involving patients in consultative roles and rarely in creative aspects or in terms of dissemination [24]. The present study is unique in that we used a participatory process to co-create PEM with patients; we then evaluated the PEM produced by this collaboration, and worked with patient groups to facilitate dissemination to the largest possible audience.

We previously partnered with online patient community leaders to identify the unmet health and informational needs of patients with congenital hypogonadotropic hypogonadism (CHH) and Kallmann syndrome [10]. In the present study, the partnership was more clearly bi-directional as patients were not simply providers of opinions; rather they contributed directly in co-creating the PEM in an iterative process. Notably, patient knowledge and expertise emerges from the day-to-day experiences of living and coping with a rare condition and therefore is inherently different from the expertise of healthcare professionals [25]. Recently, a study examining online exchanges among patients with rare adrenal disorders found that information and support were central elements in peer-to-peer exchanges [26]. Moreover, the authors noted that patient-centered care could be enhanced by better integrating patient knowledge with the care provided by professionals. In the present study, developing the PEM was a true partnership that recognized patient expertise as unique and complementary to expert clinician knowledge. We believe that this co-creation contributed to the high acceptability ratings by patients.

This evaluation process of the co-created PEM has limitations. The evaluation was only conducted on the English version. As such, the findings are not completely transferable to the other translated versions despite the inclusion of patients in developing some of the translations. Moreover, the additional validation step of back translating the other versions was not conducted and this could be viewed as a limitation. We only assessed readability once the materials had been finalized, not during the development process. In future studies, this testing could be incorporated earlier in the development process to improve the reading level of developed PEM. While the evaluation was overwhelmingly positive and a fairly sizeable sample was reached (for a rare disease population), the patients completing the evaluation were quite well-educated and exhibited high levels of health literacy. Accordingly, our ability to draw inferences to a broader population of lower literacy patients is limited. This may reflect a bias of using a web-based survey - as perhaps those using the web may have higher literacy levels. However, recruiting sufficient numbers of patients for rare disease studies has been a long-standing challenge [27, 28]. Therefore, we used a web-based approach to overcome this barrier but note that such an approach entails a potential risk of bias.

The Pew Foundation’s published report on health and the internet indicates that patients living with a rare disease are internet power users who are most likely to seek information about their condition online and find support from other patients using social media [29]. Based on our previous success combining patient partnerships and social media for the online needs assessment [10], we employed a similar approach in the present study to reach a relatively large sample (n = 63) over 8-weeks. These experiences suggest that web-based platforms are an effective means to reach and connect rare disease patients. Thus, the opportunities afforded by the internet and social media may provide novel avenues for crowdsourcing solutions as well as offering a shared venue for either clinician- or patient-led collaborations to improve quality and add value to the healthcare system [5, 30]. The European Union Committee of Experts on Rare Diseases (EUCERD) recommendations for Centers of Expertise underscore the importance of collaboration with patient organizations to provide information that is at once accessible and adapted to patient needs [31]. For many rare diseases, such as CHH/Kallmann syndrome, formal organized patient support organizations do not exist. As such, web-based approaches and social media provide a critical means to broadly reach patients, identify priorities and incorporate their perspectives and knowledge into care. This may be particularly advantageous in light of the movement to form European Reference Networks for rare diseases [32, 33].

The final step in this co-creation process was to engage in bi-directional dissemination. This has been identified as a shortcoming in much of the patient engagement research conducted in the context of rare diseases [24]. Through the work of members of the Network and patients alike, materials were adapted and translated into 20 languages by native speakers. This collaborative process is essential for ensuring that information provided to patients is culturally adapted and sensitive – a key element for Centers of Expertise [31]. In parallel to traditional healthcare professional outlets (e.g. scientific meetings, peer-review publication) patient participants are distributing materials directly to other patients via social media and postings on centralized patients sites [15]. The co-created PEM (in multiple languages) is a critical component of the list of patient resources available on the website of the European network comprising a virtual empowerment toolkit for patients and families [9]. Available information includes listings of international specialized referral centers, genetic testing labs, clinical trials, and peer-to-peer support as well as a portal for a patient registry. We are utilizing both professional-oriented avenues and more patient-oriented social media outlets to hopefully reach unprecedented numbers of patients and clinicians and overcome traditional roadblocks of implementation into practice [3436].


Partnering with patients enabled co-creation of high-quality PEM while social media and web-based data collection facilitated timely evaluation by a dispersed patient population. We believe that partnering with expert patients was an empowering experience and provides valuable contributions for developing patient-centered approaches to care. We envision this work will serve as a roadmap for those wishing to engage in a co-creation process and will help inform projects aimed at improving care for patients living with a rare disease.



Congenital hypogonadotropic hypogonadism


European Union Committee of Experts on Rare Diseases


European Organization for Rare Diseases


European Cooperation in Science and Technology


Patient education materials


Patient education materials assessment tool


Simple measure of Gobbledygook


  1. EURORDIS. The voice of 12,000 patients: Experiences and expectations of rare disease patients on diagnosis and care in Europe. France: Boulogne-Billancourt; 2009.

  2. Holtzclaw Williams P. Policy framework for rare disease health disparities. Policy Polit Nurs Pract. 2011;12:114–8.

    Article  PubMed  Google Scholar 

  3. Cohen JS, Biesecker BB. Quality of life in rare genetic conditions: a systematic review of the literature. Am J Med Genet A. 2010;152A:1136–56.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Ayme S, Kole A, Groft S. Empowerment of patients: lessons from the rare diseases community. Lancet. 2008;371:2048–51.

    Article  PubMed  Google Scholar 

  5. Vicari S, Cappai F. Health activism and the logic of connective action. A case study of rare disease patient organisations. Inf Commun Soc. 2016;19:1653–71.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Sofolahan-Oladeinde Y, Mullins CD, Baquet CR. Using community-based participatory research in patient-centered outcomes research to address health disparities in under-represented communities. J Comp Effectiveness Res. 2015;4:515–23.

    Article  Google Scholar 

  7. Fromantin M, Gineste J, Didier A, Rouvier J. Impuberism and hypogonadism at induction into military service. Statistical study. Problemes Actuels d’Endocrinologie et de Nutr. 1973;16:179–99.

    CAS  Google Scholar 

  8. Boehm U, Bouloux PM, Dattani MT, de Roux N, Dode C, Dunkel L, Dwyer AA, Giacobini P, Hardelin JP, Juul A, et al. Expert consensus document: European Consensus Statement on congenital hypogonadotropic hypogonadism--pathogenesis, diagnosis and treatment. Nat Rev Endocrinol. 2015;11:547–64.

    PubMed  Google Scholar 

  9. COST Action BM1105 []. Accessed 14 Mar 2017.

  10. Dwyer AA, Quinton R, Morin D, Pitteloud N. Identifying the unmet health needs of patients with congenital hypogonadotropic hypogonadism using a web-based needs assessment: implications for online interventions and peer-to-peer support. Orphanet J Rare Dis. 2014;9:83.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Crawford MJ, Rutter D, Manley C, Weaver T, Bhui K, Fulop N, Tyrer P. Systematic review of involving patients in the planning and development of health care. BMJ. 2002;325:1263.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Greenhalgh T, Jackson C, Shaw S, Janamian T. Achieving research impact through co-creation in community-based health services: literature review and case study. Milbank Q. 2016;94:392–429.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Wallerstein N, Duran B. Community-based participatory research contributions to intervention research: the intersection of science and practice to improve health equity. Am J Public Health. 2010;100 Suppl 1:S40–46.

    Article  PubMed  PubMed Central  Google Scholar 

  14. CheckText []. Accessed 14 Mar 2017.

  15. RareConnect []. Accessed 14 Mar 2017.

  16. Chew LD, Bradley KA, Boyko EJ. Brief questions to identify patients with inadequate health literacy. Fam Med. 2004;36:588–94.

    PubMed  Google Scholar 

  17. Chew LD, Griffin JM, Partin MR, Noorbaloochi S, Grill JP, Snyder A, Bradley KA, Nugent SM, Baines AD, Vanryn M. Validation of screening questions for limited health literacy in a large VA outpatient population. J Gen Intern Med. 2008;23:561–6.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Shoemaker SJ, Wolf MS, Brach C, (Prepared by Abt Associates IuCNHI: The Patient Education Materials Assessment Tool (PEMAT) and User’s Guide. In Book The Patient Education Materials Assessment Tool (PEMAT) and User’s Guide (Editor ed.^eds.). Maryland: Agency for Healthcare Research and Quality; 2013.

  19. Shoemaker SJ, Wolf MS, Brach C. Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns. 2014;96:395–403.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Saldana J. Coding manual for qualitative researchers. Thousand Oaks, CA: Sage; 2009.

    Google Scholar 

  21. Safeer RS, Keenan J. Health literacy: the gap between physicians and patients. Am Fam Physician. 2005;72:463–8.

    PubMed  Google Scholar 

  22. Doak CC, Doak LG, Root JH. Teaching Patients with Low Literacy Skills. 2nd ed. Philadelphia, PA: J.B. Lippincott Company; 1996.

    Google Scholar 

  23. Bronstein MG, Kakkis ED. Patients as key partners in rare disease drug development. Nat Rev Drug Discov. 2016;15:731–2.

    Article  CAS  PubMed  Google Scholar 

  24. Forsythe LP, Szydlowski V, Murad MH, Ip S, Wang Z, Elraiyah TA, Fleurence R, Hickam DH. A systematic review of approaches for engaging patients for research on rare diseases. J Gen Intern Med. 2014;29 Suppl 3:S788–800.

    Article  PubMed  Google Scholar 

  25. Hartzler A, Pratt W. Managing the personal side of health: how patient expertise differs from the expertise of clinicians. J Med Internet Res. 2011;13:e62.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Kauw D, Repping-Wuts H, Noordzij A, Stikkelbroeck N, Hermus A, Faber M. The contribution of online peer-to-peer communication among patients with adrenal disease to patient-centered care. J Med Internet Res. 2015;17:e54.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Kruer MC, Steiner RD. The role of evidence-based medicine and clinical trials in rare genetic disorders. Clin Genet. 2008;74:197–207.

    Article  CAS  PubMed  Google Scholar 

  28. Watson MS, Epstein C, Howell RR, Jones MC, Korf BR, McCabe ER, Simpson JL. Developing a national collaborative study system for rare genetic diseases. Genet Med. 2008;10:325–9.

    Article  PubMed  Google Scholar 

  29. Fox S. Peer-to-peer healthcare: many people - especially those living with chronic or rare diseases - use online connections to supplement professional medical advice. Washington, D.C.: Pew Internet; 2011.

    Google Scholar 

  30. Nambisan P, Nambisan S. Models of consumer value cocreation in health care. Health Care Manag Rev. 2009;34:344–54.

    Article  Google Scholar 

  31. (EUCERD) EUCoEoRD: Recommendations on quality criteria for centres of expertise for rare diseases in member states. In: Book Recommendations on Quality Criteria for Centres of Expertise for Rare Diseases in Member States (Editor ed.^eds.). City; 2011.

  32. Diseases EUCoEoR. Recommendations to the European Commission and the Member States on European Reference Networks for Rare Diseases. 2013.

    Google Scholar 

  33. European Reference Network []. Accessed 14 Mar 2017.

  34. Sung NS, Crowley Jr WF, Genel M, Salber P, Sandy L, Sherwood LM, Johnson SB, Catanese V, Tilson H, Getz K, et al. Central challenges facing the national clinical research enterprise. JAMA. 2003;289:1278–87.

    Article  PubMed  Google Scholar 

  35. Thornicroft G, Lempp H, Tansella M. The place of implementation science in the translational medicine continuum. Psychol Med. 2011;41:2015–21.

    Article  CAS  PubMed  Google Scholar 

  36. Bellgarda MI, Sleemanc MW, Guerrerod FD, Fletcher S, Baynam G, Goldblatt J, Rubinstein Y, Bell C, Groft S, Barreroa R, et al. Rare disease research roadmap: navigating the bioinformatics and translational challenges for improved patient health outcomes. Health Policy Technol. 2014;3:325–35.

    Article  Google Scholar 

Download references


The authors wish to thank the patients who generously gave their time in helping to develop the patient education materials and we acknowledge the Working Group members of COST Action BM1105 for their contributions. Special thanks to Ms. Virginia Hughes for her critical input during the development process, Prof. Linan Cheng of the Shanghai Institute of Planned Parenthood Research for her validation of the Chinese translations and Ms. Özlem Firat for her help in coordinating the translated materials. This work was supported by the Rare Disease Foundation and the BC Children’s Hospital Foundation (# 17-26) and COST Action BM1105.


This work was supported by the COST Action BM1105 and received funding from the Rare Disease Foundation and the BC Children’s Hospital Foundation (# 17-26).

Availability of data and materials

Data sharing not applicable to this article as no datasets were generated or analysed during the current study.

Authors’ contributions

AD conceived the study and participated in study design, conduct, analyses and drafted the manuscript. RQ, NP and NS helped draft the educational materials with patients. All authors helped revise the materials and contributed to developing culturally relevant translations. All authors read and approved the final manuscript.

Competing interests

The authors have no financial or non-financial competing interests to declare.

Consent for publication

Not applicable.

Ethics approval and consent to participate

This study (protocol #233/13) was reviewed and approved by the Commission Cantonale d’ethique de la recherche sur l’être humain which is the institutional review board (Ethics Committee) associated with the University of Lausanne. All survey participants provided opt-in electronic consent prior to completing the online evaluation.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Author information

Authors and Affiliations



Corresponding author

Correspondence to Andrew A. Dwyer.

Additional files

Additional file 1:

Questions frequently asked by patients on the social media patient support site. (PDF 51 kb)

Additional file 2:

Patient Education Materials. (PDF 3856 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

COST Action BM1105., Badiu, C., Bonomi, M. et al. Developing and evaluating rare disease educational materials co-created by expert clinicians and patients: the paradigm of congenital hypogonadotropic hypogonadism. Orphanet J Rare Dis 12, 57 (2017).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: