SATURN: assessing the feasibility of utilising existing registries for real-world evidence data collection to meet patients, regulatory, health technology assessment and payer requirements

Abstract

Background

SATURN (Systematic Accumulation of Treatment practices and Utilisation, Real world evidence, and Natural history data) for the rare condition osteogenesis imperfecta (OI) has the objective to create a common core dataset by utilising existing, well-established data sources to meet the needs of the various stakeholders (physicians, registry/dataset owners, patients and patient associations, OI community leaders, European [EU] policymakers, regulators, health technology assessments [HTA]s, and healthcare systems including payers). This paper describes the steps taken to assess the feasibility of one existing OI registry (i.e., the Registry of OI [ROI]) as a candidate for SATURN. The same methodology will be applied to other existing OI registries in the future and this same concept could be utilised for other rare disease registries.

Methods

The approach to assessing the feasibility of the ROI registry consisted of three steps: (1) an assessment of the registry characteristics using the Registry Evaluation and Quality Standards Tool (REQueST); (2) a gap analysis comparing SATURN required Core Variables to those being captured in the registry’s Case Report Form (CRF); and (3) a compliance check on the data exchange process following the Title 21 of Code of Federal Regulations (CFR) Part 11/EudraLex Annex 11 Compliance Checklist. The first registry that SATURN has assessed is the ROI database at the Istituto Ortopedico Rizzoli (IOR) in Italy.

Results

The results from the ROI REQueST have demonstrated satisfactory complete responses in terms of methodology, essential standards, interpretability, and interoperability—readiness for data linkage, data sources, and ethics to meet the needs of data customers. However, the ROI data is from a tertiary referral centre which may limit the ability to understand the full patient journey. The gap analysis has revealed that an exact or logical match between SATURN requested variables and the ROI current variables exists for the following items: patient characteristics, treatment of OI (medical and surgical) and treatment of pain (with the exception of frequency of treatment and reasons for discontinuation), fracture history and bone density. However, data on safety was missing. The compliance check has implied that the ROI implemented appropriate controls for the web-based platform (i.e., Genotype–phenotype Data Integration Platform [GeDI]) that is involved in processing the electronic patient data, and GeDI is a validated/compliant application that follows relevant 21 CFR Part 11/EudraLex Annex 11 regulations.

Conclusions

This robust feasibility process highlights potential limitations and opportunities to develop and to refine the collaboration with the ROI as the SATURN programme moves forward. It also ensures that the existing datasets in the rare condition OI are being maximised to respond to the needs of patients, data customers and decision-makers. This feasibility process has allowed SATURN to build a compliant methodology that aligns with the requirements from the European Medicines Agency (EMA) and HTAs. More data variables will continue to be developed and refined along the way with more registries participating in SATURN. As a result, SATURN will become a meaningful and truly collaborative core dataset, which will also contribute to advancing understanding of OI diagnosis, treatment, and care.

Background

Treatments for rare diseases (RD)s face challenges in obtaining both regulatory approval and positive pricing and reimbursement decisions due to lack of documentation, smaller patient populations, the requirement of specific patient characteristics, uncertain or varying treatment guidelines (when available) and smaller markets for medicinal products.

Real-world data (RWD) is defined by the U.S. Food and Drug Administration (FDA) as “data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources including data derived from electronic health records, medical claims data, data from product or disease registries, and data gathered from other sources (such as digital health technologies) that can inform on health status”; and real-world evidence (RWE) as “clinical evidence about the usage and potential benefits or risks of a medical product derived from analysis of RWD”  [7]. Whilst RWD should be a potential source of data to help address the challenges facing treatments for RDs, there is a limited amount of RWD available for RDs due to a long and difficult journey until an accurate diagnosis, lack of relevant diagnostic standards, limited knowledge by primary care and some specialists about the diseases, variance in treatment pathways, and the complicated patient journey.

The EMA has set out its vision that by 2025 the use of RWE will be a key and established contributor alongside data from other sources (including clinical trials) for a wide range of regulatory decision-making. This vision goes beyond the well-established uses in safety monitoring and epidemiology studies. It reflects a wider move by the EMA to leverage the value of RWE through its Big Data Steering Group initiative and workstreams – including ensuring data quality, discoverability, and governance, which are key elements of the existing EMA registry-based studies guidance [1]. The EMA has also developed the Data Analysis and Real World Interrogation Network (DARWIN) Project for hosting a catalogue of observational data sources [2].

If RWD is to be utilised to this magnitude, data must be of a sufficiently robust standard to be meaningful. The EUnetHTA has developed a tool, REQueST, which sets out standards that are universal and essential for good practice and data quality and which are relevant for different types of registries [3]. REQueST is designed to be broken into three distinct stages [4]:

• Methodology: type of registry, previous-use and publications, geography and organisational setting, duration, size, inclusion/exclusion criteria, follow-up, confounders;

• Essential Standards: aims and methodology, governance, informed consent, data dictionary, minimum dataset, standard definitions, terminology and specification, data collection, quality assurance, data cleaning, missing data, financing, protect, security and safeguards;

• Additional Requirements: interoperability and readiness for data linkage, data sources and ethics.

It is clear from these regulatory initiatives that there will be increasing value placed upon RWD in the future. Such data, when robust and compliant with required standards, will have a valuable role in the RD arena in providing much needed information to support treatments for patients with RD.

SATURN has the objective to create a common core dataset by utilising existing, well-established real world data sources, to meet the needs of the various stakeholders (physicians, registry/dataset owners, patients and patient associations, OI community leaders, EU policymakers, regulators, HTAs, and healthcare systems including payers).

SATURN will align with the EMA guidance that ensures the availability of the data, that feasibility assessments are undertaken, and that quality control measures and procedures are executed to support the quality of data outputs. A framework will be established to oversee and manage SATURN – including a Governance Plan, a Data Management Plan (DMP) that is in alignment with findable, accessible, interoperable, and reusable (FAIR) principles [8, 11], and a Quality Management Plan and Research Protocol.

The concept of SATURN has been described previously [10]. Here we describe the methodology for its development in terms of governance, data management, quality, and compliance to ensure that the outputs from SATURN will be relevant and reliable for decision-makers at different stages of a medicines’ evaluation and availability pathways. We specifically describe the methodology and results of a feasibility assessment undertaken of ROI as part of SATURN. The ROI, established at the IOR, one of the leading institutions for OI in Italy, is the first registry being assessed under SATURN umbrella. It consists of a well-established dataset [9] gathering records from approximately 1300 patients with OI, using a robust and validated cloud-based electronic platform. All patients treated at Department of Rare Skeletal Disorders of IOR, who match the inclusion criteria of the ROI protocol, have been enrolled in this registry.

The outcome of this assessment will allow better understanding of patient characteristics, variables captured, population size, data quality, data governance, and data sharing policies. This exercise will then be repeated with other datasets/registries for OI. A full feasibility report will be produced for each registry. This approach will ensure a consistent and complete approach to each registry and secure that the datasets being used in SATURN to generate aggregated data outputs are of the required quality and consistency.

Methods

The approach to assessing the feasibility of the ROI registry consisted of three steps: 1) an assessment of the registry characteristics using REQueST; 2) a gap analysis comparing SATURN required Core Variables to those being captured in the registry’s CRF; and 3) a compliance check on the data exchange process following the Title 21 of CFR Part 11 [6]/EudraLex Annex 11 [5] (Fig. 1).

Fig. 1

SATURN—a Stepwise Approach to Assessing the Feasibility of the ROI. CFR, Code of Federal Regulations; CRF, case report form; REQueST, Registry Evaluation of Quality Standards Tool; ROI, Registry of Osteogenesis Imperfecta; SATURN, Systematic Accumulation of Treatment practices and Utilisation, Real world evidence, and Natural history data

Step 1 The assessment using REQueST consists of three stages: Methodological Information, Essential Standards and Additional Requirements. The tool was sent by the SATURN Research Team (SRT)¹ to the IOR for their completion. Once completed, it was returned to the SRT for review. A dialogue subsequently took place between the two parties to resolve any outstanding issues. The outcome was documented in REQueST. For the Methodological Information and Additional Requirements stages, the SRT made a final assessment as to whether the responses would meet HTA’s/regulators requirements (Table 1) and for the Essential Standards section, the SRT made a final assessment as to whether the minimum standard was met. In all stages, the SRT added additional comments in relation to SATURN as applicable (Table 2).

Table 1 Completion of Methodological Information and Additional Requirements of REQueST

Table 2 Completion of Essential Standards of REQueST

Step 2 The draft Core Variables List was initially based on clinical trial endpoints for a product in development for OI. In an RWE setting, it was assessed by the ROI team and then adjusted following their feedback.

In this feasibility assessment, the draft Core Variables List was compared to the registry variables contained in the data dictionary/CRF utilised by the ROI. The gap analysis, undertaken by a data management specialist of the SRT, involved a manual comparison to identify the similarities and differences in registry variables contained in the CRF vs. the draft Core Variable List. In this analysis, each variable in the draft Core Variables List was categorised as either “Exact Match”, “Logical Match”, or “Omitted” to reflect the status in the ROI.

• Exact Match: registry contains the same variable and variable responses as the draft Core Variables List.

• Logical Match: registry contains a similar variable and variable responses as the draft Core Variables List for analysis. Special scenarios include:

  • Exact variable/variable response not collected in the registry; however, required variables can be derived using other information collected. For example, “patient age” not collected in registry; however, as “date of birth” and “date of consent” were captured, these two variables can be used to derive patients’ age.

  • Minor differences in variable format were allowed. For example, “surgery type” was not captured in registry as free text, however, registry captures “surgery type” using a pre-defined updatable list of surgeries in a drop-down menu.

• Omitted: registry does not contain the variable within the draft Core Variables List.

Once completed, the results of the gap analysis were reviewed by the SRT and then by the ROI. In addition, the ROI was required to confirm the list of omitted variables. The final decision on the degree of variable matching was made by the full team and alignment was reached after consolidating all feedback from the SRT and the ROI.

Step 3 A compliance check on the data exchange process following the Title 21 of CFR Part 11/EudraLex Annex 11 Compliance Checklist was performed under the collaboration between both the ROI staff and SRT.

The checklist was used to determine if the ROI implemented controls, including audits, system validation, audit trials, electronic signatures, and documentation for the web-based platform GeDI that was involved in processing the electronic patient data.

More specifically, the project manager of the SRT assessed the ROI data source characteristics and answered the following questions:

• Does the project generate information and/or data that is used in any decision-making process during the Research Development, or Clinical process?

• Will the project be used to support regulatory/HTAs/payer submissions?

• Does the project process, transfer or store Good Clinical Practice (GCP)-related information in electronic format that is required by regulatory agencies?

• Could the project have an impact on an already qualified/validated environment that supports regulatory data?

• Does GCP apply to the system?

If any were answered “Yes”, a compliance check on electronic data exchange process was required. The ROI research staff then received the checklist from SRT and conducted the compliance check by assessing all items on the 21 CFR Part 11/EudraLex Annex 11 Compliance Checklist.

Results

Step 1: REQueST

Responses to REQueST are provided in Table 3. All responses in the Methodological Information and Additional Requirements were considered by the SRT to meet HTA agency/payers/regulator's needs and all responses in the Essential Standards were considered to have met the Minimum Standards.

Table 3 The ROI Characteristics using REQueST

Step 2: Gap analysis

Table 4 sets out the high-level results of the gap analysis and shows for which of the draft SATURN Variables there was either an Exact Match, Logical Match in the ROI CRF or whether the variable was omitted.

Table 4 Gap analysis of the ROI dataset compared to required SATURN variables

Whilst an Exact Match and Omitted are clearly defined measures, logical matches are subject to individual opinion. The rationale for categorizing a variable as a logical match is set out in in Table 5.

Table 5 Logical Matched Variables

Step 3: A compliance check on the data exchange process

Per ROI’s statements, the ROI confirmed that that their system (i.e., GeDI) that captures the patient data complies with relevant 21 CFR Part 11/EudraLex Annex 11 Compliance Checklist. Electronic signatures are out of scope as it is not a part of the GeDI platform functionality. Detailed results of the compliance check on the ROI data exchange process are provided in Appendix A. Compliance Check on the ROI Electronic Data Exchange.

Discussions

The feasibility assessment has been based upon a review of REQueST, the gap analysis comparing the draft data variables identified by SATURN and those collected in the CRF of the ROI, and a compliance check. The results from the ROI REQueST have demonstrated satisfactory complete responses in terms of methodology, essential standards, interpretability, and interoperability—readiness for data linkage, data sources, and ethics to meet the needs of data customers.

However, a few points have been highlighted during the feasibility assessment which may result in some limitations. Firstly, it is stated in the responses that the registry has national coverage, but it is later noted that the majority of OI patients are from one institution. This is because the ROI is a referral centre for OI management. Thus, although patients may reside throughout Italy, they are seen when necessary at this tertiary centre. The frequency of visits can vary depending on the need to be seen at this tertiary centre. This can vary from several times a year to once every 10 years and differs from the primary or secondary care setting where both children and adults would have regular follow up and regular data collection.

The implications of this could be that (1) the data will only reflect the visits to the tertiary centre and not the full patient treatment journey; (2) visits may only occur as needed and not on a regular basis; and (3) regional variations in treatment may not be captured.

Secondly, in REQueST it is stated that comorbidities and concomitant therapies are collected but not disclosed to the appropriate Marketing Authorisation Holder. This comment needs further clarification as this could limit the objectives of a specific study in the future.

A draft Core Variables List was developed initially from OI clinical trials’ endpoints. In this feasibility study, it was assessed by the ROI team and then revised following feedback based on their real-world clinical practice experiences. As the SATURN programme moves forward, ongoing validations of this draft Core Variables List will be undertaken with future collaborations and assessment by Key Opinion Leaders (KOLs). The final Core Variables List aims to include all data-points which would be required to meet the needs of physicians, registry owners, patients, OI community leaders, EU policymakers, regulators, HTAs and healthcare systems including payers.

In this feasibility assessment, the gap analysis has revealed that an exact or logical match between SATURN requested variables and the ROI current variables exists for the following items: patient characteristics, treatment of OI (medical and surgical) and treatment of pain (with the exception of frequency of treatment and reasons for discontinuation), fracture history and bone density. However, there is a significant gap in relation to the collection of safety data, as no safety data is captured in the ROI at present. This is understandable as the ROI has the objective of observing the natural history of OI rather than assessing safety of specific treatments. Potential inclusion of additional fields in the future, such as safety data, would need to be agreed with ROI.

Whether or not the ROI can be used as a complete data source for SATURN will depend upon the final objectives of any study protocol and whether the ROI is amenable to integrating additional data variables into their established data collection infrastructure to achieve customised data collection.

Based upon the feasibility results, a study to understand the natural history and treatment pathway of patients with OI appears to be feasible because variables of OI type, genotype, symptoms, and treatment are included. A limitation is that frequency of pharmaceutical treatments and reasons for discontinuation are currently not included. However, it is anticipated that these variables would not be critical for such a study.

A study to assess health-related quality of life (HRQoL) may also be feasible, but this would be limited to EQ-5D, and pain score measured by age and site-specific visual analogue scale (VAS) which are currently already collected within the ROI.

The ROI does not collect healthcare resource utilisation (HRU) data specifically although has stated that some information can be obtained from the signs & symptoms/ treatment fields. Further investigation would be needed to determine if this information would be sufficient to address any research questions concerning HRU.

Should there be a regulatory request for the sponsor of SATURN to undertake a post-authorisation safety study (PASS), the ROI, with its current data collection parameters, would need to add adverse event (AE) variables to the data collected and this would require the agreement of the data owner. However, it may be possible to undertake a post authorisation effectiveness study (PAES) because endpoints such as fracture history and bone density are included. If the ROI data collection is enhanced in the future with additional safety information, then this data source could be valuable for PASS studies.

Finally, it should be noted that the feasibility assessment does not give any indication of the completeness of the data collected. Data will be entered into the ROI registry as a reflection of the normal clinical practice of the treating physician and hence there is always the possibility of missing data if a particular parameter was not clinically assessed or recorded. The extent of missing data cannot be predicted from the feasibility assessment. SATURN plans to obtain data from the ROI registry in order to assess the completeness of the data to answer future research questions.

In parallel with ongoing dialogue with the “data customers”, i.e. regulators, HTA bodies or national payers/budget-holders, SATURN aims to be able to provide aggregated data outputs to answer decision-makers’ questions in a robust and high-quality manner.

At this moment, these results can only reflect the ROI analysis. However, further databases will be added in the future, which will make SATURN more representative across Europe.

Conclusions

This robust feasibility process highlights potential limitations and opportunities to develop and to refine the collaboration with the ROI as the SATURN programme moves forward. It also ensures that the existing datasets in the rare condition OI are being maximised to respond to the needs of patients, data customers and decision-makers.

REQueST standardises the approach to assess each registry. It allows for a clear understanding of the registry and data collection process, and an assessment of each essential standard to ensure that the quality and breadth of data will meet data customers’ requirements.

The variable gap analysis between the ROI variables and SATURN dataset produces a clear overview of the availability of data, the data variables collected, and any gaps currently identified which could be addressed as the collaboration continues and as SATURN develops to meet the broader needs of the data customers.

The compliance check ensures that the ROI implements appropriate controls for the web-based platform that is involved in processing the electronic patient data and follows relevant 21 CFR Part 11/EudraLex Annex 11 regulations.

Completeness of data was not assessed in this feasibility assessment, but SATURN plans to analyse an output of the data in the future.

This feasibility assessment has allowed SATURN to begin to build a compliant methodology that aligns with the requirements from the EMA and HTAs. More data variables will continue to be developed and refined along the way with more registries participating in SATURN and with the data customers (e.g., patients, regulators, HTAs, healthcare systems including payers). As a result, SATURN will become a meaningful and truly collaborative core dataset, which will also contribute to advancing understanding of OI diagnosis, treatment and care.

Appendix A: Compliance Check on the ROI Electronic Data Exchange Process

System Name & Version	Genotype–phenotype Data Integration Platform (GeDI) v.3.4.2
Date of Assessment Initiation	24 October 2023
Date of Assessment Completion	24 January 2024
Brief Description of System Functionality	GeDI is a standardised and web-accessible platform that stores clinical, genetic, family history as well as quality of life data from patients. It is built in compliance with national and European privacy regulation and medical informatic standards
System Development	☒ Internally developed ☒ Vendor Purchased ☐ Off-the-shelf
System Classification	☐ Open ☒ Closed
Are electronic signatures used?	☐ Yes ☒ No, (Section II not required)

21CFR11/ EudraLex Annex 11 Ref	Item	Yes/No	Comment: Please list SOPs where appropriate
11.10 (a)	Has the system been validated to ensure accuracy and reliability is consistent with the intended performance?	☒ Yes ☐ No
	(a) Are documented user/functional requirements available to support the validation testing?	☒ Yes ☐ No
	(b) Is there documented evidence to support the validation effort?	☒ Yes ☐ No
	Has the system been validated to discern altered or invalid records?	☒ Yes ☐ No
	(a) Was a vendor audit performed?	☒ Yes ☐ No
	(b) Was the system developed and tested according to a documented SDLC?	☒ Yes ☐ No
	(c) Was User Acceptance testing performed in accordance with UBC procedures?	☐ Yes ☐ No	N/A
11.10 (b)	Can accurate and complete copies of records be generated in both electronic and human readable hard copy format?	☒ Yes ☐ No
11.10 (c)	Are all records protected to ensure accurate and ready retrieval throughout the record retention period?	☒ Yes ☐ No
	(a) Are system-specific documented procedures for system back up and restoration available?	☒ Yes ☐ No
	(b) Were the backup/restoration procedures tested as part of the validation process?	☒ Yes ☐ No
	(c) Can the audit trail be printed?	☐ Yes ☒ No
11.10 (d)	Does this system limit access to authorised personnel?	☒ Yes ☐ No
	(a) Define the non-biometric or biometric controls in place for system access	☒ Yes ☐ No	Non-biometric via log in (username&password)
	(b) Can user profiles be defined within the system to limit access to certain functionality?	☒ Yes ☐ No
	(c) Are internal SOPs in place regarding user access administration (e.g. assigning, modifying, and deleting user accounts/profiles)? List applicable SOPs	☒ Yes ☐ No	IO01MR Inserimento dati GeDI rev03
	(d) Are SOPs in place to document user profiles within the system? List applicable SOPs	☒ Yes ☐ No	IO01MR Inserimento dati GeDI rev03
11.10 (e)	Does the system have secure, computer-generated time-stamped audit trails to independently record the date and time of operator entries and actions that create, modify, or delete electronic records?	☒ Yes ☐ No
	(a) Are audit trails created incrementally, in chronological order, and in a manner that does not allow new audit trail information to overwrite existing data?	☒ Yes ☐ No
	(b) Are personnel who create, modify, or delete electronic records able to modify the audit trails?	☐ Yes ☒ No
	(c) Is the audit trail documentation retained for a period at least as long as that required for the subject electronic records?	☒ Yes ☐ No
	(d) Is audit trail documentation available for agency review and copying (e.g. can it be printed in hard copy or copied in electronic format)?	☒ Yes ☐ No	It can be exported in.csv
	(e) Are date and times local to the activity being documented and include year, month day, hour, and minute or maintained in a central time zone which is documented? (Note: Calculation of the local time stamp in certain instances may be derived from a remote server located in a different time zone.)	☒ Yes ☐ No
	(f) Are controls in place to ensure that the system’s date and time are correct?	☒ Yes ☐ No	An NTP server managed and monitored by the Rizzoli System Administrator is configured on all systems/devices connected in the Institutional network
	(g) Is the ability to change the date or time limited to authorised personnel?	☒ Yes ☐ No	The systemic tasks/activities are the sole responsibility of system administration technicians
	(h) Are SOPs in place for the reporting, correction, and documentation of date/time discrepancies noted?	☒ Yes ☐ No	A monitoring platform has been implemented for all systems/devices in the Institutional network, allowing the system technicians to intervene 24/7 for malfunction/failure
	(i) Are all changes to the record maintained incrementally such that the original and current values are retained?	☒ Yes ☐ No
	(j) Is the name of the user, time of change, and reason for change documented in the audit trail?	☒ Yes ☐ No
	(k) Is the name of the user recorded by the system based on the login identification codes?	☒ Yes ☐ No
11.10 (f)	Are there operational system checks to enforce permitted sequencing of steps and events?	☒ Yes ☐ No
	(a) Do the system functionality and workflow support UBC procedures?	☐ Yes ☐ No	N/A
	(b) Are warning messages programmed in the system to notify the user of invalid/skipped processes?	☒ Yes ☐ No
11.10 (g)	Are authority checks used by the system to ensure that only authorised individuals can use the system, electronically sign a record, access the operation or computer system input or output device, alter a record, or perform the operation at hand?	☒ Yes ☐ No
	(a) Are all users authenticated at log on?	☒ Yes ☐ No
	(b) Does this “time-out” if not in use for a specified period of time?	☒ Yes ☐ No	After 60 mins the user is logged off
	(c) Is the name of the user present on-screen during system use?	☒ Yes ☐ No
11.10 (h)	Are device (e.g. terminal) checks used to determine, as appropriate, the validity of the source of data input or operational instruction?	☐ Yes ☐ No	N/A. Data are collected from different sources (paper-based and/or other software). Nevertheless, data are manually input into the GeDI platform
	(a) Are routine checks being performed to determine if the source data is valid?	☐ Yes ☐ No	N/A
11.10 (i)	Is documentation of education, experience, and training (including training on use of the system and security) available for End Users?	☒ Yes ☐ No
	(a) Has system training been documented for all current users prior to processing live data?	☒Yes ☐ No
	(b) Are SOPs in place defining the training requirements for the system?	☐ Yes ☒ No
	(c) Have all users been trained in accordance with the SOPS defined above?	☒ Yes ☐ No	IO01MR Inserimento dati GeDI rev03
	(d) Are CVs on file for all system users?	☒ Yes ☐ No
	(e) Have all IT support staff been trained to adequately maintain the system?	☒ Yes ☐ No
11.10 (j)	Are written policies to hold individuals accountable and responsible for actions initiated under their electronic signatures, in order to deter record and signature falsification?	☐ Yes ☒ No
	(a) Is there evidence of training/agreement of all system users?	☒ Yes ☐ No
	(b) Are periodic audits/reviews performed to monitor compliance with the above policy?	☒ Yes ☐ No
11.10 (k)(1)	Are there adequate controls over documentation for system operation and maintenance with respect to distribution, access, and use?	☒ Yes ☐ No
	(a) Is original documentation maintained in a secured area?	☒ Yes ☐ No	The original documents are both paper-based and electronically based. All those documents are in secured area
	(b) Is documentation provided to all users?	☒ Yes ☐ No
11.10 (k)(2)	Are there adequate controls over system documentation used for revision and change control procedures that maintain an audit trail for documenting time-sequenced development and modification of systems documentation?	☒ Yes ☐ No
	(a) Do formal procedures regarding change control of system documentation exist?	☐Yes ☒ No
	(b) Is each version of software accompanied by new system documentation (e.g. release notes from vendor purchased systems)?	☒ Yes ☐ No	Ticketing system
	(c) Are all users notified of new documentation/ procedures?	☒ Yes ☐ No
	(d) Is documentation for older versions maintained in a secured environment?	☒ Yes ☐ No
11.30	Are document encryption and appropriate digital signature standards used to ensure record authenticity, integrity, and confidentiality?	☒ Yes ☐ No	Just the encryption
11.50 (a)	Do the signed electronic records contain information associated with the signing that clearly indicates:	☐Yes ☐ No	N/A
11.50 (a)(1)	• The printed name of the signer?	☐ Yes ☐ No	N/A
11.50 (a)(2)	• The date and time when the signature was executed?	☐ Yes ☐ No	N/A
11.50 (a)(3)	• The meaning (such as review, approval, responsibility, or authorship) associated with the signature?	☐ Yes ☐ No	N/A
	(a) Are documented procedures available which define how electronic signatures are executed and meaning of each signature? List applicable SOPs	☐ Yes ☐ No	N/A
	(b) Is the electronic signature subject to the same control as electronic records?	☐ Yes ☐ No	N/A
11.50 (b)	Are all components of the electronic signature (name, date/time of execution, and meaning) included in all human readable form of the electronic record (e.g. display or print out)?	☐ Yes ☐ No	N/A
11.70	Are electronic signatures, or handwritten signatures applied to electronic records, linked to their respective electronic records to ensure that they cannot be excised, copied, or otherwise transferred to falsify an electronic record by ordinary means?	☐ Yes ☒ No

CV, curriculum vitae; e.g., for example; GeDI, Genotype–phenotype Data Integration Platform; IT, information technology; NTP, Network Time Protocol; N/A, not applicable; SDLC, Software Development Life Cycle; SOP, Standard Operating Procedure; UBC, United BioSource LLC.