Typically, in NISS-Merck meetups you find yourself immersed in detailed discussions about the statistical issues involved in drug/vaccine development and/or clinical trials. This time around the focus seemed to shift from applications to implications. If you use increasingly popular open-source software, what challenges will you face when you try to bring pharmaceutical developments and/or medical devices up for review by regulatory agencies? (see Event page)
Moderator Dan Holder (Merck) brought together four experts to discuss the reasons why open software approaches make sense, but on the other hand raise issues and challenges that result. These speakers included Andy Nicholls (R Validation Hub, GSK), Paul Schuette (FDA-CDER), Tarek Haddad and Ted Lystig (Medtronic).
The first speaker was Andy Nicholls who is currently the chair of the executive committee of the R Validation Hub, “a collaboration to support the adoption of R within a biopharmaceutical regulatory setting.” He first steered the discussion away from the validation of any particular software, instead taking great pains to describe the reliability of software using the FDA’s clarification document and a software example to drive home this point. From here he talked about the issues related to the difficulty in using open-source languages for GxP analyses. Andy pointed to the R ecosystem of collaborative development and the associated risks this brings but then also discussed how to manage these risks.
“The computer software used for data management and statistical analysis should be reliable, and documentation of appropriate software testing procedures should be available.”
Andy Nicholls (R Validation Hub quoting from FDA’s Clarifying Documents)
The second speaker was Paul Schuette, Scientific Computing Coordinator at FDA-CDER. He first reviewed the FDA’s submission procedures, especially as related to standards and guides to computing environments that are typically included with submissions. He then detailed the reasons why some researchers have chosen to use open-source software including cost, innovation, shareability and performance. Paul then reviewed a number of perceived challenges to using open-source approaches including issues that may surround support, the amount or type of legacy code used within an institution, as well as the skill sets required by IT departments. Paul then reviewed a number of strategies for mitigating some of these challenges as well as some interesting emerging trends that he sees with regards to the collection and analysis of data.
“FDA does not require use of any specific software for statistical analyses, and statistical software is not explicitly discussed in Title 21 of the Code of Federal Regulations [e.g., in 21CFR part 11]. However, the software package(s) used for statistical analyses should be fully documented in the submission, including version and build identification.”
Paul Schuette (FDA-CDER)
Tarek Haddad led off the team from Medtronic that also included Ted Lystig. Tarek focused his remarks on the development of two statistical packages in R for regulatory purposes, why Medtronic built these packages and the lessons learned in the process. First of all, Medtronic’s work targeted statistical approaches that currently were not available. This work involved the use of models in the clinical trials part of product development. By using virtual patients in adaptive Bayesian clinical trials, they reduce cost and add to the evidence used in modeling. Tarek discussed how they went about this development, the documentation involved, steps that were taken to improve credibility as well as a number of considerations others will want to review.
Ted Lystig followed Tarek with more general thoughts about moving to using more open-source software including why this move would be made, and how this change can be made. These issues related to changes that may involve significant cost, i.e., the inefficiencies related to training of employees, cost considerations for software and/or the flexibility companies may require to remain competitive. Ted noted that compliance is the main issue related to making this change work and provided some advice for how to approach this. He then discussed some of the strategies incorporated into the plan that his company used to make considered but reasonable steps towards the inclusion in open-source software as part of their product development process.
“The uninitiated may feel that only commercial statistical software systems could be accepted by FDA. The FDA has clearly stated that this is not the case.”
Ted Lystig (Medtronic)
Once again, the audience at the NISS-Merck meet-ups were certainly forthcoming with their questions and comments for the speakers to discuss! But first moderator Dan Holder (Merck) asked the panelists if they have any questions for each other. Paul responded to a question about whether any submissions to the FDA have included R by stating that he recalled a first one that included one part done in R ten years ago! And now he is seeing quite a few hybrid submissions. Further questions were followed by energetic discussion including: “What is the best way to bring the community together so that everyone is not doing their own risk assessments?”, “Isn't the only way to verify/validate estimates from complex statistical models to compare results with what other statistical packages produce?”, “I sense at my company that we are deathly afraid of the FDA inspector shutting down a project because our R results aren't validated.”, “How to optimally address that with IT and QA?” and “What is the biggest pain point for the FDA when reviewing submissions that include open-source software approaches?”
There are so many questions and yes, as this meet-up has established, there are a host of others in the pharmaceutical industry pondering how to tackle these same challenges. Clearly there are growing support structures and communities in which interested individuals and companies can participate in.
Recording of the Session
Slides Used by the Speakers
Andy Nicholls, (R Validation Hub, GSK)
"Why is it so difficult to use Open Source languages for GxP analyses?!"
Paul Schuette, (FDA-CDER)
"Open Source software for regulatory submissions: Challenges and paths forward"
Paul would also like to share the following link that became available after this session: Modernization of Statistical Analytics (MSA) Framework (TransCelerate BioPharma Inc.)
Tarek Haddad, (Medtronic)
“Industrial considerations for Statistical Tool Used to Support Submissions”
Ted Lystig, (Medtronic)
"Industry considerations in diversifying the statistical tool sets used to support FDA submissions"