Video: From Assay to Answer: Powering Diagnostics Workflows with Seqera | Duration: 2748s | Summary: From Assay to Answer: Powering Diagnostics Workflows with Seqera | Chapters: Welcome and Introduction (11.12s), Introducing the Webinar (105.75s), Diagnostics Pipeline Challenges (292.935s), Nextflow and Secara (511.32s), Production Bioinformatics Challenges (790.165s), Saqqara Platform Features (1004.485s), Compliance Strategies Explored (1234.335s), Q&A and Conclusion (1668.11s), Closing Remarks (2719.95s)
Transcript for "From Assay to Answer: Powering Diagnostics Workflows with Seqera":
Hi, everyone. Thanks for joining us today. We'll get started in a couple of minutes after we've given some time for people to join. We'll see you soon. Hello everyone and welcome to today's webinar from Assay to Answer Powering Modern Diagnostic Workflows with Saqqara. I'm your emcee today, Kate Mullaney. Before we get started, few housekeeping items. To start, please don't be a stranger. You see that Chris and Lizzie have already introduced themselves in the chat. We'd love for you to do the same. So please utilize the chat to introduce, chat about what you would wanna see or hear. We'd love to we'd love to get your input. Near the chat, you'll also see polls pop up throughout your, for you to answer, so we'd love to get your insight on how you how you are finding the webinar as we go throughout. And lastly and most importantly, if you have questions, feel free to put those in the q and a tab, and we'll try to address as many as time allows. And we'll follow-up with you if we don't have time to get to through all of them. I'm incredibly excited today about this webinar as I've spent the better part of the past decade sitting at the intersection of clinical genomics and the operational side of getting NGS workflows to actually work at scale. So from validating laboratory developed tests to overseeing professional services and customer success teams that deliver them, I worked with researcher and diagnostic labs around the world and have seen firsthand where friction lives between assay and answer, and most importantly, why solving it matters for the patients. My experience led to being a committee author for Clinical Laboratory Standards Institute where I help write the standard on cancer molecular testing and how to interpret, design, and consult on these assays, as well as being a chapter author on genomics reporting software in the version version two of clinical genomics textbook. At Saqqara, I'm the VP of professional services where I lead the teams that ensure that we understand our customers' needs from the first conversation, that they are onboarded quickly and are able to put our technology to work to get the right answers for their science. And with that being said, I'm incredibly happy to introduce the real star of the show today, a former chemistry Olympiad, Ken Brewer. Ken is a PhD scientist engineer with over a decade of experience working at the intersection of software, biology, and artificial intelligence. When he was at Yale, his thesis work, was on the development of new computational methods for discovering riboswitches and ribozymes through mine mining bacterial genomes. He was able to transition that work when he joined the flagship pioneer stack startup, ProFound Therapeutics, as one of its first few hires. There, he helped architect and build a multiomic bioinformatics platform to interrogate the function of thousands of novel proteins discovered in the human genome via translatomics. After ProFound, Ken shifted his focus to the production grade software required by patient diagnostics and joined GeneDx as a senior software engineer. Here, he architect and led the migration of GeneDx production bioinformatics platform from a legacy system running on HPC to a modern Nextflow based system running in a distributed multi cloud environment, making him perfect for what we're gonna talk about today. Luckily for us, Saqqara, got Kin around 2024 where he led open source outreach efforts to the community such as NF Core. And now as a scientific solutions lead, he works closely with diagnostic companies looking to solve some of their most pressing scalability, reliability, and regulatory concerns. And so without further ado, I am going to hand it over to Ken and let him take you through everything we have prepared for you today. Hi. Thanks so much, Kate. I'm super excited to be chatting today about, powering diagnostics, using Nextflow and Saqqara. This is a topic that is near and dear to my heart, having spent time in the trenches and seeing how impactful it is in terms of delivering good results to, patients that matter if you can manage to sort out the tech stack in a way that is going to work for you as a business. And because we wanted to be a little bit more focused on kind of, like, the practical realities of working with, in this challenging environment, we put together a fictional company that we're calling Meridian Diagnostics that we're gonna use just as an example for the types of challenges that we see among, the customers that we typically work with. And so for this case, we're gonna say Meridian Diagnostics is a mid sized diagnostics company focused on MRD testing from cell free DNA. Let's say they have one clinically approved test for non small cell lung cancer. And then after enjoying the benefits of being first to market, let's say, Meridian sees its competitors beginning to develop their own NSCLC MRD tests. And the opportunity that it is looking to capture is that both Meridian and its competitors struggle to accurately call structural variants from their cell free DNA, and they want to differentiate their product by building new software for, SV variant calling into their bioinformatics pipeline. Now the realistic timeline for adding new tech to a diagnostics pipeline, like, unfortunately, the reality can be as much as twelve to eighteen months. And so let's talk about some of the challenges that in our the the company encounters on the way to trying to bring this new changes, to their pipeline. So here we have three fictional employees at Meridian who each have their own headaches that they'll encounter along this journey. First, we've got doctor Lina Fisher, who's the head of the r and d Comp Bio Group. And her challenge is that her team wrote a new structural variant caller and benchmarked it against test samples on their research HPC. But now the production team says they need to rewrite it for scrap from scratch in order to work on the cloud based system that they are using for production. Then we've got Jamie Torres, who is the production bioinformatics lead. And then once he begins optimizing the structural variant caller in their test environment with some samples on the cloud, but begins encountering intermittent failures on certain sample types. And he and his team are wasting hours trying to track down the root cause of these various failures across all of the runs that they're doing, trying to validate, the this new variant caller. Finally, we have Scott Smith who is, in charge of compliance and validation efforts, for this project. The commercial team is pushing really hard to try and get this new computational prediction tool that uses this structural variant caller into the diagnostics pipeline, but he's concerned that we'll need to run a whole end to end revalidation project from scratch, and timelines on those are just very long because it sucks up a lot of bandwidth, from the, the the clinical team. So, let's talk about each of these challenges in turn and some of the ways that Saqqara and Nextflow, can help solve some of those challenges. So turning first to r and d and bringing up once again, Lina Fisher's complaint where, let's say, that she's written a new structural variant caller, benchmarked against some test samples on their HPC, and now there's the HPC to cloud migration effort where there's a need to rewrite a bunch of the software in order to make it work in a more scalable, and cloud native way. So one incredible piece of software that is directly addresses this headache is Nextflow. And if you're not familiar with Nextflow, Nextflow is a workflow orchestration tooling that has become the standard for production grade reproducible, analysis. It offers, like, a true developer friendly experience where you have very fast prototyping, but Nextflow also handles the parallelization, the portability, and the scalability. And it's something that you can write once and run anywhere. And Nextflow has seen an absolute explosion in adoption. That means that now it is really a mature and production grade ecosystem. The graphs on the left shows the number of academic citations that are using Nextflow to publish and generate real scientific results. And you can see, especially since 2022, there's been, a a explosion in the adoption of Nextflow when compared with other workflow orchestration toolings. There's over 20,000 bioinformaticians trained in the use of Nextflow, and over 3,500,000 executions of Nextflow that happen every month. So, some of the ways that Nextflow accelerates r and d timelines, for that r and d to diagnostics transition point is that with Nextflow, you can write the same validated pipeline code and have it run the same every time, and it will run identically whether you're doing some prototyping on your laptop, whether you're doing some testing on your research RPC, HPC, or if you are doing in a production setting like in the cloud. Nextflow handles all of those transitions seamlessly and can get the code to work across all of those environments without changes. Also, Nextflow makes it easy to pin the exact tool versions. You can use containerized environments, and that eliminates a lot of the it works on my machine failures that are common in that r and d to production transition point. Finally, there's a really strong library and ecosystem of peer reviewed pipelines, and pipeline modules from the NF core community that really accelerates the time to to build your first pipeline and to get your first validated run. So some of the the key benefits that we see there is that you can validate your pipeline once, you can run it anywhere in confidence, and that there's no changes in terms of, like, the logic of the pipeline between the different environments that you run it in. So, Secura platform is the enterprise solution for, bioinformatics at scale, and it is built by the company behind Nextflow. The original creators of Nextflow, Evan and Paolo, founded Saqqara, to be the place for all of the production grade tooling that a a company needs around running their Nextflow workflows. Some of the things that, we provide on top of Nextflow is having secure access via CLIs, APIs, UIs, and, software development kits. We provide a a foundation of quality and security, including SOC two compliance, HIPAA compliance, and and and support for meeting CFR part 11, compliance standards. What's also a distinguishing distinguishing feature of Secara is that it allows you to bring your own enterprise tooling and software and cloud environments, and it will allow you to deploy into your HPC, into your Azure, Google, or AWS accounts, and integrate all of the different pipelines, data sources, and interactive tooling that you need into one cohesive location. So we like to say that Nextflow powers the science, and then Saqqara helps power the organization that that needs to run on that science. Great. So, moving on in terms of, like, the types of challenges, oh, sorry. Great. So one some of the key ways that Secura platform helps eliminate some of those clinical bottlenecks in that r and d to production deployment is that it handles the promotion between r and d staging and production. So instead of needing to do hands on manual deployments for each of those environments with Secara platform, you can simply pin those different environments to run on based off of different Git branches. And that means that you can stay entirely in your GitHub repo and and manage things via CICD and best practices. You also can have separated workspaces with full run history and configurable permissions that allow you to have, like, seamless transition points between r and d, staging slash test, and production. And you do also get API triggered and scheduled launches, and that really helps in terms of enabling, like, same day turnaround time on high priority cases. And here on the right, we have an example of some of, like, the monitoring and observability pooling that comes, with Sikara platform over a a base next flow configuration. So that's a a good transition point to some of the production bioinformatics headaches that are typically encountered. So on one example and the example of the headache that we typically see from our customers, this is Jamie Torres, from his perspective that they're trying to optimize the new structural variant caller in their test environment, but they're encountering intermittent failures, on certain test types. And he's wasting hours trying to track down the root cause of these failures across different ones. So some of the ways, that we see that, are that are some of the most common failures that some of our customers report are are are these six things. I think these are some of the most common ones that we hear about. Things around unreliable compute infrastructure, incomplete data transfer, overloaded servers, system maintenance downtime, sequencing quality issues not surfacing until, it causes failures within the the bioinformatics pipelines and then also software bugs. And, luckily, all of these different failures can be supported and helped mitigated by using Nextflow and Saqqara, and it supports those different failure types, in different ways. And so let's talk about kind of some of the the key patterns that Nextflow and Saqqara have that that support these, production scale bioinformatics. The three things that we wanna focus on here is that, Secura platform allows you to bring your compute to the data, which gives you better organizational security, faster workflows, and lower cost. Second, it's really built for the surge. By leveraging cloud based platforms, it allows you to scale instantly, to meet the demand of your big pipeline run that just finished, your big, sequencing run that just finished without overbuilding. And finally, just in terms of, like, the observability and monitoring tooling, Secura platform makes it easy to ensure that no run gets left behind and that you can launch pipelines with confidence to make sure that they actually finish. So first off, in terms of bringing the compute to the data, this is a a real distinguishing feature that distinguishes Saqqara platform from other, bioinformatics types platforms that often require you to transfer data from your infrastructure and your accounts into their cloud accounts. Instead, fundamentally, Saqqara allows you to bring your compute to the data and bring the software to your cloud accounts. One great example of this is a high performance file system called Fusion that we've built, that allows you to treat, cheap, inexpensive object storage as if it were fast on instance local storage. And so in the standard model, when an a w when an Xflow pipeline runs, it needs to spend some time at the beginning, at the end, staging the data, copying the data from s three to the local instance, then it can kick off the compute. And only afterwards, it will then spend another long part of time doing the data upload. With fusion, instead, that data gets streamed directly to the CPU, and that allows you to sort of have the pipeline run much faster without that data staging overhead. And it also means that you're not creating duplicate potentially, like, insecure copies of your data, in different places. So this model ensures that your patient data never moves, and it helps satisfy data residency and sovereignty requirements right out of the box. Now another key aspect of Secara platform is that it is built for the surge. It allows you to scale instantly without overbuilding. Because we can integrate with any, cloud, hyperscaler, you can do cloud native auto scaling to surge your pipeline volumes without any manual intervention. We also make it very easy to use spot and preemptible instances that reduces your per sample compute costs. And finally, with resource aware task scheduling, that eliminates a lot of the overprovisioning on on prem clusters to help kind of make sure that you're being more efficient in terms of your resource usage. And we've seen that from, some of our our customers who've seen massive increases in their ability to handle NGS sample volume and also reductions in, their compute expenditure when they've adopted Secura based tooling. So, finally, in terms of, no run left behind, the observability and monitoring tooling that Secara platform has out of the box makes it easy to sort of build resilient execution from start to finish. You can have real time dashboards that will surface bottlenecks before they impact turnaround time. We have functions for automatic task retry. That means that if there is a failure due to sort of buggy or intermittent, infrastructure problems, you can have that get retried without any manual intervention, kind of fixing the problem before it surfaces. And then we also have checkpoint and re automatic checkpointing and resume functionality. That means that if you have an interrupted run, you're able to restart from the last successful task and get a pipeline finished to completion much faster than if you need to start from the beginning. And so that really allows you to sort of submit your run and walk away, and Secura will keep running it so you don't have to. And with our comprehensive API tooling, you can also build all sorts of additional custom monitoring and functionality around around the types of pipelines that you need to run. So next, I wanna talk a little bit about one of the most challenging topics, and that is, compliance. Now here we have our fictional Scott Smith. And Scott Smith's headache is that he's getting a lot of pressure from the commercial team to add this new prediction pool to their diagnostics pipeline, and he's concerned that we'll need to run a whole validation revalidation project from scratch because of this new, tooling that they're adding. Now, one of the first things that I wanna talk about on this end is kind of some of, I think, what are the key strategies for navigating compliance headaches from a bioinformatics perspective. And if there's one slide, for you to take home, from a compliance perspective, I think this is probably the most important one in terms of the things that are real game changers for allowing diagnostics bioinformatics bioinformatics teams at diagnostic companies to move much quicker. First, I think it's really important to separate your orchestration from your science. So you want to distinguish between your orchestration, which is how do you get the data from point a to point b, and your science, which is what are the command line tools that you're running, what versions, and what parameters. Nextflow allows you to do that very cleanly, where Nextflow is a orchestration tool that will move data from point a to point b and stage them in the appropriate raise for the different CLI tools that you use. While the the the the science, that's very, much more focused on, hey. What are the what are the scientific tools that we are running? What is the order that they happen in, and what is the dependency between them? And if you can build your compliance framework to distinguish between what is orchestration and what a science and to treat those differently and allow them to evolve separately, you can be, in a much more nimble position in terms of how you do your compliance, strategy. Another great tool on this front is writing performance tests. You can use Nextflow and NF test to write pipeline performance test that will evaluate the accuracy of, your diagnostics assays. A really great example of this that I wanna highlight from the open source community is NF core slash variant benchmarking. And this is a pipeline that makes it easy to benchmark your variant calls, across a set of known truths. And you can build, these performance tests that you write both at the individual, like, module or process level or at the bigger picture, like, pipeline level into a CICD framework, where you can use continuous integration and continuous deployment strategies to automatically trigger testing for your development environment and your staging environments based off of, the things that you're, that you're working on. So, how does the Secara platform make these things easier, to to manage from a compliance perspective? First off, with Secara platform, you have standardization without sacrificing your flexibility. So you because you have a centralized pipeline registry that can ensure that all of the different sites that you have are pulling from the same version code base, You can lock down the parameter sets to prevent clinical operators from deviating from your validated configuration. And by using resource based access control per workspace, that can ensure access segregation between the r and d and clinical environments. And all of those different environments can also run on the same types of, underlying compute infrastructure, And that makes it much easier to demonstrate that you have consistency across your, dev staging and production environments that you can showcase to an auditor. Another area that, is very helpful on this front is, the audit capabilities. So Nextflow and Saqqara platform will automatically log every single run that happens in platform. That ensures everything is going to be traceable and reproducible by design and not by documentation. That means that you're not spending time documenting what you've done. The documentation just writes itself. You have immutable records that are created, the pipeline versions, the containers, the parameters, the task logs. You get full provenance tracking down to a per sample basis to the individual task And then a user action audit log satisfies 21 CFR part 11 electronic record requirements, which is one of the most common things that customers ask about when it comes to, compliance front. So, and I mentioned before how, it makes it easy Secura platform makes it easy to make sure everything is locked down and versioned. Every run is going to be logged, traceable, and reproducible. And this allows you to make sure that, like, when your pipelines are pulled, you're capturing the specific Git commits, the exact code revision for every run. You do container digest pinning, which guarantees byte for byte identical software across reruns. And these versioned parameter sets will help tie you your pipeline releases and will prevent ad hoc config changes. And so at any point in time, even when you go back and look at past versions of your software where you've evolved your production software over time, you can go back to any point in time and rerun things exactly how they were without needing to change your underlying infrastructure. Finally, in terms of, like, the compliance framework, we like to say that Secara platform makes it so you don't have a a groundhog day where you're continually working through the same challenges over and over. You have less rework. You're doing you have a faster path to your clinical deployment. There's no translation layer at the handoff. The staging evidence that you generate applies directly to how stuff is going to run-in production. When you generate a new compute environment that requires only environment qualification, not necessarily your entire software revalidation, and the structured run reports can be incorporated directly into five ten k, IVD, and accreditation submissions. So, once again, the the tooling really makes it a lot easier to move from r and d to production without starting paperwork over and over again. So, kind of pulling everything together, we have those those uniform operations, the fact that the audit log writes itself, and that you're not revalidating everything across every environment over and over again. So, we're gonna move to to the q and a, soon. Before that, I wanted to give a brief plug for the Nextflow Summit by Saqqara that's going to be in Boston on April 28 through May 1. We have a great set of, trainings and, hackathons and events that are gonna be present there. On the training front, we will have both beginning and advanced Nextflow training for people who are looking to, learn or improve their Nextflow skills. We have a NFcore hackathon for people who want to spend some time working and contributing to open source, which is a great way to improve, your understanding of production grade bioinformatics infrastructure. The NF core ecosystem definitely is the open source community that has built a lot of the battle tested frameworks that are being used in production grade Nextflow pipelines. And so it's a great way to familiarize yourself with those production grade Nextflow patterns. And then finally, we'll have a lot of talks from customers, and other people from from the tech industry tech, and diagnostics and pharma space talking about how they are using Nextflow and and what are kind of some of the frontiers that are being pushed forward on the, clinical bioinformatics side. So, we're gonna open up a quick poll, before the q and a. And then, if you have any other questions, also go ahead and be sure to submit those, in the q and a, and then we'll start answering some of those questions. And I will invite Kate back up onto the stage. Perfect. Thank you, Ken, and thank you for walking through all of that great stuff. We had a lot of presubmitted questions as well as some that came in as you were talking, so we'll go ahead and get started. So the first question that came in as you were going was, from a local Saint Louis guy, so we'll start we'll start there. How do you guys develop Nextflow pipelines? Do you use Docker, Versus Code, GitHub Codespace? So you can be very flexible in terms of, like, how you develop your pipelines, and and you can use any of those tooling. So so my personal develop workflow is I I like to use Versus Code. These days, I'm very much on the Claude code pill side of the spectrum, so I'm using the latest AI tooling in order to write a lot of the next flow instead of, like, crafting it artisanally by hand. In terms of, like, the underlying software tech, yes, you do want to be using some kind of containerized tool. Docker is a great choice. That's kind of, like, the industry leading tool, for, the software industry and also more and more in, like, pharma and diagnostics as well. But, based off of kind of, like, if you're on an HPC setup, then you might need to use something else like Podman or Singularity. And so there's different options that you can use for the containerization. Perfect. And the other one that came in, as we were as you were talking was from Michael. And he said, could you please add some insights on Nextflow and accelerated GPUs in NVIDIA and, with the latest things like Parabricks and NemoClaw? Sure. Great. So in terms of working with Nextflow and accelerated GPUs, one of the things that's great about Nextflow is that it makes it easy to specify, different types of processes need to run-in different environments. And you can assign, like, your any type of GPU based task to run-in a GPU queue, and that will allow you to take advantage of, like, the latest NVIDIA GPUs, for scaling purposes. And some of the, open source bioinformatics pipelines, such as NF CORSERIC, which is a whole genome sequencing and whole exome sequencing variant calling pipeline, they actually have integrations for Parabricks already built in. And so you can see examples from that pipeline of how you can incorporate Parabricks and, like, that GPU accelerated variant calling into a complex hybrid pipeline that includes both CPU based tasks and, GPU based tasks. Now when it comes to, like, other NVIDIA tooling to be, aware of, I think NIMS is another technology to be, aware of. NIMS allows you to do, like, very small, calls for NVIDIA tooling that is running on hot and ready, like, GPU based instances via API calls. So I think and and we have a Nextflow plug in for doing Nims based calls. NemoClaw, that is going to be a little bit more of the edge case of what you might be doing. If you're not familiar with NemoClaw, that's basically an NVIDIA implementation of OpenClaw, which is generating your own, AI driven clawed assisted that you give access to various tools. So I I I would love to hear kind of how people play with neboclaw and Nextflow. But if if if you feel like, like, just texting your bioinformatics agent and asking him to write a diagnostics next flow pipeline for it, NivoClaw would be the thing that you might pull together for that one. Perfect. Thanks, Ken. And I'll also add that we're currently working with NVIDIA to do some benchmarking around Parabricks. And while we're doing a lot of work to look at ways that we can enable more autonomous research AI agents, NemoClaw isn't one specifically, but definitely stay tuned because AI is a big push for us this year. Let's skip to one that came in through a submission while they signed up. I think this one's a really great example of the power of our of our platform and AI, speaking of. How does Sakeira's workflow management compare with Snakemake in terms of scalability and reproducibility for diagnostic pipelines? What are the key challenges and solutions in migrating existing Snakemake pipelines into Sicara's platform? Great. Yeah. So that's a really great question. So I I I think if I if I go back to the the slide that I showed earlier where I showed the different workflow orchestration pooling, and their citations in the scientific literature, you can really see that Nextflow has had an explosion in the last few years, while Snakemake has stayed very stable, and and hasn't grown much. And I think part of that just comes down to the fact that Snakemake is, like, primarily an academic based tool, where development efforts are being driven by, you know, a small open source community of a few individuals who kind of work on it as their passion slash side project. Nextflow being backed by a a commercial company, Saqqara, that is continually innovating, adding new features, adding support for new types of cloud executors means that it's really become kind of the production tool of of choice for for modern diagnostics companies. Some of the specific capabilities, that I I think are also kind of distinguishing features between Nextflow and Snakemake is for Snakemake, you need to do a lot of rewriting of your Snakemake pipeline when you transition from one set of infrastructure to another. Specifically, like, going from HPC to AWS, you will have to, like, change a lot of the way that the pipeline is written, while Nextflow kind of allows you to simply do a few configuration changes instead of rewriting your pipeline logic. The other thing is that Nextflow has, like, better support for software containers and makes it easier to do, like, use containers in a way that are are going to be, like, scalable across those different environments. Excellent. And then let's skip back. Kim, submitted a question during your talk. Do you have any tips on monitoring the resource use for production pipelines? Yeah. Great. So this is one of the things that Secara platform actually makes very, very simple. So Nextflow does automatically generate some resource based information, often, though, that is kind of hard to dig through and hard to access. In Secara platform, we have resource monitoring baked in as a default feature, where for every single run, you can pull up a chart of the resource usage for every single task where you compare, you know, the amount of CPUs and memory allocated versus how much was used. And, we also have a compute optimizer tool that allows you to look at a previous, pipeline run, and then generate a optimized configuration that sort of rightsizes the resource request for every single task, along with some automatic scaling logic that will allow it to retry, with some additional resources on on on a failed run. And that kind of allows you to both rightsize your resource constraints but not encounter, like, the out of memory failures that would otherwise, be your main headaches. Great. And my institution is locked in to Azure slash Microsoft, and I need to run fully within their network. How can I do this with your product? Yeah. So this is another great way how Secura platform being infrastructure agnostics is a real game changer. So Microsoft Azure is one of, like, our supported integrations where, you can run pipelines in Azure, or AWS or Oracle. One of the things that, I think I I was really proud of at our the work that I did at GeneDx is that right in the middle of our, push to get to the cloud, we switched from one target cloud provider to another. And because we had been building with Sikara platform as the central orchestrator, we were actually able to migrate from one cloud provider to another without needing to to rebuild and rearchitect our pipelines. And so we managed to still hit our deadlines in terms of, like, the proof of concept projects that we were looking to deliver despite changing our core underlying technology halfway through. Perfect. And then this next one is a little bit more about, again, how we can be flexible within our software, and then we have two regulatory, more in line questions we can we can skip to to wrap it up. In a regional bioinformatics hub that's handling diverse data sets from malaria markers to cancer panels, what are the best practices for managing NextFlo DSL two modules to ensure they remain production grade while also still being flexible enough for different institutional needs? Oh, I really like this question. I actually have a blog article that I wrote about this that, will drop into the chat. It's about building an NF core style module library. Now one of the things that the NF core community has done is they've really built an ecosystem around how to define module libraries where you could have modules of different code that you want to be able to install into a pipeline. And in this blog article, I actually talk about kind of how you can set up your own version of this NF core module library pattern using, the same CICD framework that NF core uses and the ability to install modules from, like, your own institutional module library. We do also have some additional features that we're bringing out on this front, soon. We are adding support soon for, being able to pull modules remotely directly in a Nextflow pipeline, and we have a new module registry feature that we're working on that we'll be bringing soon. And if you want to hear more details about those features and how that will make it even easier for institutions to manage, like, code libraries, definitely tune in, to the NEXT Flow Summit, because we'll have some exciting talks on that front soon. Amazing. And going again back to kind of the idea of reproducibility and and production grade pipelines, this next question says, converting a NexSlow pipeline into software as a medical device, what are some ways we can ensure reproducibility of our analysis? Yeah. So this is this is a great question, and and I think it's kind of, like, part of the core of what makes it hard to develop bioinformatics pipelines in a clinical setting. The key strategies that I think value that that I think matter here is doing lots and lots of testing. And you kind of want to have a multilayered testing strategy where you want to be doing testing on the module base level. So for every single module that you include in your pipeline, writing a unit test to make sure that it does the thing that you think it does. Then you also want to write some sort of minimal test case where you have some minimal test data that you want to run through your pipeline, and that allows your development team to do fast iterations whether or not waiting for big production runs to complete before knowing is there a bug that's going to just break everything. And then you do want to have some, like, gold standard test data that you can run through for, your validation runs before you promote to, like, your, production or staging environments where you run end to end through, like, your set of, like, standards. And you can perhaps automatically generate some of the validation data from that pipeline run. One of the the proof of concept projects that I worked on, shortly before I left GeneDx was the ability to take NF test test outputs and the pass fail status of those different test runs and automatically generate a PDF report that says, like, hey. We've met we've hit these compliance requirements in terms of, hitting these particular performance benchmarks that we're looking. And so that sort of multilayer test strategy, I think, is the way that you build your software so you can, maintain compliance continuously but still have flexibility to evolve over time. Great. Thanks, Ken. And we have probably time for one more question. So there's two that are very similar in vain, so I'll kind of merge them together. Really hitting on, what our GXP offering is at this point. So if people are interested in more based on us not having time to answer, certainly do reach out if you'd like to know what our latest is on GXP. But kind of centering around how do you manage to make improvements to your pipeline while maintaining regulatory compliance with existing validation efforts? And then within that regulatory compliance, I we did touch on this in the chat, but, you know, understanding how Saqqara is compliant to the 21 CFP part 11. So I will hand it to you to speak to how you how you validate, while not having to completely redo validation and also just touch again on the fact that we are c, 21. CFP part 11. Yeah. So in terms of, like, how we maintain compliance, I think that continuous testing strategy is a core part of that. By having, like, your gold standard testing dataset that you can run through before you do any, like, push to production, that allows you to potentially show that you are byte for byte accurate and that there is absolutely no change in your data output despite having made some improvements to how the pipeline runs. And that's kind of where also maintaining that separation of what is orchestration and what is, like, scientific software, kind of allows you to, like, move more quickly in terms of, like, your compliance strategy. Like, if you can show, like, hey. This is an orchestration change, but the underlying software tool calls and parameters are absolutely unchanged between version one of this pipeline and version two, You can potentially evolve the pipeline orchestration and then kind of do the the scientific changes, like, more deliberately when you're already doing some sort of revalidation effort that's related to perhaps using, like, new, test new, sort of sequencing kits and so on. And then on the 21 CFR part 11 compliance front, simply having a software tool that automatically captures, like, all of, like, the actions that a user takes, the change reason, like, that kind of really makes it a lot easier instead of needing to manually document all of those changes, yourself. Awesome. Thank you so much, Ken. And I think we're right at time and we're able to get through most of the questions. So I wanna thank everyone for joining today and for giving us so many good opportunities to have a good discussion. And we will follow-up with you if we have didn't get a chance to answer your questions as well as with any additional information. Thank you all.