Name: AI-powered Development, Expanded Data Integrations, Optimized Compute: What’s New with Seqera
Uploaded: 2025-11-19T18:16:36.530Z
Duration: 40 min 44 s
Description: AI-powered Development, Expanded Data Integrations, Optimized Compute: What’s New with Seqera

Transcript for "AI-powered Development, Expanded Data Integrations, Optimized Compute: What’s New with Seqera": Alright. Let's get started. Say good morning, everybody. I'm Edwin Miller, bioinformatics engineer here at Secura. So our webinar series on AI powered development, expanded data integrations, and optimized compute. So with me here today, I have Rob Newman, our product manager lead here at Secura, and Sasha, our head of AI, who you may have heard from before. So just a couple of housekeeping things. First, please use the q and a to ask questions. So you can click the q and a icon at the top up there, and then you'll be able to submit those, and then we can answer all those at the end. So, also, the chat tool will be only used for attendees to introduce yourselves as well. You can also answer the polls that we run at the end, of the session. And then lastly, we'll also send out a recording in an email in a day or two as well, so don't worry about that. Okay. So here's our here's our agenda for today. So we've got the next little improvements that I'm gonna go over here. And then Rob's gonna talk to you a little bit about data improvements and a couple of things that we have coming down the pipeline there. And then Sasha is gonna talk to us about AI, of course, and the, Secure Compute free tier as well. So Nextflow, if you're not familiar, is an open source orchestrator, for reproducible scientific workflows. It's become the leading bioinformatics workflow manager in recent years, and it's grown, because of a couple of key features that we have, such as, like, container management using various different, grid executors as well. And then also, you can prototype really quickly, parallelize your workflows, and just kind of the portability of it all. So you can write it once and then run it anywhere. And we just really love love Nextflow here at SCARA. And so in twenty five ten, we've got some exciting things coming out. So we've got the workflow input and output, generalization on those, and that's kind of an updated syntax piece. Then we've also included static typing in the languages, and so that'll actually help us with inputs and various outputs and kind of standardizing all of those. Then we've also sped up some s three transfers as well. And then we've also got a couple of new commands, as well for you. And so if you're interested in that, then you can join our webinar on December 11. Just sign up here, scan the QR code, and, you'll hear from Ben and Phil, I believe, on that, and we can really deep dive into those features and get your feedback as well. With that, I will pass it off to Rob here to talk a little bit about biological data and growing it in volume and complexity. Thank you. Am I on stage? Yes. I am. Hey, everybody. It's great to be here. I am gonna be Evan, if I can share my screen. I think I'm you're still sharing your screen. Thank you, sir. Alright. Okay. So in terms of how complex the biological data that is growing and and what we work with is pretty wild. We're seeing an explosion in the increasingly complex types of biological data generated, and it's not slowing down anytime soon. Each of these data types, as you know, have their own formats, their own tools, and their own processing requirements, and you need a real flexible infrastructure to be able to keep up with that. If you look at what's happening in our community, data sources are also changing and diversifying. Over the last decade, as I mentioned in the summit keynote, most of the data storage was standardized in a few common places, like object storage, on prem file systems, or in your LIMS or ELNs. And this was really centralized, predictable, and traditionally managed by your IT teams or just by your on prem file systems. And this gave us a lot of control, but it also created silos. And there was and there's been an increasingly a disconnect between the experiment and your analysis. Today, that's amplified. Data is stored as different types. There's lots of different data management systems, and it's looking different. There's been a surge in dynamic cloud native platforms. The data is generated but also stored in tools like Benchlane for the wet lab bench scientists, Hugging Face for your machine learning models, and Databricks and Snowflake for large scale analytics. And in today's API first world, data flowing in and out of these systems can come from anywhere, and your platform and infrastructure needs to be able to adapt to that new reality. And some of these new features that I'm gonna be talking about today in Seqara platform really, amplify and leverage, some of the skills around getting this data and displaying it in our platform. So what I'm gonna be talking about in platform today is additional support in data explorer for s three compatible storage, the totally revamped datasets user experience, pipeline versioning, the optimized compute that we now offer, and also the Studio's Git integration. So Data Explorer now supports s three compatible storage. As you know, you've always been able to access AWS, Azure, and GCP object storage. We've expanded this to be able to be compatible with any s three compatible storage. So whether that's additional cloud providers like CloudFlare, Oracle Cloud, Minio, or LakeFS, or whether it's an on prem file system. So you can now explore, preview, and select data adjacent to your object storage without switching tools or within Secara platform. And this creates a truly unified bioinformatics experience where your data interactions and your compute are centralized in a single platform. So what have we done with datasets? How have we improved datasets? We've completely rebuilt this with modern design components that actually bring it up to parity with the rest of the platform. Similar to what Data Explorer looks like or what Studios looks like, We have these modern design components with met with increased metadata, and you can filter by these metadata values. You can also have a one click view in datasets to see the number of runs that have used that dataset as an input. So, there's a column called runs. It has a hyperlink there. You can just click that and jump straight to the pipeline runs page, and she just shows all of the runs that have used that dataset as an input. You can also hide rarely used datasets. So if you can see in the, replay that we have here, there's a show. You can have hidden or visible. And so we've had a in the past, we've had a quota for the number of datasets. That quota is now gone, and you can add any number of datasets. And you can filter this view very easily by choosing to hide datasets that are rarely used. You can also now add labels. So this is another way of really simply categorizing and displaying the most important ones to your experiments. We also, have the ability or or we're going to have the ability to, support JSON and YAML formatted data as well. Currently, only CSV and TSV are supported, but we're expanding the dataset's functionality to support JSON and YAML. Okay. Pipeline versioning. So Sagera has always been the the place to run Nextflow pipelines. And throughout 2025, we've worked hard to make the everyday experience of running pipelines even better for you. If you remember at the Boston Nextflow Summit in May, we presented a new workflow run details page. And since then, our pipelines team have focused on a much requested feature related to pipeline reproducibility, Going beyond what Nexpo provides, like specifying the commit ID and the parameters that you used in your pipeline run, we wanted to extend this further to the broader pipeline execution context. And today, you can do this with pipeline versioning. So now every single launch in Siquera is tied to a snapshot in version. This makes development and iteration in r and d really fast while enabling the publishing of pipelines for broader use cases. So you can publish specific versions and also create default versions of a pipe of the pipeline. So now you can choose a branch and pin it so that the commit is always used even if someone commits to that same branch later, and you also have the ability to use the latest version every time. Importantly, every launch of a pipeline with a config or a parameter change now creates a draft version. This includes check sums, so you have confidence that nothing has changed and these really act like fingerprints for every pipeline version. We've also added custom schema. So using the Nextflow schema design, you can create launch forms in Sakara platform with just the inputs that matter to you. You can take any NF core pipeline, like the one shown here in the in the screenshot, and really make it simple, exposing only a single configuration option to your users if you wish. So it's no longer just versioning the the next flow code. Pipeline versioning brings us enterprise grade control and a new level of re reproducibility to Secera platform, ensuring that every pipeline run is traceable and auditable. We've also rolled out our new optimized single VM compute environments. Many of you have, used some of these, and you've seen the improvements that they bring. We built these because batch services require complex identity management policies and configuration of multiple services in con including compute environments, job queues, and job definitions. The new single VM compute environments have a very opinionated defaults to provide the best Nextflow pipeline and studio session execution environments with both wave and fusion enabled. Long launch delays due to batch services needing to provision the associated compute service to run a single job or a limiter and also using batch and other cloud, batch providers, there's allocation constraints. They they have a strict per region resource quota. For example, AWS only allows you to have a hard limit of 50 job queues per account per region. That constraint is removed using the single VM compute environments that we provide. So they're simpler, faster, and optimized across all the major cloud providers. And finally, what I wanna talk about is the Studio's Git provider integration. So this is a way of defining your Studio configuration in code, and what this allows you to do is build truly reproducible analysis environments. Also, as part of this feature release, you can build custom environments with just a Dockerfile. So instead of you having to define your own Dockerfile and compile it into an image or a container, you can now just point at a repository that has your Dockerfile there, and we build that build the studio for you from that Docker file. The Studio's integration, with Git also allows you to clone the repository contents into your session, and this allows you to collaborate, share, and deploy your studios much faster. And I'm actually going to jump and do a live demo of this, so I'm gonna stop sharing. I'm gonna go and pick a different tab. And what I'm gonna first share is what that studio's configuration looks like. So here we have a repository, and in that repository, we have a dot sakera directory. And inside of this, we have a studios configuration dot YAML file. And you can see here we have the schema version. We have the session details, so the name, a description, the template that is provided. Oops. And you can see the compute requirements. So if we're gonna use AW batch, the number of CPUs and GPUs and the memory. And then also you can define environment variables. And for the session management, the lifespan hours, and if you want that session to be private or not. And as you can see, we've got very or multiple different schemas defined here. So we've got a registry one, for example, and it's pointing to a registry here. So what does this look like? How does studios interpret this? I'm gonna jump to a different tab. So here, we're in the studio's interface, and if we hit add studio oh, I'm just gonna hide this drawer. K. So this is a new form, a much simplified form for studios. And we can go and grab this URL for GitHub. And in the git repository URL, we can just drop that in. And similar to how pipelines worked with getting NF core pipelines, it's gone and grabbed the master repository by default or branch by default. Sorry. And you can see here, it's actually reporting, okay. I found this configuration. And it's now saying, hey. Some of the forms and some of the fields in this form have been prefilled from the configuration. So the repository defined that we're gonna use a prebuilt container image. K? And here's the pathway to that image. Here's the compute that it's gonna use. Here are the details, and these these are just defaults provided by the the repository. We can see here there's a description, and these can these are overridable if you want to. If you want to experiment and take your base configuration provided by your git repository, you can override those in here, and you can build the studio and run it inside. This provides a centralized location for you and your team to manage your studio's configuration. And that's it from me. Those are all the latest releases that we have in platform. You might have noticed the label, in the top right during my presentation. Some of these features are in private preview. If you are interested in them, please reach out to us, and some of them are in public preview, which means they're they're available for you to use. So with that, I'm gonna pass it on to Sasha, who's gonna be talking about the latest developments with Secure AI. Hi, folks. My name is Sasha DeGhaev. I'm the head of AI here at Sakara. And today, I wanted to, walk you guys through some of the amazing features that we have, some of the amazing AI features that we have integrated with Secara. Before I start, though, I'm going to post a link to the in the chat to our, compute service. So everything that I'm showing, that I'm gonna be showing today is, running on the Secara platform. Obviously, that is, very, very useful if you are already a Secara customer. So you can go and use all the features that I'm showing right now immediately. However, if you're Secari curious, we have a $100 credit and, optimized compute service for you folks to try. So I just posted the link there in the chat for folks to kind of, to try to kind of experiment with and to try out. Okay. Cool. So I'm going to share my screen and show you guys some of the new features that we have. Alright. Alright. Cool. So the Securia AI integration with platform is really about a couple of key, key workflows. Number one is debugging. We think that we've built the best debugger for Nextel pipelines. Tell us otherwise if if, if you disagree. But I think for, Secara, especially for for Nextiva runs and especially for Nextiva runs on Secara, I think we're the best place to debug pipelines and to understand, what's going on. Number two is I think, the reason to use Secara AI is to generate data, data sheets as quickly as possible. As a buying practitioner, I've never met another buying practitioner that loves making sample sheets. It's one of those things that's, you know, very monotonous, very, annoying, but we think that using, AI is a great use case to do that. And then, number three is it enables everyone in your lab to do the simple things. So things like running a pipeline, things like interpreting results. Because of the chat interface is so universal, this is a great kind of bridge bridge interface to give to your, wet lab colleagues to kind of, get going and to try stuff out on their own. Okay. Cool. So now for the demo and to kinda show you guys what we can do today. So, for this part, I'm, again, going to paste the link into the chat. So, this is the link for this product today. This is for this is this chat, again, is available for all folks and is free to use, but, obviously, it's more impactful if you have a a Secera platform account and if you use the, the and if you load it up with some of the compute credits because then you can actually see the AI, the AI running some through some of the things. So when you log on to Secera AI chat, one of the things that you'll notice is you'll notice these four tabs here. You'll notice that runs in pipelines tab, a code tab, a data tab, and a compute tab. These are all the things that our agent allows you to interact with. So in this compute tab, I'm gonna select my compute environment, and I'm gonna do things like, hi. Can you please tell me about this compute environment? And here, we'll see Secera AI get the compute information from here, and we'll see, hey. This is a compute environment that's set up on AWS, and it's Secera optimized. So again, what this means is that it is optimized for, running Nextflow pipelines. What that means is that it has WAVE enabled and it has Fusion enabled which makes running your bioinformatics pipelines very, very easy. But you can see here that it's very, very easy for you to understand exactly what your, it it makes, your permissions and your compute environment very, very easy to interact with. Number two is you can also, you can also choose to interact with some of your data objects. So when you click on data, you can see there's two things here. There's data links and there's datasets. On our data links, we can go into our example dataset here and you'll see this comes preloaded with every secure account. We can see that we have this RNAseq folder and we see we have some test data in here. Okay. You know, let's say that we want to make a sample sheet for our, for, for an experiment that we wanna run. Well, one of the things that we can do is if we click back to the Secure AI chat, we can add in a data link. We add an example dataset, and we can do things like, hey. Can you please create a dataset for my RNA experiment? And so what Securier AI can do is that now it, is that is going to kind of give me an understanding of, hey. Here's how here's here's the, example format of of how I'm going to, do it. So it's requiring a dataset name and the two the sample information. So we'll say, yep, that For the dataset name, please use RNA Seq dataset webinar. This format works. And so now what you'll see is that Secura AI will create this, create this dataset, And we can see this dataset is created in here. Right? And so we can see that, Secur dot ai makes it very, very easy for you to interact with all the datasets that you have, specifically on your, in your that are connected to your Secura platform. Now, for most of the time when we run, we also typically, typically, we'd want to also be interacting with some of our our, code. So in this case, for folks that are new to Secera AI, we also have an integration with GitHub where if you give us your GitHub token, we give you the option to make, code changes directly to your code base based on the, based on a change that you wanna make. So in this use case, we have a a repository here that I've made called test r n a test r script, which is a large r script that, you know, isn't ready for Nextflow. Well, Well, one of the things that we could do is we can we can link this repository, and then we can do things like, hey. Could you please convert this script to Nextiva? Now what Secure AI is going to do is that it's, is that it's going to ask, okay, the repository. Sorry. So we are doing a live demo as you can see. The repository. Tact. More again. Is that it's going to go ahead and clone that repository and it's going to start looking at the at the information that is in there. So after it looks at the after it looks at the repository and it looks at the at the, at the file, it's now going to create a plan of execution of how it's going to move this file into a, into a Nextflow pipeline. Right? So for bind partitions that interact a lot with code that is not yet in in that's not yet in, Nextflow, you can use Secura .ai to very, very quickly move from, you know, from from scripts like, Python, r, bash, snake make, WIDL, whatever whatever it is that you're working with and move them into Nextflow using Secura .ai. Now this is a fairly complicated, task. So this will take about, you know, ten to twenty minutes to actually, to actually, complete. So what we're gonna do is I'm just going to show you the, the resulting pull request that this information that this, pipeline can can make. Right? So we can see here that it takes a R script and it completely, you know, and it breaks it out into several different individual modules here and kinda breaks this all out for you, into a separate branch that you can then use. Then if you want to review it, obviously, you can copy that branch, pull it down, and, kind of work with it, test it, make sure that, see see how it see how it's working for you. Then, so yeah. So this is our integration with GitHub. Now I'm going to create another window here and show you our last level of integration and that is with our runs and our pipelines. So you can see here, I have access to my runs and my pipelines. So one of the the most popular way that folks are using secur.ai today is actually for debugging, debugging specific pipelines. In here, I have our runs and I see that this run here failed, this boring Solvay on the nf core r naseek pipeline. What I can do is I can tag it here and I can say, can you please tell me why this run failed? Now, Secure AI knowing exactly what this run is all about, what it'll do is that'll first retrieve the run logs. We can see those here and then it'll retrieve the run tasks to understand exactly which tasks failed and then it'll retrieve the run task logs to get even more specific information on what failed. It's seen here that, hey, it looks like that there was a licensing issue with some of the authentication that was used for when this run was kicked off. Again, these are the types of, you know, very, very common issues that folks can run into while they're working in a bioinformatic setting. And we just wanna make it very, very simple for folks to, debug runs while using Secara, especially for runs that are on the Secara platform. And, yeah. So, to kind of get back to the presentation, I wanted to kind of, reinforce the fact that we do have the free $100 credit right now for folks to to try this out. And so if you head to the, to the link, you'll see an a great opportunity to kind of, try out a lot of the stuff that I just I just showed you. Okay? So I'm gonna pass it back to, back to Edmund to, to wrap us up. Okay. Thanks, Sasha. Okay. So we've got a quick poll here on those that we're gonna run. Okay. And then let's move on to the q and a. Okay. So let's go into the questions here. So we got a first one here from Alex. Good to see you. So he's asking Rob about when we will be able to have different data formats on platform and custom data formats here specifically. And I think I'll let Rob answer this one actually. Yeah. Am I am I I'm on stage. Yes. I am. Yeah. So, yeah, it's great that you're excited about the the new datasets formats. Yeah. So we have a feedback area where any customer or any user can submit feature requests for us. They They are anonymized, so if you created a create an account, you you will have your name, hidden, except for us who are admin users, but you can submit any feature request. There are specific feature requests there related to datasets and including the formats of datasets. So ones that we have heard directly from our customers are support for JSON and YAML, but we've also heard from customers about, Parquet, as a format that they would like to see supported in datasets. So I you know, please go ahead. You can use the search feature within feedback.secaro.io to, try and filter down ones to see if some another customer has already submitted a feature request that you wanna submit, but otherwise, you can just go and upvote an existing one. And that goes for anything within platform, anything related to Fusion, Wave, Secera AI. We have different different queues, I guess, for for different parts of Secara. So, yeah, please please, vote early, vote often. K. Great. Thank you for that. Next question here, we've got, from Owen. The so for data explorer expanded support, could you clarify what that means by local file system browsing or just other cloud? I believe this one's for Rob. Yeah. It's, it's both provided they have an s three compatible API. So there are I think, an an example is Weka. Right? So Weka is an HPC file system, but it has an s three compatible API to it. So provided you can get the quick credentials that allow SCARA platform to act to access your Weka environment and you put in the API path, you can connect to your HPC file system. K. Very cool. I got another one from Alex. What does Secura AI get trained on? Is it just, core Nextflow modules, sub workflows, pipelines, or is there more to it? Is the agent able or good integrating knowledge from user inputs or and from the lab as well? Yeah. So the, so the core of it is that we're very, very, very, Nextflow specific. So I I think most of our, most of our, users are, you know, asking Nextflow questions. And so that is the core thing that we that we focus on for the the materials that are input. In terms of the knowledge sources, it's all publicly available data, which is the Nextflow, which spans from Nextflow, Nextflow docs as the core source of truth. The I think the biggest kind of, like, difference between us and, you know, using Chad GPT or Claude as your daily drivers, specifically for Nextflow, is that we're more, we're more careful for the recommendations to be of what is current for Nextflow and not what is in the training data. A lot of the stuff that that those systems favor is, Nextflow data that is, you know, a little bit dated and is in a, older format, that was called DSL one. And we're right now in a DSL d DSL two. K. Thanks for that, Sasha. Okay. Follow-up question here. I've reached 200 questions asked from Secure AI. How can I get, more access to more Secure AI, chats? Yeah. That's a great question. So, the easiest one is to become a secure customer. If you're a student or if you're in the academic lab, we have an academic program that also loosens a lot of those a lot of those, a lot of those limits. Yeah. It's a great plug for the academic program. We'll definitely send a link out for that as well. Okay. I got another question here of, are there any limitations of the new single VM compute environments that you talked about earlier, Yeah. That's a great question. Rob? So there are some limitations because your Nextflow pipeline or your studio will run entirely on a single e c two instance. If that instance doesn't have enough resources, then the pipeline execution of the studio launch will fail. K? So for this reason, the number of tasks Nextflow can execute in parallel is limited by the number of calls of the instance type selected. So if you need more computing resources, you need to create a new compute environment with a larger instance types. So if you look at the when you do go and choose the AWS cloud or Azure cloud or GCP cloud, VM type, you will see that the form to complete is much simpler than when you're trying to complete a batch compute environment. And there but in the, advanced options, you can choose an instance type. So, it's sometimes better to, on-site of caution and maybe increase the size than what you may need, and then maybe pare it down later on. But it's it is significantly cheaper, it's faster, and it's more optimized for Nextflow and Studios. But I would would say that if you have a really big and complex pipeline, then it's less those, VM types are less suited for that. K. Great. Thank you. Any more questions from anyone? There was one there was one question that I got, about the studio's Git provider integration, and someone was asking if they could see, a raw Dockerfile. And I'm I'm happy to do that. I can I can share my screen and kinda show what that looks like if that's if that's helpful? I just need. to yeah. I need to be able to just share my screen, which I don't believe I can do. There we go. Now I can. Okay. So let's jump into one with a Docker file. K. So you can see here, if we go up to the dot directory, we have the config file here, and then we have the Docker file right here. And if we click into the config YAML, you can see it's a local reference to the raw Docker file. So it says Docker file, Dockerfile here. Okay? So let's go back and let's look at that. So for those of you who haven't created a, a custom studio before, what you need to do first is pull in our connect client container and a version of that container. And then that basically gives you all the secret source that allows you to build and run a studio a custom studio in in platform. And then you've got all this additional configuration. So this is a raw Dockerfile. This is relatively complicated because it's doing quite a little quite a lot with Versus code here. We could definitely jump down to one Dockerfile simple. K? And if we look at this, this is a super simple one. Right? All it's saying is, okay, we're defining an argument saying the connect version we're gonna use is 0.8, which is our web server that does all the the backwards and forwards communication between platforms and studios. And here's the base image. So you're just substituting that argument in there, and then you're just grabbing the Ubuntu image and installing Python. And then you're running a super simple hello, and then you're all you're doing is using bash to fire up Python, an HTTP server on the connect port. So, typically, that's like, you know, whatever port you want, but that defines how Studio's through platform communicates. So they again, Docker files can be as simple as you want or they can be as complex as you want depending on what you're trying to write. We do have a couple of blog posts, that the scientific development team at Sakera have written and published a couple of months ago that talk I think they they give examples of using Streamlit. They give an example of using Shiny. I think Dash as well. There's there's three or four, maybe even four or five different custom Dockerfiles that show you step by step how to build your own custom studio and deploy that. And now it's much easier because prior to this git provider integration, you would have to build that Dockerfile and put it in a container registry, either public one that you have access to or a private one in ECR. Now you can just give it the raw Dockerfile, and it will build the container for you and provide it for you. So it's much, much simpler. K. Great explanation. Thanks, Rob. Okay. So with that, thank you everyone for attending. Please let us know if we can help in any way, and feel free to reach out. And, look forward to seeing you at the Nextflow deep dive here in a couple weeks.