Video: What's New in Nextflow 26.04: Strict Syntax, Records, Static Typing & the Module Registry | Duration: 3732s | Summary: What's New in Nextflow 26.04: Strict Syntax, Records, Static Typing & the Module Registry | Chapters: Webinar Welcome (86.505s), Language Updates (216.93s), Nextflow Core Features (612.5s), Static Typing Deep Dive (825.545s), Module System (2276.935s), Meta Maps Future (3163.945s), Migration Tools and AI (3285.54s), Closing Remarks (3533.92s)
Transcript for "What's New in Nextflow 26.04: Strict Syntax, Records, Static Typing & the Module Registry":
Hello, everyone, and welcome to today's webinar. We're just warming up and giving everyone a few minutes to kinda join and and get started. While you're waiting, why don't you drop in the chat and say hello? Maybe tell us where you are. I think we've just started a poll so you can, tell us which versions of Nextflow you're you're using, and, and kind of get excited for the webinar we're about to do. Thank you everyone for joining. If you're not sure what I'm talking about with a poll that's a little bit on the the sidebar of of the the webinar interface, it should say chat poll and q and a. There's a couple of posts, coming up in, in the chat. Hi, Peter. Welcome from Belgium. Today's format, we're gonna have, Ben and I chatting. I'm gonna do a quick intro with a few slides, and then Ben is gonna do the really cool and interesting stuff where he's gonna go and do some live demos. If you've been to one of our webinars about Nextflow before, you'll know the format. And then we've tried to plan it so that we have a nice little time for q and a at the end where everyone can kinda jump in and chat. We have a bit of a history of getting too excited and and not leaving very much time for q and a, so we'll we'll do our best. I promise. Some great places coming in here. We've got Boston. Ben, I already know you're in Texas. That's cheating. Netherlands, Sweden. Yeah. Sweden. Cambridge, Copenhagen, San Jose, Boston, Illinois. That's just great. What a global spread. Who'd have thought it? I know it shouldn't be a surprise now, but even so, I still get really excited whenever I see how global our communities come and all the kind of crazy places when, all the different kinds of work that people do using Nextflow for. It's really, really inspiring. Francesca, welcome. Okay. Let's get started. Just giving everyone a a few minutes to get started. So let's see. Firstly, you're welcome. My name is Phil Yoons. If you don't know me, I am a senior product manager for open source software at Secura, and Secura is a company that is responsible for Nextflow and other projects which go around it. Today on the webinar, we have myself, and we also have Ben Sherman who's senior software engineer at Next at Secura and is one of the key people behind, building Nextflow along with Paulo and and Jorge and the rest of the team at Secura. So there really is no one better, to know about all the the internals of of Nextflow and the motivations for why we're doing it. If you're new here, then, maybe you haven't seen this slide before, but just to give some context. Nextflow is we're getting increasingly confident now about saying it's kind of the the standard for running bioinformatics workflows or or really any data analysis workflows in the life sciences. We've got this lovely plot here just updated with citations from 2025, showing that more and more people are using Nexo every day. It's got this real momentum as you're really growing. And then the number of runs is just kind of crazy. We're kind of getting more and more traffic and, and we're running hundreds of thousands of Nextflow runs per day, which is both terrifying and, and fantastic to see. So today's webinar is all about the latest release of Nextflow. We have two stable releases of Nextflow every year. And so the latest one that went out just recently is twenty six zero four. And it's kind of a lot of stuff goes into these releases, but there's kind of four main kind of categories, which we're gonna touch on today. The big one really is language updates. Many of you will know that we've been working through a series of kind of new changes to the syntax. So which you use to build your next flow pipelines, and that's gonna be really the the vast bulk of what we talk about today. You might have also seen if you've been keeping up with the news that we've, extended the next flow registry, to include modules now as well as well as plug ins. We have, new features around data lineage and how that can be consumed. And whether I've just got, like, one slide on a very cool new feature we're developing at Secura called intelligent compute just because I think it's interesting for a lot of people in this audience. Okay. So language updates. Why why are we changing things? Change is difficult. Change is uncomfortable. It's always hard when, you know, you kind of just get used to something and then it starts shifting. The motivation for all of these changes really goes back way back. We started doing Nextflow surveys in around 2022, I think. And we already knew what people were gonna say before that, to be honest. People told us again and again and again that the Nextflow is great. It was super powerful. But the error messages are just really difficult to work with. They're really vague. Often, you would only find out that there was a problem in your pipeline halfway through the run, which could be hours or even days in, which is really frustrating. And and it's kind of groovy syntax people are not super familiar with and had some kind of weird intricacies, which were a bit nonintuitive. Now back in 2022, we didn't really know what to do about this, and we kind of nodded and agreed. And we've been sitting on it for a long time. But we started to kind of think about how we could really approach this to the fundamental layer level and and kind of bring real solutions to the people writing Nextflow pipelines. And so we started building. And in 2024, we started shipping a couple of releases, which were the start of this kind of wave of of fixes. The big one was really in around twenty four ten where we brought in a new syntax, language server. So this is basically a from the ground up rewrites of how Nextflow understands and passes the Nextflow code that you write. Instead of it being a DSL layer on top of Groovy, which was the limitation for so many things like error messages, and now the actual language was passed by a bespoke language. So it wasn't just a DSL for Groovy. It's really next local language. That was the seed. That was the start, and we've been rolling this out progressively ever since. In twenty five zero four, that went from just being in Versus code into core Nextflow. We had some new functionality as a result of that, such as Nextflow lint commands, which can check for errors and formatting. And then we start to bring in things like outputs and inputs, and then static types at the end of '24. That's meant to say twenty five ten. Sorry. And then, really, twenty six zero four is the end of this wave of new syntax. And so this has been a really long plan that we've been thinking about for a really long time and executing for a really long time. And twenty six zero four is is really the last big step where we're bringing out new changes. Okay. Oops. So very, very quickly, Ben's gonna cover this in more detail, but strict syntax, is now on by default in twenty six zero four. Before, you had to enable it. Now it's on by default, and you can disable it if you want to, but it's kind of a new standard for any new pipelines. This means that we can have really amazing error messages which point you exactly to the character in your code which is wrong rather than just kind of blanket saying there's something wrong in your pipeline somewhere, with really beautiful colors, of course, and all that. We have typing and parameter inputs for your pipeline. So we have strict typing so you can really enforce the kind of data structure and data types that you're working within your pipeline. Renew for twenty six zero four, we have native record types, which is something that Ben's gonna talk about in a lot more detail. And really, this is a way for you to structure the data that flows through your pipeline. Nextflow really what differentiates Nextflow quite against other workflow managers and other things like it is it is very data centric. Bioinformatics is a data centric field. And record types give us a way to understand and structure our workflow around the structure of the data that we have in very intuitive and powerful way. On the back of record types, we have now static types and static type within processes. If you're not familiar with static types, all I mean by this is you're defining everything, whether it should be like an integer number or a string or Boolean true false or file path, things like this. You as a pipeline developer defining those upfront and then they're enforced throughout and errors of course early. It's one of the most requested features we've had for Nextflow for years, and it's now infiltrating through the entire language. Then very quickly, on from the language, we've got the Nextflow module registry. If any one of you if you've previously used the NF core module system, it's it's got a lot of overlap. It's basically we've tried to kind of mirror that functionality, but brought it into core Nextflow. So now it works with any Nextflow pipelines, any Nextflow modules, and you anyone can publish them. We're not so it's encompasses the NF core community that goes beyond it. Really, really powerful. And we've already seen great uptake of this in the last couple of months since it's gone live, which is fantastic to see. You can pull them really importantly, you can run an individual module now by itself. You don't have to build a rapid pipeline. If you just wanna run fast QC or you just wanna run multi QC, you know you have a module for it, which already has the inputs and outputs, already has the software you need to run it, all of those things. You can now run that module in isolation, which is fantastic, especially for a agentic use and AI. And then you can install and publish. So I know I'm whipping through this really quickly. I wanna give them lots of time to talk about this stuff in detail. Right. And then, last couple of slides, and then I'll let Ben talk. Data lineage, this is something we brought in last year, and it's a way that Nexo describes all of this data model it has. It has all of this information about your channels, and everything. And in the past, all of that was basically lost at the end of the run. You published your your BAM files or your output files, and that was it. Now with data lineage, we keep all of that information and publish it in a as a format, in JSON files so it can be consumed. We released this last year and it was very powerful, but many people didn't really know what to do with it. It's a lot of data. It's quite complex. And now in secure platform, we have native support for data lineage, and we're starting to drip feed new features which are based on it. So in Boston, we released a we showed a demo of how to use this. And now with if you're using secure platform, you can trace you can look at any file in data explorer and see exactly which workflow generated it, which task, and click on it and see the script that generated it. Then you have full traceability through the platform, which is super, super powerful, and we've got more goodies to come as well off the back of that. And then finally, I said I'd mentioned it, secure it intelligent compute. It ties into Nextflow. It's not really part of Nextflow itself. What this is is a new scheduler for scheduling jobs, primarily across all the different cloud compute systems. Basically, we've been working with systems like AWS batch for many years now. And while they work very well, we can see places where we think we can do better. And and secure intelligent compute is basically built around all this extra information that we know so that it can schedule jobs and hack jobs onto, machines in the cloud as efficiently as possible. It all works pretty much automatically. You don't need to give it very much configuration. And then we're we're starting to do our real in-depth benchmarking now and seeing massive cost savings, which is really, really cool. Yeah. Okay. That was, like, super fast. Don't feel too worried if I you'd kind of feel overwhelmed by this because now I'm gonna step off the stage. We're gonna bring Ben up, and he is gonna start talking to you all about the amazing Nextflow language updates and showing you what they actually look like in reality, how to use them, and the rationale behind them. So, Ben, Hello. over to you. Alright. Hello, everybody. I'm gonna pull up my screen and we're gonna get right into it. So thank you, Phil, for that introduction into everything. A lot of goodies to be excited about in this, latest release of Nextflow as well as platform. I'm personally very excited about intelligent compute. But let's go back to the Nextflow language and talk a little bit about, where we are, where we're going. So as you may have seen from that timeline, we've been pumping out a lot of language features over past few releases. It may seem chaotic. The reason is because we sort of we were sort of traumatized by the DSL two migration. If those of you who have been around long enough to remember that shift from DSL one to DSL two, it was it was a very huge monolithic change. You sort of had to do everything all at once. And so this time around, we decided with the next iteration of the next full language, we wanted to make sure that we introduced features sort of one piece at a time, both to allow people to play with them and share feedback and make sure that we get them right, and also so that you can adopt them incrementally one piece at a time. You don't have to, refactor your entire pipeline just to take advantage of, let's say, strict syntax. And so those wave of new features have basically been wrapped up and concluded in Nextflow twenty six zero four. There's maybe a little bit of polishing around the edges we're gonna keep doing, for for the next version. Static typing, for instance, is still technically a preview feature, but for the most part, it's quite stable. And it sort of reached a point where we can sort of show, all of these features, combine them together, into one big change and and to show what it looks like. And so for those of you who've been if you've been waiting around to sort of let wait for the chaos to settle, now is a great time to sort of take, assess everything that that's that we've added and see how you could apply it all holistically. And then for those of you who have been, on the edge and adopting every new feature as soon as it comes out, I appreciate, I appreciate your edginess in that regard. And I apologize for any of the chaos that you may have had to deal with. So what we have now is essentially what would have been DSL three. We're not calling it that. We don't use that, sort of terminology anymore. We just we just we just talk about it in terms of Nextflow versions. Right? So there's Nextflow twenty five ten, twenty six zero four, twenty six ten. It's kinda kinda, like, kind of the same versions as Ubuntu if you use that. Right? Every six months, there's a new Nextflow version, and with that, that comes updates to the Nextflow language. But broadly speaking, we can think of the Nextflow language as there's sort of two modes now. There's what we know as DSL two, which I have here on the left. This is just the classic RNA seek pipeline sort of condensed into one file, written, in in the way that, you know, that everybody writes Nextflow code today. And then on the right, what we have is what I would call static typed Nextflow. This is sort of the next iteration. Now the left side is not going away. Right? Both both of these modes will remain valid, basically forever as as far as as far as at least as far as we're planning now. But the static typing is sort of an opportunity for you to step into this this new way of writing Nextflow code, which we'll get into a in in a moment. Phil mentioned the first step we took into all of this was adding the strict syntax parser, the linter, the language server. All of that was not, you know, like, new language features. It was really just, a better version of DSL two, sort of DSL two the way Palo intended. Right? So it provides you the Linter and the language server provide you with guardrails to sort of write Nextflow code in in in the way that's best practice and all of that, without, you know, getting into, say, like, groovy code patterns that don't necessarily work well with Nextflow or making a mistake. And then instead of, you know, getting a proper error message, it just runs incorrectly or something like that. So regardless of where you are, you can adopt the strict parser today. That's why we feel confident to sort of have it on by default in twenty six zero four. Whatever code you're writing, depending on how you've written your code, you may have to tweak a few things, you may have to fix or remove a couple of small things. But for the most part, this code on the left here is just regular Nextflow code. It's all compliant with the strict parser. We know that because there's no error messages. Right? And, you know, as soon as you, you know, start throwing in straight commas or something, it will catch that. And that's that's been around for a while now, so I think people are are broadly familiar. Once you've made that small step into being compliant with the strict parser, you get all the goodies of the linter and the language server and all of that. Right? So now you can start to if you're using the language server or the Versus code or whatever editor you want, you could start doing things like jumping to definition. You can being able to hover over processes and very quickly see, okay, what are the inputs and the outputs, things like that. And then once you've done that, the next, sort of phase, which you are free to step into now or or wait later and do it later or not do it at all. Right? It's, it's it's for your benefit. But this is where we get into this code on the right where we we're talking about now static typing. And so the and and, basically, all of these other language features, workflow outputs, typed processes, typed parameters, all that stuff, we can sort of wrap it into one thing, which we call static typing. And the the broad idea behind all of this is being able to model the structure of your data as it flows through every step of your pipeline. You may already be doing this outside of Nextflow with things like sample sheet, schemas, dataset schemas. You may have things like that that allow you to specify the structure of of datasets as they go through. Of course, you know, even individual files like fast queues and dams have a particular structure which which you could get into, if, say, you're writing a tool that processes that kind of data. And so we just want to bring that sort of visibility into Nextflow itself. Previously, it is very limited. Right? Starting from the bottom from the top here, you can see that we can define params and we can give them default values, but, we you can't really say what type they are. Right? Whether it's a number, a string, a file, you can sort of guess from the default value, but then you've got things like here which are null by default. So what are they? You have no idea just from looking at the code. Whereas when we step into the code on the right targeting twenty six zero four, now we can see those same params are defined. First of all, they're all defined in one place. So the params block replaces these sort of one off param assignments. And then each parameter has a type annotation next to it. And this pattern here of name colon type is pretty much what you'll see across the board, across the entire language, params, workflows, outputs, processes, all of it. And so now we can say, okay. This reads is a file, which the type that we use for that is called path. And, we don't have to specify we don't specify default value here because we can that just means that it's a required param. So the user always has to provide this. Whereas with this one, it's a path to the MultiQC config. It's a path. We actually just give it a string, and it just sort of knows that it treats that as a path. And so then moving further in, again, it's it's all the same code. We're still working with, and in fact let me let me check something here. Okay. So I think there's maybe I just had a one little typo here. I'm I got scared for a second. So moving further into the code, the the main there's sort of two things going on, on the right hand on the right hand side. One, we've switched to using the static type syntax. Right? So we're using all of this new syntax that allows you to specify the types. The other piece is we've refactored the pipeline from using tuples to using records. And this is also sort of a a separate piece. You can adopt static typing while still using tuples. But once you're using static typing, it sort of opens up at the again, another opportunity to, model your data more precisely. Tuples, are very practical. You know? You just stick two files together. You throw in a metal map, and you just they just you're they're able to travel together through the pipeline. Right? But when you're unpacking a tuple, it's difficult to keep track of, say, what's the meaning of the structure. Right? So I may know that I have right here, I've loaded in a tuple with an ID and two fast two files. But unless I'm looking at, you know, this map operator, which is unpacking the tuple in this way, it's it's hard to keep track of that. It the the type checker can look at that and say, okay. This is a tuple with a string and two paths, but what does that string mean? What are those two paths mean? And a record allows you to fill in that gap. And so what we've done here is we're loading the the CSV file sample sheet, and we're loading in each row as a record. And now, when I hover over these things, and it may or may not show up right here. I think this is because, yeah, it's not gonna show it right here, but we'll see this further on. The type checker actually keeps track of all of the, fields and, and their types. Okay. I think I see now the issue was that I needed to add in the actual field names here. So let's do that really quickly. This is probably just a classic copy paste error. Okay. And so we'll see, later on how the type checker, keeps track of all of that. And so let's scroll down to the RNA seek workflow. So this is where we see sort of the meat of the pipeline. Right? Because what what we have going on here is there's sort of a core workflow, which is the RNA seek analysis. And then all of this stuff on top is just a wrapper around that that sort of allows this pure channel in, channel out RNA seek workflow to interact with the real world. Right? It wraps it calls RNA seek workflow, but then it exposes params to the external world. So to you, the user, allow you to pass data in, and it sort of loads that into channels. And then it takes the output channels, and then it publishes them through the output block, which we'll we'll get back to that as well. And so the RNA Seq workflow comparing it side by side here, you can see, it's structurally the same. What we've done is we've added these type annotations to the inputs and outputs. So you can see before, we have this reach channel transcriptome, but, again, we don't know what types they are. And and when we're talking about workflows, it's it's not just about, like, strings versus numbers versus paths anymore. It's we're also thinking about data flow types. Right? Channels and values. Is is this a is this a channel, or or is it a data flow value? Because that's gonna affect you know, when you call a process, it's gonna affect whether the process gets called one time or once per sample, things like that. You can run into weird bugs. So this also allows you to distinguish between those two types instead of having comments or things like that. So we can see here we've passed in we've we're we've explicitly defined the inputs here as a channel of records, and then we've got the transcriptome just as a regular path file. Now digging into this here, not only have we documented it, we're also using a record type here. So it's no longer a channel of tuples. I could've if if we had stuck with tuples, then then it would have looked more something like this where we got tuple and then, string path path. Right? And and like I said, this is for one thing, it's very verbose to write, but also it doesn't communicate a lot of important information like, what is that string? What are those two paths? And so on top of migrating to static types, we've also migrated to records. And while you could just say channel of record here, that would be perfectly fine. That would just sort of that just means that, you know, the record can have whatever you want in it. When it comes to defining this workflow, you you ideally wanna have these ports, these inputs and outputs, as precisely defined as you can because that just gives, you know, who the user of whoever is using that workflow, be it a human or agent, by the way, gives them as much information, that that they need to to use it correctly. And so we've actually refined it by using this user defined record type called sample down here, where we define exactly which fields and types we expect there to be. Now there's you may have worked with typed languages before. You know, Python has sort of a type hint system, other languages like TypeScript or Java. Some of you may not have worked with types at all in which this is gonna be, hopefully, a more pleasant introduction, than than with other languages. My feeling is that many languages take types a little too far, and they make it really complicated. You get into things like class hierarchies and inheritance and all of this stuff, and it can sort of get in the way of just writing the main code. Right? So with static types and with records, we've tried to take a more lightweight approach, just give you the functionality that will help you on your journey of writing a well defined pipeline, without getting in your way. And so one of the things where that comes in here is that when we say that this input is a channel of sample records, this is not actually like a strict requirement. It's more of think of it more of a a minimum requirement. So if I pass in a channel of of records here, and let's say for whatever reason, that the CSV sample sheet here had additional fields. Maybe you had, like, a strandedness field or just other metadata related to, you know, the project that you're working on that you you wanna keep that metadata flowing through the pipeline as it goes. That's totally fine. The only thing that this input cares about is that the records coming in have at least these fields. There needs to be an ID, and there needs to be two FASTQ files. And if this record up here has extra stuff in it, that's totally fine. And so this gives you a nice bit of flexibility, especially because I know that it's common to have additional metadata flowing through. It also comes in handy later on, when you're passing channels between processes. I'll mention that in a moment. So maybe that's enough about that there. Aside from that, the workflow logic is more or less the same. You're just it's the same sort of channels passing through. The only difference is that now all of these processes, are using records where they were using tuples before. So you can see here this FastQC process. Previously, these would have been tuples. In fact, we can see that over here. So you can see FastQC is taking in this this tuple of the three items and spitting out another tuple. And on the right, we've simply refactored it to be records instead. And let's actually click over to those so you can see the difference. All it is is you you you have this record function instead of using tuple. And then on the input, you specify the types that you want. And then on the output, you just sort of create the record that you wanna build, and you can pull out, you know, files out of that the task generated and things like that. And so this is also nice because this you can you can use this whole process with records without actually using a custom record type. You could if you wanted to. I could come in here and say, sample colon sample. Right? But I I don't need to. I can just define the fields that the process needs. And here again, the same logic applies, that this this record coming in could have additional fields. It could even have additional files, but fast u c is only gonna load in these two fields from that record. It's not gonna load in the extra files. And so this is nice because it allows you to, where previously, the common pattern was to have all all of these, like, skinny tuples where you had, say, like, a tuple with the meta map and, like, one file type or maybe two file types. And then you would have to join these skinny tuples together whenever you needed some sort of combination to pass into a particular process. Well, now you can say, just keep all of the all of the information related to a particular sample. I'll say, I'll say all the per sample information, instead of having all these skinny tuples, you can collapse them all into one fat record is what I like to call them. And then you can just pass that fat record into a process, and each process will only load the fields that it needs. And so I actually believe that going to records will greatly simplify a lot of the workflow logic that people are writing. And then you can see here, we are joining the so we we run FastQC, we run Sam and Quant, and then we wanna join those two things. Again, we're joining them into this fat record instead of emitting each output separately. Notably, the join operator now, when you opt into static typing, it joins records by name instead of joining tuples by index. So before you would have you know, what this one would be doing would be you know, you would have this, this two tuple here, and then you would have the two tuple from quant, and then it would join on this ID field. Well, now we've actually got these explicit records. It actually feels a lot more like SQL. You can just say join it on this ID key, and and then what that produces and the type checker tracks all of this. So without me writing out all these type annotations, the type checker can just look at my code and see, okay, you've joined, this FastQC record with this quant record, and you see we've got ID FastQC, ID in quant. And so it it knows how join works, and it says, okay. So now I know that this record coming out of the join has all of these fields in it, ID fast u c quant. And then again on the output, when we emit that channel, we use an explicit record type here more so for documentation purposes so that, you know, if I, as a user, am hovering over the RNA Seq workflow over here where it's being called, I can quickly see, okay, these are the types that it's using. And then it also sort of serves as a sanity check. This samples channel coming in doesn't know anything about the aligned sample type, but the record contents do match up with the with the types with the requirements specified by aligned sample. So, again, this samples channel could have additional things in it, but the only thing that this emit cares about is that it has, at minimum, these fields. So it's sort of providing a RNA Seq is providing a guarantee to the caller that if you call me, I'm gonna give you this samples channel that has all these fields in it. And then we're also giving out the index, and you can see the index is a data flow value because we just called it on the one, on the one transcriptome file. And so then that all propagates back up to the entry workflow. We collect everything for MultiQC. We run that, and then we use the output block. Won't go too deep into output blocks since we've talked about it before, but it's, you know, similar to the emits. You just take the channels you want to publish. You publish them in the, well, publish section of your entry workflow up here. And then in the output block, you declare these outputs. It sort of mirrors the params block in the sense that you've got name colon type, and then there's additional things you can do in here. The main things being you can use the path directive to customize the directory structure. So your output directory can be organized however you want regardless of how the channels are structured. And then you can also create an index file, which is essentially like a sample sheet of your outputs, which can just be a nice sort of home base, you know, that points as a single entry point that points to all of your outputs. You could pass it to a downstream pipeline if you wanted to, for instance. So I think that's that's everything we wanna cover for static types and records. This is quite a lot, but hopefully, you can see that, you know, all of these language features come together to form this clear picture now where you've got everything is typed. And, if you really wanna go all the way, everything is using records, so it's a lot more structured. And the type checker is able to give you just a lot more helpful information when you get into that. Things like, you know, keeping track of all of your record structures. And in all of that, all you really had to do was define the inputs and outputs, you know, sort of the boundaries of your workflows and your processes. You wanna define your types. And this is this is this is I really love this part because this is actually giving you an opportunity to, like, think about the data that you're working with and how you wanna structure it, how you wanna name it. Right? This also becomes useful for agents, because it turns out agents can do, quite a lot better with with even if you just give them the type definitions. They can reason a lot better about your code. So with that, let's change gears a bit and talk a bit about modules. We've got a I wanna take a few minutes to talk about this, and then we can dig into questions because I'm sure people have a lot of juicy questions. So the other big thing we added in Nextflow twenty six zero four was a the module system. Now modules have existed in Nextflow for a long time as sort of an informal concept. Like, you know, it's it's a process in a script. Right? That's what a module is. N f core is also taking that and run with it, and they've you've got the whole n f core modules repository. What we wanted to do was provide a sort of a native way to manage modules. So this is both like searching for modules, installing them into a pipeline, writing your own modules, and publishing them to a to a central place. And then also, this is the real new thing, being able to run modules directly instead of having to run them through a pipeline. This opens up a whole lot of possibilities. Again, going back to agents, you could now have an agent that just runs individual modules one at a time and sort of do, like, exploratory analysis, trying out different modules, taking a module, running it different ways, trying to find the best rate best way to run it. And then at the end, maybe you encapsulate that exploration into a fixed pipeline, and then that sort of gives you that packaged reproducible analysis. But what I wanted to mainly talk about is how this affects the the pipeline, like, the project structure. And so the main entry point here is, if we go in here and I'm gonna have to move my mic over a little bit, get my keyboard. So if you go to we say Nextflow module. This gives us, the whole CLI, and you can see basically, the whole module use and development cycle, can live within this CLI. The first thing might be that, say, you want to search for something. So we maybe we want to search for different kinds of aligners. And so you can actually just search by keyword. What we've done is we've taken the n f core modules repository, and we've, like, auto synced it into the Nextflow registry. So the same website we're using for plugins, we're also now using for modules. And so, basically, all the NF core modules are you just are just here out of the box, and you can search for them by keyword, and then you can install them if you want. What I was gonna do was, let's I was gonna, I think, in install let's just install, say, like, the star aligner. So let me look for star and make sure I find the the right module for that. I think it's just in, of course, star align, but I could be I could be, hallucinating there. Let's see. Yeah. And, of course, star align. So let's say we found that, and I'm just gonna copy that, and then we can say Nextflow module. First of all, we can just look at some information about it. We can just say view, and that can get us some basic metadata. It can show us things like the kinds of inputs and outputs that are expected, also just like who wrote it. And then it even gives you a little usage example at the bottom here. So you'd have to fill in the actual files, but it it gives it sort of gets you as close as you can to if you wanted to run this model directly, you could do that. And so you could, for instance, take some data from n f core test datasets, whatever is appropriate for that module, and then you could run it directly. And that's and and then and then you can get into, you know, tuning parameters running at different ways, without having to go through all of the ceremony of, like, wrapping it in a pipeline. And so then let's we can install that, and it's going to install it, and it using a similar structure that n f core already uses. So if we see over here, we have this modules directory in this pipeline, and it just added the star module or the star line module in there. And it's got the script, the meta YAML. It's even got the tests. So if you got enough tests, you can you can run those. And then after that, it's just a matter of wiring up the models that you want into a pipeline. Right? We don't have enough time to get into the the, the the development workflow, but the way you would do that is, you could create a module using this structure. You could also just run next to module create, and it will, you know, bootstrap a module for you. And then you would, you know, build it out, test it, validate it, and then when you're ready, you could publish it to the registry. The way that works is that you can claim a module name space in the registry. So for example, n f core is a namespace in the registry that we just manage automatically. But you could say claim a namespace for your name or your organization. And then once you have that namespace, you can just push whatever modules you want into that namespace. We we sort of we back out after that. It's a very hands off approach. And so that's the way you can share modules for other people to use, and then they would install it through install and so on and so forth. What I did here was I, basically just asked Claude to rebuild the classic or sort of toy RNA seek pipeline, but using registry modules instead of the custom local modules that RNA seek uses. And so that's what it did. It actually did quite a good job, and it actually uses, this Nextflow modules CLI. So this all these commands are not just for you, but also for your agent if you're into that. This is this is something we sort of shadow dropped over the past week. Paolo and I have been working on a repo for Nextflow specific AI skills. It's called it's just called Nextflow IO agent skills on GitHub. You can install that as a cloud code plug in or just through whatever skills, installer that you use. And what I used here was this, the run workflow skill or create workflow skill. I basically told it, like, rebuild RNA seek with the registry modules, and it read everything. And then it it installed all the modules using the next one module CLI to install modules, to run them, to test them, to verify them. And then it just generated the wiring for that here, and so we've got all of that right here. And then it even hooked up I had it you know, I hooked it up to n f core test data. It handles all of, like, the software dependencies. So you can actually very quickly get to an executable pipeline, even if even if just as a first step, and then you can continue to iterate on it. And so, well, that's about everything I can think of. Phil, maybe you wanna come on and see if, we've got any other questions or anything I missed. We definitely have questions. We have a lot of questions. No. That was great. Thank you very much, Ben. That was really super helpful. I'm gonna try and try and kind of go through some of these q and a. If you've asked a question, we don't have time to get to it. Don't worry. We'll try and follow-up, with with them after the event. And, also, we had a question. I I should have said it in the introduction, but I forgot. Yep. The whole webinar is being recorded. You'll be able to watch it at your own leisure through this platform with the same kind of link that you got via email. It's there after the event finishes for you to watch as many times as you want. And, also, pretty soon, we'll edit it, trim off the start and the end, and we'll put it onto YouTube so it's just publicly available for everybody. Okay. On to q and a. Number one question with the most upvotes, seven upvotes, Francine asks. And miss I guess this came up when you're talking about typed input parameters, Ben. Does this replace the purpose of next flow schema dot JSON? No. Not exactly. I would say it supplements it. So a lot of the functionality that the schema gives you is now available natively. You can use the the in other words, the params block can't do a lot of the things that the schema was doing. Things like making sure you didn't mistype your parameter or making sure that the param you gave was the right type, things like that. But the especially if we think about, like, in a schema plug in, that that has a much richer it it does it does a lot of other kinds of validations beyond what the params block can do. So I would say, well, you probably wanna have both. And we're we're we're looking into ways of basically the the the ideal flow that we wanna do, is where you start by writing the params block. Right? Because that's just Nextflow code that's gonna be that's wired into the rest of the Nextflow pipeline. And then we we wanna provide a way to generate sort of a skeleton Nextflow schema from that. And then so that you wouldn't have to write out all the boilerplate again. And then all you would do is go into the schema, maybe add a few extra things, that the schema can do, and then, we sort of provide ways to maintain both. Yep. Exactly. And the two will be kept in sync automatically as well. That's the other beauty of that server. The JSON file will be there for easy access and for enrichment, but the core the kind of, the the ground truth will be in the next flow code always. I'm gonna hop around a little bit just to make sure we get the top voted questions. Going to record types, will we be able to import and export and use libraries of records instead of writing just inside the workflow. There's there's a few similar questions along this theme about standardization of record types and sharing of record types. Right. So this is something that is gonna be interesting for us to watch and see how the community evolves around this. But, to answer your basic question, you can import you can include record types just like processes and workflows. So you could, for example, have a file with all your record types in one place and then, you know, import them across different different places. With the pattern the pattern that I was following, like, my my personal what I would do personally was is probably define the record types next to the main workflow that they're being used. Right? So we had that RNA seek workflow that had the records for the takes and the emits. My first instinct is just to have the record types there with that workflow. And then if you're any code that's using that workflow might, you know, include those record types as well, but, you know, you can do it however you want. Same thing goes with the processes. I wrote the processes in a way that didn't require any record types just because I find that to be simpler. But if maybe you wanted to, keep things in sync better or something, you could use record types there as well. And in that case, maybe it makes more sense to, you know, just have a types library. Right? So, I I'm interested to see, you know, what the community does with that. We'll see. Also, just tying these questions together with the last one slightly, we also in and, of course, we have schema for, like, input CSVs. So this is something else we'd like to tackle in the future is kind of combining maybe records, record types with kind of the the schema for specific records, which I think is kind of an interesting thing. Right. Louis asks, can records be nested? Can you have a Russian doll set up of of records? Yes. You can have records within records, and you can have records within records within records. You can go as deep as you want. That's another thing I'm excited to see how far people take that. It's records all the way down. Got a a question here about static typing. Do you does the can static typing be applied within config files as well as Nextflow scripts? So there is some basic type checking within config files. I think right now that's living in the language server. So depending on if you're using the latest version of the language server, you may see that it shows warnings in your config if, say, you assign, like, a config option that's supposed to be a number. Like, if you assign a string to it, you might see a warning about that. I'm still refining that piece of it. But, yes, ideally, we'd like to have, you know, the same level of type checking and config as we do in scripts. Does any of this new language and syntax affect, NF test? And does does Nextflow twenty six zero four make it easier to write tests? Or, yeah, is it still still compatible with NF test? So I believe the community has managed to keep NF test up to date at least with strict syntax. I don't know how much people have played around with writing tests for, like, typed processes. My guess is that it probably works, or if it doesn't, there may just be some small tweaks that need to be made. I think I think I largely see NF test as being sort of orthogonal to the type syntax and things like that. So the type syntax sort of helps write clear code. NF tests really verifies things at the runtime level where you actually run the module on real data and you verify that it spits out the correct outputs. Right? So these two things definitely go together. And we're certainly looking at ways to, we definitely wanna make sure that NF test works with Nextflow as seamlessly as possible. Have some sort of because that's that's just as essential to, you know, the type checker passing. Right? Yeah. There's I'm gonna wrap up a couple more questions about record types, and then we'll we'll move on to some questions about modules. Alexander asks, does it mean that NF core will get rid of meta maps in the channels in the future? This has actually been, an ongoing discussion, might even call it a debate, within the at least among the NF core folks that have been playing with with static types and records. There's a couple different ways it could go. It may just stay as a meta map. I could also see it becoming a meta record. So instead of it just being a map with arbitrary keys, maybe it becomes a record so that you specify exactly which keys absolutely need to be there. Also, the meta map keys could just be sort of absorbed into the record itself. So you would have, like, all the fields that are normally in the meta map would be just in the record alongside all of the files. Right? That's also an option. There's different trade offs to all these things, especially when it comes to n f core because n f core is designed to be very, like, generic. So it needs to be able to be it just needs to be sort of flexible and usable by everybody, and so that puts constraints on it. Whereas if you're just writing your own processes in your own pipeline, you may be free to write it however you want. So if, say, you wanted to, like, let's say, explode the MetaMap into the record and just have it all sort of a flat everything flat in the record, you could do that. Whether NFCore ends up doing that, we'll just have to see. Yep. It ties into the reusability of modules across different pipelines and stuff. But, I mean, I remember talking about this just when we were first designing records and thinking about how this would apply to to MetaMap. So it's and some of what you spoke about with fat records and stuff, I remember kind of coming up when you're talking about MetaMaps a long time ago. Yeah. There's a bunch of questions about updating people's pipeline code. Obviously, it's a kind of a big cons big concern. People something everyone's sitting here thinking this looks great, but now I've gotta go and kind of update all my pipeline code. There's some people asking about kind of Claude's, Secure AI, some of these tooling, like, how well that works. And and, very quickly, as someone mentioned Secure MCP for this, I'll just note that the MCP is for kind of working with secure platforms. So it's great for running, pipelines and getting traces back and all this kind of information about the infrastructure of secure. By AI agents, not not really designed for working with Nextflow code so much as more secure at AI. Okay. Secure cosignist. So, Ben, Ben, just take us generally through some of the your recommendations for tooling for updating syntax. Yeah. So we're we're tackling this on multiple fronts. So you've got a few options. At the basic level, of course, we've documented everything. So all the new functionality is in the Nextflow docs. We've also written tutorials on how to migrate, taking that RNA seek pipeline as as an example and walking you through how you would migrate it. So you can certainly go and and read that and learn how to do it. The next level beyond that, the language server also has some basic, conversion commands. So there's a command called convert script to static types. And I found that works pretty well at least for individual processes. I think it can also generate the params block. So if you have a Nexo schema JSON, it can scaffold that params block for you. It can convert a process from, you know, the the legacy syntax to typed syntax. Still using tuples, though. It doesn't convert to records. So there's some basic functionality there. But, honestly, I think where everything is going, everybody everybody probably knows this is just some kind of AI skill. Right? I mean, ideally, we should just be able to give Claude, like, the Nextflow docs. We should be able to give Claude that migration tutorial and just have it do it all for you. Right? Now you might be able to do that today. I'm sure there's some there's some cowboys out there who will who will give that a try. And that may actually work just fine for simple pipelines. For bigger pipelines, I'd be a little more cautious, but this is what we're working on now is trying to develop some AI skills that provide that same information in the docs, but maybe in a more targeted way for your agent so that you could just have your agent, do these migrations. And and also maybe thinking about ways to keep it conversational, I guess. So, you know, the AI will migrate what it can, but if it reaches an ambiguity, you know, making sure that it surfaces that back to you and says, okay. How do you want to handle this part? Right? Migrating to strict syntax, it's not really a migration, but, you know, you don't really need to change what your pipeline does, to make it work with strict syntax. It's mostly just tweaking. Migrating from tuples to records, well, now that's that's actually opening up a lot of new questions. Right? Like, now you've gotta think about how you're modeling those record types, making sure that does it in the right way. It'll still it can still be functionally the same in terms of the files that it's spitting out, but your code might look quite different. So we we do wanna make sure that the AI can guide you through a migration like that with with care. I would also flag that many of the features we bought, sir, which are good for humans, are also good for agents. So for example, the Nextflow language server can be loaded by Claude and and Secure AI and and many other LLMs, which means that the the agents get real time feedback about syntax. And, yeah, we have we're still kind of building out more and more skills, especially into Secure AI and coscientists and also, as Ben mentioned, this new one registry for any agent type. So, we're we're leaning into it more and more and, very keen to hear feedback from the community. I'm gonna we're nearly out of time, so I'm gonna speed run a couple of questions here, just really fast because I wanted to get to them. Private modules, or Nextflow registry for for modules and plugins. Yeah. We've we hear you lots of people asking for this. Basically, the public registry will stay public and free for everyone forever. The API surface is public. So if you wanna build your own private registry, then that's also free forever. And we will have a commercial solution for that from Secura, which we're working on right now and hoping to to have it available pretty soon. If that's interesting, drop us a line at Secura, and we will give you early access as soon as it's ready. And and it's really good for us to hear about what you need. Custom module namespace, is a paid or free feature in the public registry? Totally free. Nothing paid for it at all. And, other IDs or is it just Versus code? There's a there's a if you go to the language server, GitHub repo has an issue about other IDs. People have been contributing different things. The language server itself is agnostic, and people have been tying it into other IDs and other editors. So go and check that out. There's a bunch of community work there, and and and I'm sure people will be happy to work together. Okay. I've gotta stop because I get in trouble when I, when I go over. But, exactly, we've got here the couple of the last slide with a couple of QR codes for you. I'd like to draw attention to two main things. I very briefly touched on secure intelligent compute right to the start, which we're just we're soon gonna be launching into private preview. If you're interested in that, scan that QR code or go to that website. Let us know your name and your contact details, and we'll be in touch, so that you get early access and get your hands on that as soon as it's ready. We're really, really excited about the the possibilities that this is gonna open up to people. And it's just a bit of a no brainer, really. It saves you loads of money, so why not? And then, the other thing is the Nextflow summit. Many of the things we've spoken about today were, mention we have where you can see it more about if you go on to YouTube and see the talks from the the Nextflow summit in Boston that we had, fairly recently. The Nextflow summit will also be back in October. We've got training and hackathon in person in Barcelona, and the summit with all the talks will be online and virtual using the same platform we're using now. We had huge numbers of people come last year. We had some fantastic talks, and I'm I'm I'm expecting great things again. So, registration is open. Hit that link and and go and sign up now. And there's also early bird discount now, I think, until the end of July. So definitely worth doing sooner rather than later. And then, finally, if you go over to the sidebar next to the q and a, you will see that there's a a poll that's just been opened, where you can basically tell us if you want any more information about any of the things we've spoken about today. It's a lot to talk about in a short short period of time, and we know it's it's tough to get through everything. So please do fill out that poll. Let us know what you think, and, and, yeah, be in touch. With that, thank you very much, everyone, for joining us today. I hope this has been interesting, and, hope to see you again in another webinar soon.