Transcript.
The whole session as spoken: Nathan Tarbert’s introduction and the Q&A, and my talk and live demo in between. Timestamps open the recording at that moment.
Generated from the recording with Whisper (large-v3-turbo). I corrected the names it misheard (mine, CopilotKit, AG-UI, CLiF) and left the rest as spoken, so expect the odd slip. The captions file is the same text, timed line by line.
Okay. Portugal. Brazil. Hey. I'm from India. Iowa, Poland. Love it. Very cool. We just went live on LinkedIn and Twitter. So, I want to welcome everyone. This is going to be really exciting. So, we have already a bunch of activity in the chat. Hey, Kenan, from Istanbul. Yeah. So, this is really cool. We'll give it maybe another minute or so as people find this event, potentially, like through Luma. But, yeah, a few hundred people, actually over a few hundred people registered. So, we're deeply, deeply appreciative and super excited. Hey, Eduardo from Mexico. Ruchik C.A. is at California.
Asha is in New York. Excellent. That's so cool. If you just joined, we're going to get started. But, please continue to say hi. Gabriel is in Portugal. And, we'll just kick it off. So, what I want to do, actually, that we've never done before. So, this is also pretty exciting. Is, I'm curious, what is your role at your company? So, I'm a developer relations engineer. Jerel is staff engineer. Yep. An engineer. We'll get to intros here in a second. Yeah. But, it helps us to know who's in the room with us. So, yeah. Very cool. Yeah. Very cool. And then, I'll just leave this poll up as we kick it off. So, we can start off with a quick agenda.
What we're going to cover today is functional copilots. Really, how to make them work. And so, you may have already been tasked with, you know, you've got basically the mandate from above. Or, maybe you are in the, I see some people are in leadership. So, maybe it was your mandate to, let's add copilots and workflows into our production applications. And so, it's not a new story, but we're going to talk about how to build them, how to make them reliable, and then some really key insights. We'll save a few key insights till the end, where we'll talk about how to get your agent to learn. And so, I, next slide please, Jerel. So, some quick intros.
I'm Nathan Tarbert. I'm the Senior Developer Relations Engineer here at CopilotKit. And I spend most of my time meeting with our users, teams, and customers. My goal is to be the person right in the middle between you building with CopilotKit and our engineering and product department. If you run into blockers, I'm happy to help. And then, I love bringing solutions back. So, my goal is to make you successful. I am joined by Jerel, who I will pass it off to. And I'll let you give your own intro, Jerel. Let people know what you've been working on. Sure. Hi, everyone. I'm Jerel. I'm a Staff Applied AI Engineer at CopilotKit. And I'm the guy responsible for OpenMuse, OpenDots, and OpenDuet. We've just seen some of our open source applications out there. I'm primarily the guy behind the community-maintained projects. At the same time, we also have OpenBot, which is a key part of life. We're going to talk about that in a bit later.
Basically, my job is to help show and apply AI in different use cases. That's why I'm super passionate about what we're going to talk about today, which is functional agents. And I'll get back to that again later on. Back to you, Nathan. Thanks, Jerel. Very cool. So, for people who are new here, maybe you aren't aware. We are CopilotKit. Can you go to the next slide, please, Jerel? Thank you. We're CopilotKit, and we are open source. We have two massive projects. Well, we have actually a lot of massive projects. But CopilotKit and AG-UI, I'll get to AG-UI in a second. But combined, we have 55,000 stars. And we are very passionate about open source. So, some quick context on CopilotKit. We are the framework for building agentic applications. So, a typical app has a front end, but your agent lives on the back end. We're the layer that connects the two and gives your agent a UI.
We also are a key, and we'll talk about this later, in taking that further to help your agent learn. So, the frameworks that we support are React, React Native, Angular, View for the web, but it goes beyond the web into Slack, into MS Teams, Discord, and WhatsApp. And you get a full suite of building blocks when you build with CopilotKit, where you get generative UI, human in the loop, front end tools. And we support all of the services. So, with CopilotKit, we'll talk about this later. But when it comes to business use cases, we are, you know, Fortune 500's use CopilotKit all the way down to hobbyists.
And we're also the company behind AG-UI. So, AG-UI is an open protocol underneath it all that CopilotKit is built on top of. And it's an open standard for how agents and front ends talk to each other. Essentially, the bridge between the agent and the user. And we're really that connective tissue. So, what gives the same agent, you know, work in a browser can work in Slack or React Native or OpenBot without rewriting your entire application or even rewriting your agent. So, that leads me to why we're all here today. Building functional Co-Pilots. And Jerel, I will pass it over to you. All right. Let's get this party started. Awesome. All right. Thanks, Nathan. So, by the end of this talk, we have a bunch of ground to cover. So, I'm going to like brisk walk through it. But essentially, I want everyone here to end with this after the next 50 minutes.
Which is basically, you'd be able to design and build functional agents with CopilotKit and learn how to improve their performance in production. Also, having like a mental framework of what are the things you need to care about and like consider when building functional agents. But before we dive into it, what are functional agents in the first place? Essentially, I define these as like copilots that are like specific to a particular job and industry. And they've gone through a recent uptake up and again. It started with coding with Cursor. You know, the whole idea with Cursor was, let's bring this LLM, this agent, inside of where the developers are working, inside of the IDE. It started as a port. Then they later on, they built their entire product from scratch, agent first, as like more users gathered into it. What we're seeing now in 2026, and this is why we're discussing this and we have like a lot of people like join in for us today, is that this is now bleeding across different functions or different verticals.
Harvey and Legora for legal support share healthcare for a bridge. And a quick shout out for like the folks at Kino AI, they're also like super interesting. They're bringing this into the side of like video editing. Essentially, it's a good example or like it's a good indicator of how like AI has progressed in the last couple of years. Like now, we've started with coding because it's like the easiest and we have like the frameworks ready. But once we start building for different functions, we see that there's a lot of things we need to bridge and like guide that weren't like necessarily hurdles in coding and in Cursors up and rise. So we're going to be talking about these and considerations of what are the factors you need to consider to like take it into production. And how do you actually ensure that you're setting yourself up for success in the long run? And by allowing your agent to learn like from user behavior and like improving that piece. Okay. So while we're all here, I saw like Nepal, around 60% of the people here are engineers and around 20% are leaders.
And like we have like a diverse like set of people here. Most of us here have been tasked or took the initiative of like, how do we bring agents into this particular role? Like agents for X. And I'm sure many of us here have been asking, hey, can we build an agent for our supply chain department? Or can we build an agent for the HR department? So essentially, this is going to be like a core crux of the conversation. And just a quick recap from how this has progressed in the last four years. And it's just four years, it seems like an eternity ago. We started from chat assistance where we chat GPT and we go with like, hey, can you write me a function to do this particular task? And then copy paste that into your IDE or like, or can you write me a contract clause for this particular piece that I'm trying to negotiate? Copy paste that into your doc. Now we're moving into functional copilots where we're bringing that into many different verticals. And it's just like a huge improvement. Like you see the huge improvement in that across industries.
So we see here, like coding cursor, where we did a quick MVP for Open Harvey, which is just essentially an open source legal agent. As well as like, you know, AI. Recently, it's like, instead of like using Premiere and integrating via a plugin, how about we rebuild the entire like video editing software and start AI first. So essentially, these are the two schools of thought. The legal space are more integration heavy. But other verticals like coding and video AI and I'm seeing more and more getting into this are more like, let's rebuild and make this AI first. So yeah, like plugin and rebuild. What's the right answer? I don't think there is one. You need to do both. Again, you need to meet, you need to meet and build for where your users are and also anticipate like where they will be. And we see a strong push and pull with this from like, okay, we need to integrate, we need to build a plugin for this particular app. And then later on, we find that more and more, we start building it inside of the app.
You can see that in Figma, you can see that in Claude, where there was initially a couple years ago, like they had strong integration and they were actively working together to like make those strong integrations. But now we see them like actually like build similar features. Figma now has more agentic features and Claude is now more design related features. So essentially when you're building like functional copilots, these are the things you need to like consider and build for. Like if you're building for a particular job, what are the tools that they are using? What are the patterns, UX patterns, UI patterns that they're used to in their day-to-day jobs? And then how do we make the trade-off decision of whether to build or integrate or integrate then build and then like the trade-offs across that. So before we get into this, I'm going to do a quick lay of the landscape. To build a functional agent, there's a lot of layers to it from choosing the right model, agent harness, data integration, context and knowledge, domain capabilities, products, and security and governance. It's a lot to cover for a 15-minute call.
And I don't think we'll be able to like scratch the surface of all of this. So a couple of things that are out of scope for this conversation, we won't be getting a deep dive into agent harness and models. But what I've done instead is I've created the open repository to help you facilitate and guide through this activity on your own in your day job. Like, hey, I want to build this functional copilot. And Nathan has shared the repository in chat. So you can just like work through with this. And it's essentially like a set of skills that will help you like think through this stack. And it's better to expect some changes around that. But I hope that will help you get through this in your actual day job. One thing to talk about while we're here is security and governance. We're going to split like the two, I'd say, key milestones when we're building functional agents, which is MVP and prototype and production grade and ready. I'm going to talk about the MVP and prototype. It's very important that you still do the MVP and prototype so that you know if it's actually worth it.
But for as you get into building for functional use cases and where business and actual data is involved, you realize that some of these are actually very important to secure. So there's like security and governance is becoming like more and more important as this goes across. And like there's going to be some trade-offs in the skill that in the repository created. There's also this thing you can talk about it like, hey, we're on MVP mode. And hey, we're on like production mode. And what are the things we need to consider when we're going to like deploy this now? Okay, so let's say you're tasked to build a legal tool for your legal team, like a legal copilot. You first start with who are the key players inside of the story? What are their responsibilities? What do they care about? And essentially do a map of like, who do we need to build for? And what are the layers of depth and care do they have for the workflow or for the function you're building for?
And we're going to go through this. Again, this is just an example. There are many of these, but you typically get this from talking to your stakeholders. Like, hey, who are the people we're interacting with? And one of the things that you need to do is to actually talk to them and map out what they care about. For those who have been in the space of like building startups and engineer for the past, for more than four years, like before AI, you'd know there are different modes of thinking for this that we still use in Apply today. Like product thinking, systems thinking when you're starting to building, and then design thinking, which is what we're actually going to be talking about across this. It's like principles borrowed from all three. And as we've seen how AI changed our roles, like it's up to us to actually do bits and pieces of all three. It used to be like a team of specialists, but now you have to like master all three modes of thinking when you're building functional copilots. Especially if you're like a one-man team or like when you're tasked to like build a lot, like you're tasked to build a lot and you don't really have the time and there's potential mounting up. You need to do like all three roles for that.
So just laying like the ground for this. Again, there are many ways and milestones to set this up. But essentially for the repository, for the sake of the stock, I've just like created like, Hey, first things first, we need to gather requirements and do research. And I've essentially asked AI to like, Hey, help me like codify how I'm doing things when I'm building applications. And I'm going to walk you through like how I've been building them in the stock as well. But essentially there are five steps, which is one research and user map. In my personal workflow, I do a lot of like mood boarding and research before we actually get into like building. So one of the things that have changed in the last like couple of years is in the past, there was this thing called like the prompt engineer. Like we have to be detailed with the prompts, you have to put the right context there, put like instruct the agent the right way. But what I found to building across building with AI almost every day for the last four years is that we're doing less and less of that.
And more and more of like working with your agent to build the context. So some people call this context engineering. I just call this research. Essentially what you're doing is research. You talk to people, you gather data, you map out existing workflows. You take UI, UX maps from existing tools and then analyze it and then figure out what to do with it for the task at hand. For the designers in the room, I saw like around 15% answered in the poll. Essentially, this is like the mood boarding and UX mapping and codified into a skill. I used to do this by hand back when I was a designer, like almost a couple of lifetimes ago. You take a lot of screenshots, map them in Figma, like give it a lot of comments and then write it into a PRD and then give it to the engineer to ship. But now it's like, okay, we need to get this done within the day. So what we're going to do instead is like, okay, what is the name of the tool that they're using? Is it up live? Can I download it? And then can I point my agent to it and actually map the workflow?
And I find that instead of doing this for a day to a week in a team context, like we're able to shrink the design sprint to a couple of hours with your agent. And that's the goal with the skill user map. So it's like roles, software jobs and flows. And I'm going to show like what that looks like in a bit. The next is context and access. When you're building functional workflows and you're moving away from MVP and prototype plan, you actually need to connect this to real systems. So this stage is mapping out where do we need to connect this to? And what does that actually look like? Is that a direct integration? Is that an API key? Is that an MCP? Is this something we're going to build in the app? Is this something that we're going to connect in Slack? So we're going to have to like map out what that looks like. Then lastly, spec and build. Once you map the context, it's a lot easier to build the right thing once you do the first two things right. And I find that if you spend most of your time researching and context building, you can just leave your agent on autopilot and go to sleep. When you wake up, it's already done and it's the right thing for you. And you can give it a couple of feedback.
But essentially, that's the key part. So these first three and a little bit of like the fourth piece, which is to deploy, is the MVP and like prototype plan, which is very important before you even get into production. Because you have to like understand and play with it and show it to stakeholders. Hey, is this what we want to like share? One key hack I'm doing like before securing, like secure and cover, because once you get into step five, it's going to be a huge, like the iteration cost is a lot slower. One thing you do there is like, hey, let's record all of the flows. And your A can do this, by the way, like record GIFs and flows for the user map. So that when I wake up the next day, I have like videos I can play and I can just review. Did it build the right thing according to my expectations? Or what did it get wrong within the shortest amount of time? Next is like, okay, so what does this look like? So step one research is, in the past, this would have taken like me and a UX researcher, and then we're going to map out the entire flow. We're going to go through and watch them do the entire thing.
So what emails do they send? Can we get a copy of those? Can we, what documents, tools do they use? And what, how do they use them? Do they do comments? And how do they manage versions? Like, what are the different things they're doing across a particular job to be done? So a key part there is, let's say you're building a legal copilot. For anyone to understand this a bit well, you just need to like refresh memory if you've ever had experience working with a lawyer. You'd find that they primarily interact with emails and documents and they do a lot of revisions, redlining revisions, especially when the other side has a lawyer as well. And then you have to like go into like back and forth with them. And then sometimes they maintain a spreadsheet to like keep the financial figures in mind. And then lastly, this is something that is less apparent, but it's more internalized. They went to school to understand the legal space, the rules, the policies, the things to do right. And the things that would be like a cluster fire of like, like things to do wrong. So you also need to like create those playbook.
And like, I think that's the hardest part. And there's many, there's many ways of doing this from interviews to buying datasets. And then we also are sharing a new, we also are doing this thing that I'm going to share earlier, which is learning from user behavior, which will help you build this up. We have to be PG dated. So like cluster fire. Okay. And then next you have to map out the time spent. This really helps when you want to then like, so this part is also very important, especially when you want to then justify the time spent and token spent building this thing. You have to quantify like, hey, how much did it used to cost? And how much value are we actually doing? This part is, I'd say, one of the most important part that people miss. It's that, okay, this entire process is a 10 day process that we can bring down to a single day. Like that's a lot of value, especially once you consider in like many different, many different, like, like people's time spent on this thing.
And you compute their salaries, it becomes like an exorbitant amount of time and across an organization. That's how you justify investments in functional copilots, which I'd say is a very important thing for your role as the majority of us here engineers. One of the things that I'd say my engineering friends struggle with, and I used to struggle with, is that how do we justify the value of what we're building? And this part is the thing you need to do well. It's like, okay, what did it used to cost in terms of time? Do some quick rubber hat, like, like, napkin calculation of like, what is that actually in financial value in terms of time, like exponential mandates, you know? And then that's when you figure out like, hey, this is something worth automating. Or, hey, this is just actually a 10 day thing. And the best we can do is bring it down to nine days. And it's like a one day difference that's not worth automating, because like the things are out of our control. So by mapping this out, you get a key, you're able to make informed decisions on what to automate and what to leave alone.
Okay, so for our leadership people here, and I'd say the senior engineers and designers, you must have heard this across time, like the job to be done for your merit. I'd say it's very important to understand this and like internalize this in today's age. It's not commonly talked about anymore as it used to be in like pre-AI stage of like software development lifecycle. But like understanding this is what would separate an MVP that gets to production and an MVP that dies in prototype land. So, but essentially what this is, is to understand job to be done, you have to understand that people don't buy your software, your application, your AI agent, like for its features. People hire your solution to solve a problem that they have. And that's the job to be done.
There are some key frameworks here, but essentially like what I want to keep in mind is when like this particular thing, when a situation happens, I want to be able to do this so that I can get this particular value. And again, since this is a mock situation or in the legal use case, like the value differs company to company. Okay. So the next step is to context and access. We, this is an increasingly more important part of building agents in 2026. Um, and I recently wrote a book about this in the last month or so, but essentially there are many layers of AI agents starting with like, hey, here's a chat bot inside of an app. Second is like, hey, let's bring that agent inside of our machine, like laptops, like Claude, like Hermes and OpenClaw and give it access to essentially that the entire laptops applications and whatnot and let it do stuff. And that like grow a lot of value and locked a lot of value for people in, um, their day-to-day like use cases.
Um, but recently we've seen the third, the third and fourth layers mature in the last couple of months where we now have brought that into the cloud. Now your agent is not just using your computer, which is also like a huge security concern, by the way. Um, they now have their own sandbox, um, like cloud computers that they manage and like allow them to be always on because they're now separated from your machine. No, no more walking around with your laptop half open and trying to connect with your hotspot running across like, uh, like place to place, you know. Um, now your agent actually has their own computer and that's, um, level three. Um, the fourth layer here, before we get into the fourth layer, we actually, a quick shout out to the OpenBot team. We, we have, this is essentially what we've been building with OpenBot, OpenMuse, Duet and DOTS. Um, essentially we're open sourcing this layer, like the third layer of the new agentic, um, like paradigms. And, um, we, we allow you to self deploy this, like, and essentially like make it more accessible for, for enterprises.
Um, the fourth piece here is actually something that got overtaken quite recently, but it's, uh, still a huge piece of how people work. It's now bringing this agent, like the, the first three layers into organizations in a multiplayer setting. Like we've seen this early this year with Claude Tag, where you can just add Claude in your organization and give it a task, go back and forth. Um, and that's like, like now like, uh, a multiplayer setting for like agents with their own computers. And it's like driving a lot of value across organizations. So when you're building your own functional use case, you have functional copilots. You need to consider what context does it need to have? Is it relevant for this agent to be accessible in Slack? Or is it something that we need to lock in and keep, um, the access close to, um, Gmail and, um, like, like the drive that you're storing the information on? So that's a key thing that we are, uh, that you have to consider here. Oh, by the way, if you are building agents to connect to Slack, we've built Channels SDK. And it allows you to like help, it allows you to like really like, uh, it's an open source library that we have that allows you to connect agents to Slack.
And it's like, uh, like really straightforward. Okay. So next is spec and build. If you do the first two stages right, this is the part that I'd say is the easiest now. And it used to be the hardest, like, like, uh, before AI, this one, this spec and build took weeks and months to actually nail. But now like the hardest thing to do is the first two pieces research and context access. Um, spec and building is something that you can do overnight and almost asynchronously. Like once you gathered the right research and the right context, um, and you put it in a place that's accessible to your agent and read me in a doc or in a HTML file or in transcripts, it's, it's almost instant. It's almost, um, autopilot at that point. But if you skip research and context and go straight to spec and build, you're going to pay the price of iterating more and more and spending a lot more tokens just because like, Hey, um, like it gets quite close to what you want, but it's not really there.
And then you present it to your stakeholders and then your stakeholders say like, and that's kind of wrong. I mean, you're like, you're eroding the trust bank and that, that's something I don't suggest doing, especially in the corporate settings that, um, when you present something and they, it's not hitting, like, it's like, do that enough times. And everyone's trust and like your ability to build will also like, like, uh, diminish. So you have to be mindful. And the way to build this right is by doing the first two stages, because it's now like a lot easier to spec and build out agents, um, and energetic workflows. So in CopilotKit, one of the things that we do is, uh, we bridge your applications context to your agents. And this allows you to do a lot of like cool things. Like when you're writing, um, like let's say in the legal use case, when you're writing documents, when you're writing, uh, uh, the thing that we've mapped earlier from the research and build, uh, just going back a sec. Um, when you're, what we've identified here is one, they need access to email. Two, they need documents. Um, they need like a document reader and a document editor. Um, they need a way to manage revisions.
They need a way to compute. And then there, there, there has to be a way to like set up rules that the agent might not get, um, in the first round. So when we interpret that in this, um, in the spec and build is, is we have to consider like, okay, um, are we going to build a plugin to Word, um, and Google Doc? Or are we going to rebuild Word and Google Doc from scratch within the open source library? That's a key decision you have to make. Um, in the earlier years of AI, I'd say they were just, just a couple of years ago, honestly, but it feels so long. Um, you, people really lean in towards building a plugin, like, okay, let's integrate existing tools. Um, but in 2026, like, I'd say that's, and it's a hot take, that's the wrong thing to do, especially as organizations and these giant corporations are waking up and locking in their application. You know, like one thing that separates Metamuse from the other applications is it has access to Instagram and Facebook that Grokbot just doesn't have.
Grokbot has access to X and then, um, all of these different, like differentiators, um, that they lock their agents in and out of is what's going to make or break like a functional use case. Yes, you still need to like figure out how to connect your agent to an MCP. It's still an important thing, but if that's your entire strategy that puts you to risk, um, especially when you are, um, like, especially when these giant, like, applications start to lock down and build their own agents. It's a hot take, but I see it coming. So what, what to do, what to do now? Yeah. So, um, at, let's say you are like Harvey or like Kino AI, which is like the video editing software, instead of like integrating with Premiere Pro or like Microsoft Word, build the set of features that you release your seed. Um, uh, and you scope it, you map it out with your agent so that it's easier. There are a bunch of open source libraries that you can bet to like, like bridge the gap. And if you build that inside your application, that gives you a lot more context. And, um, CopilotKit allows you to connect your app state or app features to your agent, um, and manipulate it directly.
So instead of like integrating to a software built for humans, you're building a software ground up that is AI first, that your agent natively understands and interacts with. Um, like another like function that we have here is like, like use for the tool. So it allows you to like manipulate like the state of the application as well, run a task, graph a red light, fill the cell. Um, now when we are like on the agent experience, one thing that, uh, we are going to briefly discuss is like generative UI and components like use components. Like how do we make the experience not just blocky words of text with, um, and there's some generative UI components out there that in and of itself disturbs a talk. And if you're interested, shoot us a message. Like that's, that might be something we're going to talk about. Um, but generative UI is an entire spectrum of things that would allow you to improve your agent experience. Um, and then lastly, um, human in the loop, which is a key part in designing functional copilots. It's understanding when to bring something up to the human and require the plan. Um, require, uh, require human intervention for the plan.
So that's a key part of this as well. Um, okay. So, um, from the task to the approval gate, uh, like essentially here, this is like the human in the loop component. Uh, let me see if I can screen share here. Uh, let me just, oh, I accidentally turned off the agent that's running. Let me just run that up and I will circle back to this. Uh, okay. Um, I'm going to circle back to the demo, but, uh, one thing here is like when, one key part when building functional agents is the trust between the, the professional and your agent. It's not often talked about, but one key part in building a good agent experience is like having the right level of show and hide and inspect. Uh, what do you mean by that? It's like, essentially it's just an exercise of trust building.
If you were a professional lawyer and you abstract everything away into this dropdown section, like, like done, um, and recorded reason. I like, and you're bombarded with like, uh, like a done, like contract that's already done as your first time interacting with AI. It's really hard to trust that because you can't see the process, um, of like making it like, like it's, it's the old, like, show your solution, um, paradigm in, in school where we, where we, uh, like to understand how to arrive at a certain point. And it's like, like to trust the end goal, you want to be able to like monitor how it was made and at the same time, like track that. So this is also an important part in the secure and governance part later on. It's like, okay, logs and audits, um, and rollbacks is a key part. Um, and then, uh, let me see if we got it up now. Okay. I got it up and running. Okay.
Let's see if I can quickly swap my screen here just for a sec. Okay. Okay. So just to showcase what that looks like, uh, let's say, so this is Open Harvey. This is part of the repository we shared. Um, and currently it's using, um, a mock so that it's like a lot easier to like, like show and render. We don't have to wait. Um, but a mock is actually like mocking like the response of an LLM that, uh, it would work in reality. So let's say let's review, um, that like an MSA, which is something we've identified that the users need. Um, showing the review plan here, um, for most users is like, uh, you can just like go ahead and build that. But for the professionals you're building for, for the legal professionals you're building for, it's an important thing to show like, okay, um, are we aligned with how the agent is going to tackle this task? And even if it's just a brief run through over what's happening, that allows, um, that builds trust between the professional and the agent. Um, and let's say we approve the plan here.
Uh, we see, again, this is fast because we are, uh, running a recorded demo on this, on, on the back of the a mock. But essentially that would take a longer time and then, um, it would show that, okay, what, what's processing and how it's, um, like, like what's processing and how it's manipulating like the data here. And then one key part, uh, cause again, this is a quick build for this demo is if you're building to replace, like, it's a lot, it's super useful to pattern match what they're already used to. Like, and it's easier to do with AI. Let's say you, like, agents work a lot in Microsoft Word or in Google Docs. Um, ask your AI to do the slash user map and point it to those applications. Let's do a user map of Google Docs. Let's do a user map of Microsoft Word so that you can, you can, uh, yes, uh, so this demo, you can run this locally. It's in the repository, um, under slash starters, um, and, uh, just to expect some updates on it. But at the same time, this is, uh, this is something, this is a starter you can actively, like, like, modify and build with.
Um, and once you do the user map, your agent will now understand what you mean by, like, click copy. Microsoft Word. And before you get into that, you need to see it to actually map out. I like looking and asking my agent to actually do a mood board of the entire application. So that I know that it understands and it gathers it into context of the different screenshots or, um, like, the particular news flow. And then it makes it easier to build once you have that. Because in the past, you'd have to describe that in words and in PRDs and in user stories. I'd say it's super hard to even do that, even for an agent. It's a lot easier to show and ask your agent to like, hey, let's build context together. Download it inside of your computer and then, like, go through the entire flow, map out. It's a lot better if you can do this with your, um, with your professional. But at the same time, that's a, that's a key thing. Or just, like, like, do it yourself and, like, giving context to your agent. Anyway, that's, that's an example of, like, building for functional agents with, um, CopilotKit. And, um, a quick behind the scenes here is CopilotKit runtime connects you to threads and memory.
Uh, we still need to find a way to, like, deploy this in the cloud, which is, you can do that in Daytona, E2B, AWS, or QuickCloud. Or do that in self-hosted local way. That's something we also support. Connect that with Slack and Teams. And then your application is, like, essentially, um, a web app that, like, runs with these. Um, I'm risking to swear that's short on time, but there's a key part I want to show on, like, how do you actually do secure and govern. Um, there's a lot of things to deploy. There's many different flavors in doing this. I did my best of, like, trying to consolidate how I think about deployment inside of a skill. So it's inside of the repository as well. So you can, uh, like, set this up in, set this up in, uh, uh, like, work with your agent to deploy this that meets your needs. You can just, like, work with it in the deploy skill. Um, lastly is one key part that I'd say is also something you need to consider in the MVP. Especially when you're building for business use case is the, the guardrails, um, building the audit rail. It's a lot easier to build now.
Um, I've also, like, consolidated a checklist in, inside of, like, the secure and govern, um, like, part of the repository. So that you can just ask your agent to, like, hey, um, are we, like, secured and governed? What are the things we need to care about? Um, because often it's easy to miss these things, but this is what makes or breaks the production application, um, makes or breaks a prototype breaking into a production application. You have to, like, consider these things. Named approvers is very important, especially for business use cases. Like, who approved this thing? Um, like, that's important for many different reasons, but it's also just, like, to ask questions about, like, hey, um, what happened here whenever something went wrong? Or, like, hey, like, like, uh, like, understanding who's responsible for the thing, especially as you build, like, human to agent, like, like, essentially bridges with, um, these functional copilots. Um, and then lastly, again, this is, like, a preview of what it looks like, like, the scale. It will give you, like, an HTML file with things to check for your application. And it gives you, like, a prototype or a production, like, like, barrier, like, this is safe for our laptop demo, but you know what?
You need to check these, these boxes first before you get into, like, production, or else you might be putting your company at risk. Okay. So, one last thing. Okay. Once, let's say you've gotten first a one to five done and deployed, and your stakeholders love it, you deployed it internally, people are using it. Like, but after a while, you see your agent breaks down. And the reason for, like, your agent fails and breaks down for a couple of, like, key moments inside of the workflow. And it becomes complex to build data sets or complex to identify and improve. This is why we actually spent a lot of time, like, thinking about this and, like, helping our, like, customers, like, like, build on this. It's, like, because one of the key problems here is, like, it's really hard to build a data set for fine tuning or describing the problem in the first place. Because it's really hard to build context around it. Like, it's not a linear trajectory, a yes or no you can build a neural network with. Like, it's a lot, a series of steps that are unreliable.
Like, the steps changes each time to accomplish the task. Even if the task is the same, like, the steps to get to the task is very different across time. So, it's really hard to, like, build and improve on. Like, to build a data set that you can, like, use your agent to improve on. So, this is why, let's say, you've gotten live and your agent isn't working as well as you are. How do you improve it? This is a problem we care about. And this is why we've also launched CLiF. This is still in early access. We're working with design partners to get this, like, to a broader audience. So, essentially, if you're interested, like, shoot us a message. Like, later on, we have, like, a slide for that. But to explain what this is, Nathan talked about AG-UI, which is essentially connecting your agent and your user. Like, your agent and your users inside of the application. We're essentially bridging that gap. And by doing so, we have contacts on the agent trajectory and the user trajectory. So, by doing this, we get contacts for both. And it's not going to our, like, service. By the way, we don't have that.
It's going directly to your service or your cloud. And you can, like, self-host this so that everything stays local in your business sense. But by doing this, you get what we call a full stack trajectory. So, agent trajectory, which is the series of steps to do a particular task. And the user trajectory is, like, what does that look like? Like, the series of steps to do a particular task for the full stack trajectory. And that is a, like, this dataset is becoming more and more important to own for your organization. Again, this is something that you're going to self-host and deploy. Like, it's more and more important because, like, your professionals and your agents now start to work together and mesh between tasks and their day-to-day jobs. And if you, like, have a better understanding of this, it's a lot easier to improve on. So, the key example here is that, let's say, an agent trajectory tried, and that this is just, like, the agent trajectory, like, the evals you get from LangSmith and, like, building with LangSmith and, like, the other eval providers there.
But they don't really have context on the user actions, which is, like, the highlighted part here is, like, here's, like, the agent actions, but what did the user actually click and work about? So, what this looks like is, let's say, in a financial use case, your user clicks across, like, the different sections of the app. Essentially, that is what we keep track of in the full stack trajectory, user, agent, user, you can see this here. And that's something you can, like, gather in and build, like, a dataset on. So, AG-UI is connected across majority of the frameworks out there, so you don't need to migrate whatever, like, agent framework you're using. And it's also connected to majority of the channels from web apps, Slack, mobile, and Teams. And then, by doing this, you get to label what's learning from, like, there's also, like, a way to thumbs up and thumbs down these, like, tune their repository, clean it up. Then, once you get to that, you get, once you've done that, you have many different ways to improve your agent that's just not accessible for you because, like, before this, because it's really hard to get these contacts.
In the past, when I was still a designer, we'd record screens from people using our applications. We still do this, like, in a production application with, like, different, like, telemetry trackers. But it's really hard to bring that together with your agent. And that's something uniquely possible with, like, what we've done with, like, CopilotKit, and AG-UI. So you can, like, just start building these product insights, like, and can turn them into skills, eval candidates, and can use this dataset as you scale into the organization to fine-tuning URL. So, okay, so that's the learning piece. But what are the, like, now, let's say, it's gotten approved from prototype and internal use case. Now we're going to productize this across the entire company. Now you're opening up a whole can of worms that, like, look daunting to tackle. But once you know what these are, it's a lot easier to, like, work around them. And there's a lot of, like, providers you can use or custom build. One thing that we can, like, learn from is, like, OpenBot, which is something that we've built for enterprise in mind.
So we have key guardrails there, like, and audit logs for the genetic applications, which is an important piece, especially that you need to keep track of all of these user and agent actions across the use case. So it's not just for improving the application. Some use cases and some functions require this for compliance and regulatory purposes. So you need to, like, set these up in a way that supports that. Again, all of our, like, software here is mostly open source. So you can, like, self-host them. If you have any, like, if you need any help, like, deploying that, you can talk to our team in setting that up. So everything you saw today is ready to deploy in the repository that Nathan has shared. If you weren't able to gather the link, it's just on the, this functional copilot's app here. And then one, let's see a Paul here. It's like, have you ever built a functional copilot? And, like, love to see if, like, where people are out of that. And then last thing, if you are going to take anything away from this song, it's that when you're building a functional copilot, this means that you have to build an environment where your users and agents can work on tasks together.
And across all different use cases, across all different functions, this remains to be true. Because, like, our workflows have now just merged and some industries have adapted faster than others. So you have to figure out, like, what are the functions that we get into this. And then lastly, if you build it with CopilotKit, you get to learn from both your user actions and your agent traces. And that just allows you to, like, get a lot of, tap into a data source that is mostly unavailable otherwise. So that's a key important piece here. Okay. That's pretty much it. Nathan? Awesome. Super educational, Jerel. A lot of people, I'm sure, are also interested in the slides. Will those be available? Yes. Very nice. Yeah. Yeah, we can definitely share. We will be sharing the recording and we'll also share the slides. And also the QR code is for CLiF.
So it's everything that Jerel talked about in getting your agent to learn. And it's using AG-UI to do that under the hood between every user interaction in your application. And then teaching your agent or guiding your agent or modifying your agent. The goal is to improve your product. So everything is self-hostable and it's still in. We launched it a few weeks ago, but it's still in early access. So we would love to hear from people that use it and get your feedback on it. You can talk to our team about it and they will get you all set up. So, amazing. We are, I know we had questions. So we have a couple minutes left. There was, let's see, let me scroll up.
One question we had Jerel from Faramars asked, can I push a signal fan out into Slack channel or DM and take a reply back into the same agent thread? And when does agent tag leave preview? Yes. So that's something you can do right now with the open source software. And if you need help setting that up, you can always talk to our engineers and we can walk you through how to self-host this. But I personally have done that with OpenBot. Like, hey, here's our open source version of Grockbot. And then if I want to carry the conversation in Slack and bring that back into OpenBot and then have a way to connect the two together, it's something you can do with channels, SDK, Slack integration. And then there's still some pieces in our manual that you have to figure out in terms of deployment. But having the SDK makes it a lot easier to match and integrate across what you're building. It doesn't have to be OpenBot.
Like any application you build towards, you can just integrate with our channels, SDK. And it will give you access to not just Slack, but you can do that in iMessage teams and whatnot. And in terms of bidirectionalness, you can have that conversation in Slack and then continue that in OpenBot and then request for contacts in Slack. And that's something that you can do or ask your agent in OpenBot. Like, hey, send this message out in Slack and it's going to show that. Yeah. Yeah, totally. And then so other people were asking throughout the presentation. By the way, Jerel, you got a lot of claps. Oh, thank you. Mega props. Yeah. Yeah, it was really good. And we are at time. But I just shared our Discord link because if there's any questions that didn't get covered, please join our Discord. We're happy to answer. I'm hanging out there. Jerel is hanging out there. Most of our engineers are there as well.
Well, how many times can I say as well? There's thousands of AI builders or people building agent applications that are great to network with. And so the community as a whole is amazing. And so you're welcome, Will. We appreciate it. So thank you, everyone. We're sorry we didn't get to every single question, but please, yeah, just join our Discord and we can continue the conversation there. Jerel, I want to thank you again for, it's a lot of work putting, we've been planning this for a while, so I really appreciate it. And thanks everyone for coming. Thanks for having me. We'll see you all in Discord or on X or on LinkedIn. Reach out to us. Thank you. Cheers. Bye-bye.
