Agent Frameworks Considered Harmful — Rémi Louf, .txt

Two weeks away to find out what agents can do

00:12Hi everyone. So originally I thought I was going to give a very technical talk but I saw was in the leadership track which I'm not sure what it means but I was like okay I'm going to do half high level and half technical. So it's more a story about what I you know what I did in January because in around December um agents kind of became really good uh you know there was a step function something happened uh I think it was opus 4.6 six and that's when I realized and I work in AI where I was like okay this thing is

00:45really happening and so I took two weeks out I took two weeks away so I'm the CEO of text uh you know which is 15 people company I just told my CTO I was like okay I'm just going to go away for two weeks and I'm just going to dive in this thing and try to understand what we can get out of it and how good it is and so the story is you know it is the story of me scratching my own itch for two weeks and trying to figure out how we can use actually use agents and what are good primitives to build agents and whether you know it already exists.

01:15Um this was a really clickbait title but actually it turns out to be a good title even for this talk. Um so what you can see here on the left the castle is my office. Uh that's true. I do rent an office in Castle and the small thing with an arrow uh that you can see is like this robot mower which kind of works unattended all day every day.

01:40Uh it just does its stuff in the background without anyone having to use a remote control or think about it or anything. And I kind of wanted the same thing for my morning because my mornings are always the same thing the

The morning he wanted, and the robot mower

01:57first couple hours. It's browse market news review of like linear could be Jira my CRM and also I like to walk for an about an hour in the morning and then the next hour is spent trying to process the really long voice note that you know was recorded while walking and you know all I wanted was my morning briefing with my coffee and that's kind of what we've been told for a couple of years like what the future would be.

02:21Um, but then when you really start working with it, even if you're not coding, all you get is a TUI today. Uh, so it's amazing. You can code. You can actually, you know, I started doing things that were not coding in it. They're great for this.

02:37Uh, agents are great for this, but it's kind of the equivalent of having a robot like a tractor mower that you still have to stay on even if it's driving by itself, right? It's kind of

A terminal you still have to sit

02:53very frustrating because you have to it can do many things but you still have to be on and so of course the labs didn't stop there and they came up with apps uh which I call basically SSH with vibes that's great but in this situation when that came up I was like wow that's awesome I don't have to use like a ter like SSH on my phone anymore codeex is great however I noticed I just started you know I was on my walk and I was just instructing the agent to do things while I was walking and so I wasn't thinking you know very clearly anymore.

03:21I just started running agents on my phone during my morning walk. And this is not great because this is the equivalent of this is you're kind of midway uh you know it's not the tractor that you have to stay on. It can actually do something without you being right next to it.

03:37But you still have this remote control that you know you kind of have to change the trajectory every now and then. Uh that's useful. It's kind of absurd uh when you think about it and actually when you look at people like on their phone all the time just doing this, it's kind of absurd and it's clearly transitional.

03:56Like surely we're not it's not it's not going to stop there. Um and so I did a very dumb thing as a co which is I started coding uh don't tell my board and I started to build the dumbest thing that could possibly work. And of course it became a really a crazy rabbit hole.

04:16Uh the repo is there if you want to take a look at it. The code is not amazing. Uh but it works. Um so the first thing is that you know I started using frameworks. Uh I mean there are great frameworks. I'm not going to name any frameworks because they're all good in their own way and they all have flows in

Why the prompt ended up in markdown, not code

04:35their own way which is fine. Uh but I spent all my time actually editing the prompt within the code and I was like this is actually not very useful. So I'm like everyone here, I hate YAML u like the next guy, but I still found that this was actually a lot easier to start implementing agents without code.

04:49Uh you can version it, you can diff it, uh you can review in the PR. Um but it's just and it's just so easy. You can just, you know, write your file, you drop it in a folder, and then it just magically appears once you have the runtime and it just magically works.

05:04Um, and you know, then I needed like my market watch to run every morning while I'm, you know, while I'm walking in the fields. And for that we have things that, you know, uh, have been around for a while, which is chron jobs. uh and schedules specify you know when the agents need to be run and also we'll see it's very important later uh they publish uh they publish events and you know markdown and chron obviously you know it's much more complicated than under the hood but the interface is this you don't write code

05:47and that's the whole product so far and honestly just mostly worked at this point I'll I'll come back on mostly uh later And so this is actually a real picture of my one of my morning walks. And so what I do is I record voice notes while I'm walking.

05:59Uh but chron you know chron jobs I mean people would use chron jobs for this because that's what's available in codeex today but they're not ideal because they cover when but this is just one point in time. It

Cron gets you when, events get you because

06:15doesn't cover because this happened and you know things that happened in our system like automatically when you drop the voice note now in the system it will emit an event and an agent will react to that event and it's the same thing when you have a new email a new entry in the CRM I mean anything a new PR that's open a new PR that's merged etc just reacts to events it's not just a chrome job and that and that means that you know agents of the Voice note processor is just you know not a chron job but here you have accepts and returns.

06:47So it just declare what it accepts and what it returns as an event and here it accepts a voice note transcribes it turn it into durable notes on the right and it emits a new event and for that it uses structured outputs. We'll come back to this.

07:02And you know, now we finally have the future we're promised because that voice note agent emits voice note processed. And then I have my daily brief agent that actually will take the output of the chrome job. We'll take the output of the voice note agents and we create my daily brief which is posted as a slack message.

07:24So the slack message.post post event is actually uh is actually like um a process actually subscribes to this and emits uh and sends a slack message to me. It's actually this is a real this is a real thing. It's working. I can show you after on my phone and you know there are frameworks that are going to sell you the fact that you need graphs for this in code.

07:52Uh you do not need graph uh in this case. All you need is events. You have no edges to maintain. Agents sub simply subscribe to events. Anyone can come in and edit this. You don't need to. Yeah, you don't need to know how to code. You just need to know what events exist in the system.

08:02Fun and fan out are free. No code and it's just drop a file and the topology emerges whatever the log says happened. And you know then of course I I tried to run it. So the first version took about I mean you know I cheated I cheated I used uh I used codeex and it took about like a day to write like the first thing uh out of my week but of course I tried it and it broke uh so these are real examples actually the dates know but it's real examples it's like the first day delay brief was posted to Slack twice um on Wednesday one of my voice

What broke in week one

08:50notes completely vanished and then you know towards the end of the week. I I kind of like played with the prompts all week and the market brief was garbage, but I didn't version uh I didn't version my changes and I couldn't remember actually what I changed in the prompt that made the thing completely useless.

09:10Now, if there are distributed or existed engineers in the room, you probably know this shopping list already. uh there is nothing new under the sun and you know each failure so each of these failure modes that you found actually led to building one piece of what turned out to be a runtime.

09:32So the lost note actually turned into a log. I just wanted everything to be saved forever so that I

The log: nothing lost, everything causally linked

09:40could go back to it and look into uh into what happened. the duplicates. It was because I was not following, you know, it did several attempts and I was not following them. I didn't have a proper queue. I wasn't counting the attempts, etc., etc.

09:48And then probably the most interesting part is the last prompt. I got into a really deep rabbit hole in there and I just ended up uh building a content like a content addressed system for this uh content addressed system. You can think of git uh you can think of nyx and any other build system.

10:11And you know that was and I didn't do this because I wanted to design a runtime. I mean by that point I still just wanted my agents to work and I also like the distraction. Uh and I just paid off debt as it appeared like errors as they appeared.

10:31I hope my board won't see this talk. Uh so the log the log is the systems memory. Uh nothing is lost and everything is observed. uh you can you know you only have one append only events table on the left it's a real common line uh in like command za events and you get all the events they are causally linked as well like you know which event triggered which event which happens to be super useful when you're debugging and you know even with three four agents you start having like major debugging headaches so that was super super helpful Um and everything is querable which again for debugging.

11:16The second thing is you know okay we have a log so we can trace back things etc. But it's still really hard to know what went into the like what went to the model what prompt was sent to the model again because what you see when you're using codecs is kind of a lie.

11:33like you kind of have like a live chat session with the model and so you tend to think that oh that's what the model saw and you know that's exactly so I can understand what happened the truth is that's not exactly what the model saw um there are many reasons for that one is I mean compaction obviously is a big part is a big thing but also you know there are just quirks also you know openai doesn't share or enthropic for that matter don't share the thinking with you the thinking traces so you have no idea I mean kind of have an idea of what went in but not completely either.

11:58And so you need something different. Uh you need something different and that was the big rabbit hole which is trying to find a

Content addressing the prompt, and free diffs and replays

12:15way or build a system where you can trace back to what the model saw internally. And so what I did was basically built I mean nothing new. This is basically how build system works. Uh so you have different parts for a prompt. You have your system prompt.

12:28You have a description of your first skill of a second skill. Then you have the description of your tools. You have your user message which is the question to the model. Each one of those is stored and addressed and you know stored somewhere as a identifier which is a hash.

12:43And so when we build a prompt instead of building a piece of I mean before rendering the text we actually represent the prompt as a list of these u of these hashes. And so what that means is that down the line when I have a model answer, which by the way is also stored in the same way, we can trace back to the prompt very easily.

13:07And then from that prompt, we can know exactly what went into the model's context, which actually matters a lot. I mean, it matters a lot for debugging, but it also matters. I mean, it makes compaction a lot easier. You're just manipulating a graph, right?

13:16You're not manipulating strings. It's just a lot easier. And it makes KV cache management a lot easier as well indirectly. And but I think that when you know I guess probably the main advantage that's when you use that scale is really auditability.

13:38It's like you can know exactly what happened with that agent and why it returned what it returned. And so you know I'm just going to go pretty pretty quickly over this. Uh what you get once you have this graph is you get diffs like you can say okay what changed between these two runs like which com which components changed was it just my message did I like give the model a different skill did I give it a different tool so you can just yeah you can just run this function and it

14:11will show you you know the difference between uh the runs so here you have you know three components that were identical there's one which is you know the user message changed And then you had all these other messages that were actually you know that were continuing.

14:20It's continuation of a single session. Uh then you have another thing for free which is replace. Uh replays turned out to be really useful for me because after a while I mean when I saw the cost ramp up like the thing when you have observability is that you do realize that cost increase very quickly.

14:38I wanted to try with open source models and so I wanted to rebuild old request for to eval. and see if I got the same thing out, if I got something satisfactory, if I need to change anything. And turns out that once you have uh you know this content addressing system, you can rebuild the request from the graph and you can just replay it exactly the same and you can you know resend, you can use a different model.

15:10Uh you can use a different request if you want, you can uh you can change it and so yeah, you get actually a lot of things I mean for free. You need to implement a thing. Um and so this is kind of different um from what you find I mean what I found when I started doing this it might be different today because it was a couple of months ago is that out there you had a lot of libraries so it's just frameworks and frameworks just

A kernel, not a framework

15:34call code uh your agents leave inside their abstractions um and I don't like analogies with you know operating system okay everyone has used that analogy but okay let's say a kernel like runs processes and your agent kind of is a process. It doesn't matter what it does actually.

15:50Uh but the system can schedule it is built to isolate it. It can isolate it and journals it with the log and the agent definition. So the markdown is user land like you don't need to use it with that system if you don't want to actually have a front end that doesn't use this markdown uh this markdown format at all.

16:05And okay, here's a very important point and you know that's kind of a takeaway and it's also what justifies me working on this because disclaimer structured outputs is our specialty and we've been working on this for three years and it just ended up being a big dog fooding project and the reason why I did this at the beginning was not because I absolutely

Typed calls and typed events as boundaries

16:30wanted to use our software. I didn't necessarily want to you know forklP to other software etc. It's just because Infropic was terrible at structured outputs and so like 20% of my events were wrong and were rejected by the system. So that's why I ended up doing this.

16:45And the goal, you know, the job of the kernel is actually to make bad actions impossible, not just unlikely. And so you have this two boundaries with between agents and the external world. The first one is type tool calls. The two tool calls uh you don't want, you know, you don't want to call tools that don't exist, etc., etc.

16:56and also the boundary with other agents which is typed events and this is non-negotiable I found like you can get a lot of errors just from this uh I wrote a really long blog post about this uh it's if you follow the QR code you'll find it and yeah and the result of that is I actually deployed it uh within the company after I built this and now today after a month of deploying it we have 20 agents on the left that are not just contributed by technical people by the way which is kind of what markdown uh what Maron gives you and then on the right is you know we deploy it's called

17:39the internet there's the briefs there's a bunch of I mean there's a bunch of things as you can as you can see kind of like a few you know as a conclusion a few lessons uh the first one is that well executed background agents are really magical uh they feel like this you know robot mower that I had at the beginning is I really just sit down and I come back and I have this morning brief that is probably even better than what I would have had just doing it manually and it just appears in my inbox every day and processes my you know random thoughts.

18:04Uh the

Lessons, and build before you buy

18:13difficulties that you meet doing this kind of thing is just good old engineering problems. I mean there's really nothing new under the sun when it comes to orchestrating these things. It's just good old software orchestration. Uh open source models are there.

18:22Uh they're good enough. I replaced so I don't have any third party APIs anymore. Now I just use open source models and even on my laptop I use a local model. Uh so it's good enough for what I do with it. For coding I don't know but for what I do with this it's good enough.

18:39The infro category is definitely unsettled. Uh I tried a few things before I started building myself. And I would advise that today like definitely start building before you buy. Uh so if you're a small company, if you're a tech CEO, it's kind of an advantage because you can just do this without, you know, tasking engineers to do this and get them off track.

18:57But I would definitely try to build before I buy just to know exactly what I need and you know the limitations of what exist. Uh also I will say that to people building uh frameworks for this is please eat your own dog food. Sometimes it's pretty clear that people are building you know agent orchestration frameworks etc but not eating their own dog food.

19:21So please do and the other thing is I'm really glad I took this two weeks off to play with the field because that completely changed I mean that changed the trajectory of the company. I know we're an AI company we should be in it etc but you know business is such that you're always thinking about the next thing the next thing the next thing and it's the same everywhere.

19:39But what I'm urging you to do is to stop and actually immerse yourself in this and try to see how useful it can be for your company. Uh so you can steal the code. Uh it's not a product that we sell and we don't intend to sell this. Uh you can read our blog as well.

19:57Uh so I haven't explained uh this yet but I will publish something about it. And thank you for your attention.