Why The Harness Matters More Than The Model | YC Paper Club

Why harnesses matter

00:08Welcome to YC Harness Club. I worked really hard. It took me like 10 its with Gemini to have Harsha riding a lobster. Gemini didn't want to do it, but we figured it out. Okay, welcome to harness night. Um, first order of business. Uh, how does everyone like the new YC paper club look?

00:24Well, this is like an idea that I kind of blurted out at EV, our head of design at YC, and two days later, she came back with this, and I'm like, "This is amazing." So, Ev in the back there, please take a bow. Very exciting. All right, why harnesses?

00:49Um, I mean, it's just a rapper. This is just scaffolding. This is just like prompt engineering. Um, why would this be at all worthy of a night? Uh, here's a great little Reddit that was only a month ago, which is actually uh the most aggressive.

01:03I'm not sure this kind of prompt engineering belongs at a top tier machine learning conference. Um, here's another great one. Uh, context engineering is not a research problem. Um, so I think harnesses have long been belittled uh as um subpar research.

01:17yet it is uh literally gives us an 18% bump in the difference between harness one and harness two. Um, and as Seth will tell us is the difference between getting ArcGI to work and not. And so this is a uh obviously worthy of some some amount of research.

01:35And I think if you look at this the classic meter plots release date and how long uh an agent can be running. Um, literally we go a lot of this progress has been because of harnesses. And so I call this the static harness era where there isn't self-improvement on the harness.

01:52And then this later latest latest um maybe the last 6 months has all been on the self improving harnesses and think and we'll get through all these and it's I actually saw this plot in a presentation by the CEO of trajectory. I actually really liked it.

02:05was talking about how the we we keep measuring perplexity and you know this somewhat correlates to IQ and and how intelligent the model is and so we keep pushing more and more up this intelligence uh IQ dimension but we're not leveraging test time experience very much so we're generating all this test time experience we have a new domain uh and it does and we don't really quickly adapt and if you remember one of the first YC paper clubs we did um I've been doing this experiment where um the as I increase the number of samples online

02:44how do you actually learn from just batch size one we don't really don't have that structure we have ICL and then once ICL gets saturated after just loaded like 40 or 50 it doesn't actually improve um on the valet at all then you have to go to Laura small rank then you go to Laura big rank then you go to SFT and it's kind of weird that we have this like different training procedures and so I think that's really where harnesses are shining right And what Arc AGI exposes, that's kind of the main uh um uh point of ARGI is how

03:16quickly it adapts to uh a new problem, a new a new distribution and does well in it. And so, um, ArcGI actually went through the batch with me, winter 26, and we helped Greg, you know, look at this and like the amount of thought and attention, as I mentioned last time, that goes into these these these games to make sure they're all orthogonal skills from game one to game two, so that it's isolating this fluid intelligence measure.

03:40Um really uh Claude uh Opus was one of the the first that was actually verified um on the the hold out on the private that no one else has access to other than Greg and Chalet. Um and the best that they got was 30%. And just with um some harness uh this thing that doesn't deserve any research just some wrapper and some scaffolding we can get to 95 and AVO from Nvidia got to 100%.

04:06So, Prime Agent and Nvidia, which both recently just came out. And the other thing I want to add, so when uh Carpathy launched his auto researcher thing in March, I want to say it was um I forked it and I was playing around with it. And all I wanted to do was make like a little user interface to kind of see what's happening and track it and and I ended up building a harness by accident.

04:25I didn't mean to, but it

Building an auto-researcher by accident

04:32was just like I wanted to see it. And and basically what it is is you specify a purpose. And in this example, which is actually a true one I gave, is like diffusion LM don't beat ARLM. But maybe if I ensemble, if I shard the diffusion LM into a bunch of different ones because there's such high arithmetic intensity per GPU on a diffusion model versus AR that I can actually um in aggregate by sharding them, I can get better better results.

04:57And so I just give this to a per as the purpose to the to my uh uh to my suite of agents um my swarm of agents. I give it some seed ideas. I want to vary the ensemble size shard at different amounts 100 times, 10 times, five times, whatever.

05:14Um and then I specify a valometric. Um and maybe I want to do GSMAK and like GPT2 setup or something like that. And then I have a scoping agent that will kind of look up papers that have that are are similar, look up GitHub repos. Um, we'll give it to a PI agent named Chris Ray.

05:34Um, who will then uh give it to a research agent um named John South Khan. Uh, there he is. Um, and then he'll work on it and then he'll start doing some stuff, give it to a council for some help. That's where I come in and me and Yaso will give you some feedback and then you work on it a bit more and then whenever you're ready and Chris Ray will kind of keep tabs on you, keep nagging you every every hour or so then it goes to this author agent to say okay now start stop freeze the idea start writing ablations and um and then start writing the paper and so then

06:08what this has turned into so this is the scoping agent uh and then you have this cockpit that kind of you can view from anywhere we do the uh tail scale up so you can actually view this URL from anywhere and see how it's progressing and and talk to it right there.

06:21Um, it just sends you email updates if you build the whole 1B thing in here. And then it actually starts publishing some papers and like I've read the papers. They're actually like good. They started out in March, April like not so good like you know May or whatever.

06:31But and now I just basically give eight ideas to eight H100 nodes and each one has eight eight H100s. and just keep going. And then I check in and they give give me back papers. And it's kind of wild what we're what we're dealing with there.

06:48And what's changed really is just the scaffolding that you put on top of it to allow it. And I I didn't spend an enormous amount of time on this, but this is largely how I I do a lot of my research, at least the the initial idea. Um, and so yeah, so I'll just give this six different ideas of things I want to try and just leave it alone and let it rip.

07:06And so the things that we can now do just because of harnesses on the same exact weight file is just wild. Um, so I

A five minute history of harnesses

07:14looked on I spent the the weekend just in prep for this reading um a whole bunch of the classic literature from self-refined to reflection to Voyager to all the tool former and I just wanted to just do like a my my shot at a five minute like history how did we get here?

07:29Um I think it was it's kind of important and I don't think a lot of people that are entering AI now kind of like have the context. So this is not going to be in chronological order. It actually doesn't make sense to to teach it that way or show it that way.

07:41I'm not I'm going to skip a lot of papers and I may course grain the papers absurdly. So, apologies. So, the initial harness the GPT2 February 2019 is a while not end of end of sequence loop. That's and then we basically have top P sampling and then we have the environment and that's the harness.

08:02And there's really not that much. There's no tool calling, there's no skills, there's nothing like that. And so that's the V0ero harness. And so basically if you have this GSMAK example, you're a math, remember the system prompt. You are a math teacher.

08:15This is like the persona stuff we used to have. Um there's a context. Susie has five bucks, she spends three. How much does she have now? And then there's just no chain of thought. It was just like four hashes and two end of state and end of sequence that is uh the GS MK format.

08:27We measure uh what happens after the four hashes. We get accuracy and you get a plus one or minus one if it's wrong. And then the entire last six years has been giving more functionality into the harness into a static harness. And so we said okay well what if we give that back into the context a bunch of examples like that like well here's an example and let's say now we'll switch it to four and one and hopefully we'll have um from that previous example we'll say oh okay this makes sense I can will help to learn and actually this is with a fshot learners paper um back in July in 2020

09:05and then we had chain of thought and says well directly predicting hash hash 2 is maybe difficult what we'll do is we'll smear the the compute the logic over many more tokens and we'll train it to get to that two now rather than just output the two and that was the chain of thought idea uh very cool and that was all context innovations uh and output space innovations action space innovations um and then then we came up with these tool former and webgbt webgbt actually came out first then tool former which is the idea of giving it tools and

09:37it can call a tool that's just really just a JSON object with a bunch of things specified but in this example just let's say subtraction instead of me calculating in the weight file what is 5 - 2 I can just call Python and call sub 53 and it'll tell me two and so that's pretty cool and you can expose all the tools in the system prompt and that's where tools come from and then we figured out megpt which is big the one of the coolest tools is being able to read and write to your own context and so before then all we could do is just

10:08append append append append to the context now we said what if I actually give you create read, update, delete on the context itself and we'll se separate just this one little chunk called a memory that you'll be able to to update. And then Voyager said, okay, well, we have these tools, but like what if I want to chain together these tools to achieve a task and then I learn it and how do I distill it back into the the system prompt to make it learn forever?

10:35And this is the skill this where skills kind of came about. And uh they did this on Minecraft. Um and this the Voyager paper and it's very cool paper. This is like largely now what a skill is. And so I have this skills.md and I have the name of it that I can go and search and here's the procedure.

10:49And then uh intercode this idea of like again in action space innovation if I can actually output code uh then I can basically now I have on the-ly uh tools or on the fly skills. I actually don't know if it's a considered a tool because a tool is technically an API.

11:02Am I outputting a function as a tool or a skill? I actually am still unsure. Um and then the react came first then self-refine then reflection. But this idea if I have multiple um agents that that have different roles and they can help self-improve self-improve on the context uh then I can um get smarter and smarter.

11:27So, I'll have take some action. In this example, I I I flipped the three and the five. Whoops. Um, I sent it to an internal evaluator to say, is this right? No, it doesn't look right. I can either, you know, keep looping here or I can go to the actual environment to get back a reward signal.

11:44Um, I go here uh uh to the reflection. They'll say, "Hey, you actually flip these two, go back, and I can improve the result." Um and this is the one of the first like uh uh ideas of like being multi- aent uh being let the letting the agent reflect on its own um output and improve it.

12:02And then this idea of multi- aent goes even further where I can actually spawn one of the tools could be I can spawn a a set of sub aents um and they they persist in these persistent ripples and they'll be able to be running and I can interact with them and I'll have the sub aent list.

12:25I can invoke those ones and I can keep adding to that the the launch sub aents and then RLM went even crazier to allow this in a recursive fashion so that um and I exposes this RLM query so I keep uh recursively calling the RLM query to uh solve a larger class of problems and that the leafs whenever I want I can call the LM query to spawn that LM agent as well and then I have this main orchestrator agent that's running all that and that's what I call like harness v1 one, this whole thing is like static harnesses.

12:56I'm not improving on the system prompt. I'm not um updating the harness itself. And this is this you, of course, we're going to have to have GStack on the top. He's the one that lets us do all this. So, we thank you, Gary. Love GStack. Um but there's some other ones that deserve a call out as well.

13:10And basically what that means to summarize all this, you have some agent spec, you have some system prompt here. You say, "How many turns am I allowed? How many tool calls am I allowed?" You don't want to allow infinite. You specify a tool list.

13:21you specify a skills list, the sub agent list, and that's largely what the V1 um is, and then you put it in a loop. And this can be spawned on a prompt if I'm asking it to do something in my Slack channel, which we'll hear about QM, which is like I use it every day.

13:39It's the team that made it is here. It's super exciting. Um and or it's on a crunch and just wakes up every hour and it decides to do work just like you know, anyone else. Um, and so there's some session management, there's a loop, there's this context compilation.

13:50We're actually creating the context. I'm

Self-improving harnesses

13:56putting uh uh all that into an LM call. I get back uh the action. And then I I may or may not have some tools that I need to invoke and append back into the context. And that's basically harness v1. And then the cool part, this is the most exciting part where we're seeing a lot of advancements, um, where you're letting the harness itself learn.

14:16either we're learning the system prompt or we're learning the harness itself, which is very trippy. And so, one of the famous ones that I actually wanted them to talk, but they're actually running a 150 person DSPY meetup tonight and in in this in San Francisco.

14:29They couldn't make it. Um, but I've done some podcast with them before and a super great community. Is this DSPY? So, demonstrate, search, uh, predict. uh they have this idea of of basically uh taking a train set a small set of examples and then learning the optimal system prompt.

14:48So I keep iterating, iterating, iterating. I can't back prop through that process, but I can do uh something called genetic programming where I'm finding candidates, I'm merging candidates, and with some merge rule, I'm evaluating, seeing what happens, and I keep working and working.

15:00And I basically gives me CRUD over the system prompt itself, and allows me to choose any system prompt. And then Darwin machines actually go a step further. Not only are you allowed to change the system prompt, but you're allowed to change the harness itself, the harness code that is actually running.

15:19And so you can imagine basically what happens is you have this archive of um many different uh agents which is harness and system prompt. Uh and you sample from them. You push them uh through you'll actually evaluate how how they did on some fit fitness function.

15:33Uh you'll uh add it back into into this archive state. And I skipped over the the self modify. You can actually you have a a meta harness that actually allows the the agent to modify its own uh harness so that it can become a different harness and then that basically loops around loops around and eventually you get better and better agents over time.

15:53And then the meta harness which is the main harness is to produce harnesses, right? Which is a really meta concept. Um and so this is the output space where you're configuring multi- aent context compilation. Um, and you're you're doing this and you have allow CRUD on all of it.

16:11And so it just keeps adding more and more uh CRUD into the harness code, all the green that you see here, the uh meta prompt, the system prompts of all the agents, how many agents there are. Um, and you kind of grow this uh this meta harness over time.

16:25And then one of the authors of this paper is the lead author is actually here tonight. Super exciting. You got to drive down and talk to him a lot of it about um continual harness, which is I love it. And they go even a step further where one they add they they add some extra color on the classes of memory and so they add this history thing.

16:46You know he'll he'll go into a bunch bunch more detail on that memory breakdown. But the coolest part I think is for the classic RL people. I see Robert back there. He definitely would would enjoy this Dagger style online learning where you can actually update the weight file itself.

16:58So you're actually doing a test time training on the LLM based on a small amount of examples that you just learned which I think is actually a huge huge important research direction that we should get working. Anyway, that's it. How you do all

Tonight's speakers

17:23right so um tonight we have uh three authors. Uh Ben got food poisoning this morning so he couldn't make it. It was very sad but we have three tremendous authors. uh Seth uh student under shein is I say it right uh at Princeton a researcher at Prime Intellect and the author of Prime Agent um John Sadvalone who's a the first time we've had a a call back um pres presenter very excited about that PhD under Chris an Aelia um and he's going to be he's the author of open Jarvis with Ivonica uh from my lab uh hazy which is a personal uh open Jarvis and which I think is really cool.

17:59And then for Josh and Rean who Josh just got promoted to be head of YC labs which is really exciting which I think deserves a little round of applause as well. Um be talking about QM and so QM there at YC we had this general agent that we use and they came out with QM a month ago and it is like meaningfully uh better and is so functional and I use it every day.

18:26All right, that's all I got. Thank you so much.

Seth Karten: Prime Agent, a self-improving RLM harness

18:35Hi everyone. Uh, great to be here tonight. Uh, my name is Seth and I'm going to be presenting Prime Agent, which is a self-improving ROM harness. Um, my job here tonight, I think, is to try to convince all of you to take a very first principles style approach to thinking about how you build your harness.

18:48So, we're going to be very, very basic here to begin with. If you think about and I love the uh introduction right from of all the background literature um which I thought was so cool I think is very complimentary the how I think about this as well um but if you think about just the raw LLM itself it's actually just this sequential processor that you know has some fixed weights and has some visible context and it's it's taking some tokens in and it's putting some tokens out uh to make the next decision.

19:17Um we don't really think of an LLM that way these days. We have a set of files that it has access to. We give endless programs and tools for it to use. Um, you can even message other sessions of LLMs that are going on and create sub agents in order to do all of these very cool things.

19:37But, but in its basics, it's just tokens in, tokens out. It's a neural network making a prediction. The harness itself is the layer between the LLM and the world that adds things like this persistent state tools and compute. Uh, here we have a diagram of how we think about prime agent.

19:48um from the human's perspective. So you open up your prime agent uh on your computer just like you would cloud code, codec, pi, etc. Um and it gets you into this agents view and this agents view is an overview of all the agents that you have going on for all your parallel sessions with like a very tight like tight summary of what you're using and then you can hop into one of those and check it out.

20:12there you're going to be at this root session and this root session is your basically the project orchestrator over all of these different sub aents that it's controlling and you don't have to ask it to start sub agents it will leverage sub agents when they're useful um and these are all programmatically called because this is all based on uh the recursive language model principle where everything is in this IPython shell um all your tools all your memories all your sub aents uh and and we expose uh for further coordination these uh these messaging paradigms.

20:44So you can manage all of these um and then these agents can directly interact with your environment which could just be like the programs and files on your computer or running an H200 node cluster maybe for your auto research. Each of these agents are then backed by this persistent dam on your computer.

20:56Uh this is so that you know when you close uh your laptop or you you control C out of the session, it's still running in the background. You have to actually stop the session so that make sure that you're continuing to running. And then we also have these other features we exposed from continual harness where it's able to provide like live CRUD operations on all of the components that we mentioned um in order to manage its uh memory skills sub agents um persistent and and prompt its own system prompt persistently.

21:26The way I like to think about all of this context that we're building up is that we have this sort of almost like like here I have like L1, L2, L3. is like a cache, right? It's like what is the most accessible information that we're working with and at the very like fastest like readily available information, you know, the models to be able to retrieve that really quickly.

21:48It's the model weights. So, everyone always wants to get all the information in the model weights. Um,

Context as an L1, L2, L3 cache

21:54but then we said, okay, well, maybe we don't have all the information because we don't want to have to fine-tune every single time to update because that's very expensive. So, we have this uh active input context. So, we're using lots and lots of tokens on the input.

22:01we might have some in context examples like we've seen previously in order to add to these different capabilities but at a certain point we run out of context. Um and so the very like earliest form of harnesses that we've seen that are still used to this day even by those who say we want the most minimal harness possible is compaction because compaction is a very generalized tool for the agent to be able to uh summarize its own context history in order to work past its context length working window.

22:33You can think about um once we go beyond like what are directly like inputs and outputs from the the model here into this L2L3. You might be familiar with the L3 which is more of the dispatch state. So if you're working with a file system, you can read and write from uh main memory.

22:50Uh if we're at the L2, which is I could think uh at a means in between uh what the active context is and working with your file system, you might have a live uh ripple, which could just be running things directly in Python uh an IPython shell like you're in a Jupyter notebook.

23:02And all of those variables are saved directly in your RAM. your agent can then programmatically manipulate them and run all sorts of programs directly on the information there saving tons of tokens rather than putting it directly into context.

23:14Uh you can also create sub agents and it's the same thing you're basically saving context here because you can task the agent with a specific set of information in order to perform some operations and the report back at the end. What is interesting here is what we talked about compaction for the active context, right?

23:31You have your context history. This is helping to update it over time. So you can continue to leverage this. But once we go beyond this, we need to be thinking about how are we doing these update. We talked about CRUD. How are we do beyond just creating reading?

23:43How are we updating and deleting our context over time beyond our uh or the state over time beyond the context line. Uh I like to think of this at the ripple is this aentic garbage collection where we're just cleaning up the variables in our state as well as like what sub agents could be used.

24:01And then so to make sure that our RAM doesn't crash my computer laptop every day. And then on top of that, we have this uh notion of refinement where we're updating and deleting the skills and memories and prompts that are stored on your system.

24:17You can think of that so that way you don't crash your actual uh out of space on your hard drive as well. And so this very much is a here's how we express this thing and here's how we revise it over time. The other perspective that I really like to think about and I really trying to push because harnesses are almost going towards this like agentic operating system that we're creating uh is when I think of it um more metaphorically here is that when you look at the raw LLM it kind of looks

24:44more like a touring machine where you have this ticker tape uh and you have all these instructions that are going in and then it's performing some set of operations and going out. But when you look at a harness it's looking a lot more vono like a vonoyman computer.

From Turing machine to von Neumann computer

24:57you're able to do these read and write operations on external memory and that makes it much more powerful and another class of problems than just what a touring machine is able to express on its own. And so yeah, the idea of like how do you build a good hardness?

25:08You want it to be the most expressable thing you can imagine. So some some like early harnesses before it gets into the data flywheel where the models can do themselves are very specific. Plan, act, critique, do these exact specific um steps.

25:22Well, now har uh the models are able to do that themselves. You can imagine like we we don't have like a react loop that we necessarily need to explicitly impose. The models kind of have natively uh figured this out. But what they haven't figured out is how to um you know they have to be able to have the expressibility to call compact.

25:41They have to be able to have a Python ripple so they can run programs. Um they have to have the ability to programmatically create sub agents and access state and have different feedback mechanisms. Th those are model controlled expressibility features and if you removed one of those you're actually removing a capability that it won't be able to do otherwise.

25:59The way we manage um and I'm sure you're all familiar with the RLM paper uh from my co-author Alex um fantastic bit work. What we do beyond what was in the RLM paper is we think about the age sub aents as these persistent subsessions. So each the parent station can create uh spin up a new RLM sub aent and each of these are then emitted.

26:17They run some task and then they finish and report back to some end state to the parent session. These are then idle. They're still working in your RAM. At any point, the parent session can then send a message to one of the sub aents to continue working and it has all that good context that you built up over time so that you're not missing information or have to reuse information that was already developed in a prior context.

26:42And then of course you know we don't want to use a lot of RAM. So we can move them offloaded uh in an inactive state which then can be called back at any time by messaging them in this persistent sub agent setup. I talked a little bit about uh continual harness uh which we have in a a prior paper of mine which talks about cuding the entire uh harness state.

26:59Um this is another feature that we want in our coding agents leverage all of our prior history. So you can imagine like some set of trajectories where they have some actions and outcomes or something happened um at a at each turn. And so we just kind of want to expose the ability for the agent to leverage all that information in order to update what the future harness is going to look like.

27:22Are we do we need to change our system prompt? Do we need to create some skills? And skills I think of as a set of instructions or a program in order to to achieve some some specific goal. uh memory which could just be long-term storage about things that are important as well as the sub aent specifications that we talked about in this very persistent manner.

27:40Were there uh certain sub aents that we want to reuse at a later time because the context is useful and just having the ability to do this kind of reflection or refinement um over time. It's very powerful for the models to have. They're not perfect at this right now, but this is one of the the capabilities that you want to you want to build your harness such that it is a bit better than what the current models are able to do.

28:05So then you can get those reasoning traces and use that to leverage your next iteration of model and they'll be able to handle the harness and be able to bootstrap themselves into a higher and higher performance. One of the coolest features that we have in uh Prime Agent um that we we've had since the beginning of when I was working on this, this is one of the first things I added um is the ability to message between any any two agents um within like some nuclear family setup, parents, children, uh siblings.

28:29Um and the reason why I did this is because I was I was constantly

Messaging between agents

28:35trying to figure out what's the best way to like myself to manage all of the agents I have doing everything for me in five different directions, five billion different directions every day. Um, and it would be so much better if they could just like share their contacts directly with each other and coordinate.

28:46And turns out that's fantastic for like typical software engineering and long horizon jobs as well. Uh, the last thing that we look at when it comes to how did we want to design our harness is we were really thinking about long horizon performance.

28:59I want to go run some jobs and I don't want to have to babysit my agents the entire time and when when I'm ready to come back and check in, I can check in with them and see what's going on. And this is a perspective that I also really lack seeing in a lot of the evaluations that we're looking at.

29:10Uh a lot of times if you run a model for not enough time or say, oh well the model stopped working after this amount of budgets, but then this other model kept working with using more budgets. Well, first of all, you're not even using the same fixed expenditure to compare the models.

29:28But second of all, that could also be hiding performance that you're missing. Uh the way that I look at long horizon performance eval is that I want to see what's the practical plateau. At what point will we only get incremental gains in performance as I throw more test time tokens at it?

29:39I have a couple experiments that I'm going to show after we've shared design philosophy here about how we created uh prime agent. Um we're going to talk a little bit about test time scaling and uh our our TI results as well as looking at um you know does is it actually helpful and why is it actually helpful for our information management for the ripple that we're working on these long

ARC-AGI results

30:04contexts and then um when we have these really really long like almost ultra horizon long horizon uh tasks uh how do we sustain these like multi-day work and like what actually goes on when we have these refinements um over like these settings that can last like a week at a time or more.

30:18So, this is a result that you probably all seen. We actually have a one additional data point that we added here that we didn't include in our original result uh just to compare across harnesses. We solved this uh we went out, we're trying to figure out what is the the best uh eval that people care about these days when we're running our harnesses and we're like, "Oh, we should do RKGI."

30:34I like, "Oh, yeah. Yeah, I remember. I I ran some results with continual harness and we got 20% with um Gemini Flash uh or sorry, Gemini Pearl." Um so, I I think we can get at least 20%. people who think that's really cool that our like general harness that didn't even like wasn't even structured for ARHI did really well.

30:51So I went online I was like okay I need to find a good system prompt because I don't want to make sure that we're losing information. So I found a another community leaderboard called prolong and I just grabbed their system prompt and I was like okay I'm going to grab their system prompt forget the rest and I'm just going to throw this directly into prime agent.

31:02Uh and then I ran this and I was like oh my god the first run that I got it hit 99.9% and then I looked at the logs and I was cheating. Okay. So, I was like, "Okay, I got to do proper sandboxing here. Like, let's set this up properly." Uh, and then so I spent another day on this.

31:20And then, and then I went back and I was like, "Oh my god, I got 78% with GPT soul. Like, this is going to be a great result." Um, and then we're back. It's like, oh, let's let's compare a couple other ones. And so, it's again, we just took the prompt, uh, general prompt that basically says, uh, use a world model to solve ARC AGI 3.

31:37Uh, here are the actions that you can take. um you have uh and then the general system prompt for prime agent which is like you have a ripple you can call sub agents uh you can use the it programmatically um and and so we went through I went through the traces and it's basically doing a bunch of different um like calls of the coding in order to like check out these different scenarios and analyzing the images and doing like image processing and it's a lot of really cool um stuff that uh it

32:04seems like it was doing reasonable reasoning while leveraging the the ripple that we had um as like one of the main things that was able to enable build this. Uh so I went through and I ran a couple other ones. We did GPT tero 25.7% which is really cool.

32:15You can see that compared to like what were the um like the week before we did this uh open AAI was like the guys the harness matters a lot when you're doing evaluations. We use the responses API. This is the result that we got. Um and we we ran Terra and and got almost like we we didn't run to completion this one but we got really good results in comparison.

32:34And then we go um that that we're already achieving higher than some of like the GBT soul extra high which was crazy. And then uh we went and we did Opus and hit 95.5%. We're like that's insane. We also compared to a lot of the other harnesses.

32:48So some people ask me like did you run this with cloud code? Uh I did. Um unfortunately the results weren't very good. Um, and so rather than having bad results, I just deferred to the the original cloud code results and some other people have run it uh with similar configurations to prime agent and gotten much better results since then.

33:05Um, but what's interesting is that a lot of the really

Emulator Bench and GPU kernels

33:12popular harnesses don't necessarily do well when prime agent does well. So like for air agent, um, we spent a lot of money very quickly and uh, we had to cut it off because I spent like $5,000 without making much performance. Um, not saying this is the best they could do, but it cost a lot of money to do so.

33:24Uh so I think that the cost to performance uh ratio is very important and one of the things that does save money is being able to programmatically work with your context. Uh we ran a bunch of long horizon um evals as well like oolong and some coding uh emulator bench which is going to come out soon which is a program bench alternative and we found that it was mainly parody or slightly better than these other harnesses like you across different models versus doing like pimono cloud codecs with glm 5.2 to Opus 5 and 5.6 as our setting.

33:54Another one I thought was really cool is we have this like program bench alternative called emulator bench where we're trying to reproduce entire emulators of computer systems or in this case creating like a Game Boy Color and check that out.

34:11And we found that what's really interesting is because it has this um ripple access in the RLM, it's able to use these programs in order to kind of do these like out of experiment uh loop designs in order to um try things out in a lot more expressable and free way before submitting the final solution to the greater.

34:26Uh we also tried this with uh GPU kernels um and we got about par results uh across different um both soul and kimico. One is better, one is worse. about par um which so we we're not overfit to like any one particular um evaluation here. Um what's interesting for the long horizon stuff is we had some auto research uh experiments that we did with the nano GPT speedrun but we scaled it up.

34:56We said let's give it uh 8 by H200 for um a week and see what happens. And you might be like, okay, prime age is going to do so much better, right? Because it's able to do all this programming. Uh, it's a little high variance. We can't attribute um any of the benefits to with the harness versus the model there because it's a very hard task.

35:12But what we can do is inspect a lot of the behavior that we've seen. And what's really interesting is that we're seeing models like deep 6v4, GLM 5.3, and Kim K3. Um, you can tell these were done a little more recently than our first results.

35:24Uh, and we took these and they were doing like what we call out of loop experiments. So we were trying to say how can I run experiments on like the CPU and like look at the parameterization and do hyperparameter search and analyze the data so that I don't have to spend like all my time running expensive H200 experiments uh because that takes the majority of the time.

35:44So it's it's running experiments that are not the main experiment in order to optimize them. I think that's really cool behavior that we're seeing uh as we we shape what would be what kind of things we need to for the expressability for prime agent.

35:51So you can use like really good auto research because you can imagine if it's good at auto research, it'll be good with you. It be even better with a human in the loop to bootstrap your experiments. Uh and finally, we also streamed a 7-day factorial run which used a total of 633 agents um across uh 23 million tok output tokens in order to make like steady uh tech technological advancement across the tech tree to continue to progress over time.

36:18And here it uh one of the main benefits is I can use like these sub aents that can divvy up into different tasks in the factory in order to research and build and gather resources and build the next items to design the factory. Um as well as it can use the refinement to leverage what happened in the past in order to help in the future um over these very long context so it doesn't get stuck.

36:44And one of the most interesting things here is that it does not get stuck and it continues to make technology progression even at the end of our uh stage. Um this is more like a Gemini plays Pokemon kind of uh conclusion here. If there's one thing that uh I find interesting today u but like what takeaways you should actually add to your own harness.

36:57Um I think that you should think about agentic context management. Uh you should think about swarms and looking into further depth RLMs and trying to run standardized eval. All of the results that we can they showed today can be run with our uh verifiers uh package that we have at Prime Inslect.

37:15Um and shout out to my collaborators who are fantastic and I love working with. Thanks. All right, next up we have John. Hey everybody. I'm super excited to talk

Jon Saad-Falcon: OpenJarvis, personal AI on personal devices

37:33about a project um that we've been working on at Stanford. Um, I've been working on this with Ivanka Orion, my my co-lead author, as well as our adviserss Hazeni and Christopher Ray. So, personal AI is everywhere, but it's mostly cloudbound today.

37:46Uh, we see lots of different harnesses and projects focused on making daily writing, research, coding, and scheduling. But projects like OpenClaw and Hermes agent typically rely on cloud LMS um for most of the intelligence and for most of the most of the queries.

38:04Um, what does this mean? It means that it's pretty costly. You're getting thousands and thousands of dollars in API costs if you aggregate it over a year. Um, it's not private. You're often sending your most personal um, data to LMS up in the cloud and you don't necessarily know where all that data is going.

38:17Um, it also requires you to rent your intelligence as opposed to just simply owning it out of the box. And finally, it tends to consume orders

How far behind are local models

38:28of magnitude more energy than just running these LMS on your laptop. And so the local LMS are finally good enough to actually run a lot of these queries that people care about. And so we see that um the the current LMS of today are only 6 to 12 months uh behind whatever is the state-of-the-art frontier models um of before.

38:43So you see um LMS today such as Quen 3.8 27B um that achieve roughly the same performance as like Claude 4.6 Opus um back in the day. So that was kind of the state-of-the-art model back in August 2025. Um, and that gap seems to be closing uh more and more as the hardware accelerators that we have um for our laptops and for our workstations get better and better.

39:09Uh, just this week we saw a new release from Apple um with the new Mac Mini. And so we're seeing this renewed focus from Apple as well as Nvidia to build accelerators specifically for personal use cases. And

The five primitives of a personal AI stack

39:21so with this project, we wanted to explore the the main question of can we build the core of a personal AI stack, namely the model inference, the agent execution, the memory, the learning, basically the parts that are mostly reliant on the cloud today entirely on device while staying competitive with these cloudonly stacks.

39:34And so we decided to propose open Jarvis. Um name needs no needs no explanation. Um but we wanted to explore just how much of this we could run on device completely for for free uh while preserving uh security, privacy and quality. And so to construct open Jarvis we wanted to create the simplest set of primitives for which you define any sort of harness or or personal AI stack.

39:57Um the first one is whatever user interfaces you need to use. Um the second one is the actual agentic logic around composable reasoning and using different kinds of intelligence and tools. Uh for the intelligence, it's whatever LM you're using as your engine for keeping everything going.

40:09Um so this could be Quen, GBDO, OSS, Gemma 3N. Um and then whatever actual inference engine you need to run it. So this could be O Lama, um Llama CBP, VLM, SG Lang, um including whatever hardware you're running it on. So this could be Apple Silicon, Nvidia, whatever you need.

40:27um for actually making all of these agents and intelligence useful you need some set of tools in memory that can be run through a standard MCP protocol um and you need some sort of uh set of primitives for actually doing learning whether it's prompt based techniques like Japa or DSPI um whether it's weight based techniques like gpo and sftt and Laura um you need some way to actually get this agent to improve over time and actually be able to make it more personal and more effective and so to

40:57kind of walk through like what opens looks like um we tried to go with all of the standard um interfaces that people are already accustomed to. Um so we wanted to give people the ability to interact with it through a desktop and actually just run it as they would normally expect, but then see all of the savings that they're getting in terms of dollars and energy.

41:10Um we also wanted to give people the ability um to run different kinds of continuous agents. So different kinds of agents that are persistent in terms of cron jobs and being able to run standard protocols um day after day. Um, basically we just wanted to to make this like plug-and-play with all of the workflows that people are already accustomed to running.

41:28Um, and we wanted to make this something that can get people to have their first experience with LMS on device the same way people had their first experience with ChatGpt or Claude back in the day. Um, so yeah, and so yeah, to step through a little quicker, but yeah, here's like a nice way to like set up new persistent jobs.

41:46Um, yeah, we have all of these different components. We need some way to actually optimize it. And so we wanted to get out of the way of the LM as much as possible by just creating a simple spec of these five primitives by which they could go through the optimization.

41:59And what we found is that by going through this whole optimization loop, we were not only able to get significant dollar costs uh dollar cost reductions, but also significant latency reduction and significant um improvements to overall quality on these tests.

42:11And so this this configuration is meant to simplify down to just the five main things that people care about when they're building these these LM uh harnesses. So the intelligence, the engine, the actual agentic logic around it, the tools or learning systems required for running it um and the whole optimization um for the whole spec as a whole.

42:31And so something that we thought could be interesting to help bridge this gap between local and cloud LMS is to actually have the cloud LM go through and manual and uh and automatically optimize the whole LM the whole local stack. And so this is a nice way of taking advantages of the

Letting cloud models optimize your local stack

42:49capabilities of cloud LM to diagnose proposed changes and gate um to create improved solutions for these local LMS while not incurring the cost of those cloud LMS when you actually deploy these um local stacks at inference. And so what we found is that these uh these open Jarvis jobs that were these open Jarvis um configurations that were actually optimized by cloud LMS like cloud or chat GPT um were much more effective than uh local stacks that were just deployed out of the box because you could actually cater to the specific LMS

43:21the specific harness uh that was needed uh for for different kinds of workloads. And what we found is that even with the ondevice LMS of today, we can rival cloud LMS on different workflows around personal AI um personal use cases, coding, agentic tasks.

43:35While there remains like many tasks for which um like local local size LMS are not enough um the gap is surprisingly closing um month after month um as these LMS become better distilled, more effective and also we get better accelerators um for running them.

43:50And so even with LM of today, we can get 800x lower lower

800x cheaper than the cloud

43:57costum in terms of actually running them as well as a significant reduction in latency. Um what we also found is that no matter which cloud LM that we cloud LM we picked um it was useful um in terms of optimizing the whole aentic loop for these uh local local open Jarvis configurations.

44:10Um we found that um the Opus series, Opus 5 as well as GBD 5.6 Soul were were naturally the best. Um but was interesting to see is that you could pick Gemini, you could pick um other other uh larger um LM families like Kimmy and GLM and use them to optimize these local configurations so that you could capture those efficiency gains, capture those performance gains um for local inference later.

44:33What we also found is that the whole open Jarvis harness was cheaper to optimize than alternatives which might might require more data or more LM calls. Um we found that like this set of specs um and this set of primitives um was most effective for local LM settings because it got um the whole optimization loop and the whole um set of LM abstractions out of the way of the cloud LM to just optimize the whole system and just make it make it really fast and really effective.

45:02Uh looking forward, we're excited to keep building out this project. Uh we think in the very near future you're going to see um a huge maj a huge proportion maybe even a majority of uh people's daily inference calls going to local devices and on-prem uh laptops or on-prem workstations as opposed to the kind of standard of today where everything's being pushed out to the cloud.

45:17Um we think these trends are only going to continue because the accelerators keep getting better and the LMS keep getting better. Um and so if you're excited about um anything in the stack, whether it's uh better local LMS, better accelerators, better inference engines uh for deploying uh beyond data centers, um please reach out.

45:37Uh we'd be excited to chat. Include a QR code of the project. Um if if folks are around here afterwards, we'd love to chat. Thanks. All right. Now, we have our own YC. We have Josh and Rean.

Josh France and Regan Bell: QM, YC's agent harness for work

45:58Hi, I'm Josh and uh this is Rean and we're working on QM, which is YC's open-source uh agent harness for work. QM is one system that uh gives every employee at YC an open claw-like assistant uh that's like fully customizable and available in Slack or via web UI, which is uh what we're looking at here.

46:17Um each person works within QM in their own personal context that has its own sandbox files uh and crons and it can they can also work with QM in a multiplayer setting like a slack channel. People use QM for a pretty broad range of things uh like a lot of automations like email triage like legal and finance workflows.

46:43It's really good at editing documents and pulling data out of our internal database. Uh it can also spin up live internal web apps and help with stuff like planning events. Uh but it's designed to be broadly helpful for the range of tasks that someone might encounter at YC uh on a day-to-day basis.

47:07So you might be wondering uh why we built this and it's really the result of a string of internal agent projects that have kind of unwound throughout the years and all of which were really riding this tailwind of increasingly capable models.

47:26Um the first one we built was in like

A history of YC's internal agents

47:32January of 2025. We internally refer to it as the quote unquote like general agent. Uh, but it was pretty straightforward. Just kind of a system prompt with tools uh in a loop. It was one sizefits-all. Um, sort of like everyone was talking to the same thing.

47:44Uh, and it was pretty straightforward architecturally, but like still uh very or surprisingly good at answering data questions. Uh, interestingly like the scope of what the general agent was good at just increased I guess unsurprisingly as the under underlying models got better.

48:05Uh, and we eventually hooked it up to Slack. We added crons uh and gave it a a few more tools so that it could be more be capable across more domains. Um, in June of 2025, uh, by then like a lot of our engineers started using cloud code and codecs.

48:23Uh, and we realized that you could pretty easily run these in a VM. And then, uh, we hooked that up to a Slack tag, which was a pretty powerful medium for people who just wanted to like run a one-off code change. Uh, we also configure configured it to run our CI pipelines and then spin up dev environments for testing.

48:46Uh, and so people could come in like describe a bug or something they wanted to see happen and the bot would go off and actually solve it, which is like a pretty powerful um thing for someone who like maybe hadn't uh made a code change before in their life even.

49:01Uh but we also on top of that had a small loop going where we would observe sort of how the bot failed uh where it went wrong and then update the agents.mmd which was uh present in the codebase at the time um to make sure that the thing got better as as uh we like observe the usage and so in January

OpenClaw and a fleet of 50 agents

49:26of this year a lot of the partners started using openclaw and one thing to know about YC partners is that they're incredibly busy uh between like office hours uh they get tons of inbound email they're always reading applications so like any tools that can give them additional leverage are incredibly valuable to YC so open cloud particular was useful because it was the first agent that a lot of them had used uh that had their own computer that had its own computer and so this made it like very customizable in a way that the previous paradigm of agents was

49:57not uh and it functioned almost like a personal assistant And so in April uh the question became like could we provide this to every employee at YC uh without uh like buying everyone a Mac Mini effectively. And so we ended up provisioning a fleet of like 50 plus uh Hermes agents that were running in VMs.

50:17And these were definitely pretty helpful but they required a lot of configuring for people to get value out of them. And it was just like inherently kind of difficult to manage this fleet. Um, it was sort of like a whack-a-ole situation where I would have to sort of like SSH into these individual instances and fix them.

50:38And so the follow-up question became like we've got we've gotten a lot of value out of these agentic systems. Uh, like let's build something that tries to address some of the downsides of running this big fleet of Hermes agents. Uh, while still maintaining the personalizability and some of the like the stuff that people were really getting value out of.

50:57And so,

Pulling the brain out of the sandbox

51:04um, yeah. Okay. So, basically, uh, there's a pretty clear trend from whether you could see there from what Josh was showing you. Basically, um, the models are getting better exponentially. Um, and we were starting to see just increasingly impressive returns from giving them more and more capabilities.

51:16So, OpenCloud gives the agent its own computer and we start to see really impressive returns from that. So around May this year, we started thinking just like how far can we push this if we just keep pulling on this threat. Um we really like the the lens of sort of unhobling.

51:33I don't know if you guys uh read situational awareness when it came out in like 2024. Um but that was kind of an era when like test time compute was just starting to become a thing and like you know we're starting to give agents tools for the first time and there's sort of this intuition that what agents can do like there's a little bit there's kind of more intelligence in the models than we're than we're using in a lot of cases.

51:50Um, and it's like if we really push uh the frontier in terms of just like what we're offering up the agents as as capabilities that they can make use of uh like magic can start happening. Um so the first way that we do that with QM um is by essentially pulling the brain of the system up out of the sandbox.

52:09So with uh you know with Hermes and with OpenClaw you effectively have um the agent has its own computer which is super powerful but it's also trapped inside that computer. So that causes a few issues just from you know uh us trying to administer that system when there were you know uh even a few dozen of these things it starts to become unwieldy almost immediately.

52:26Um but the other issue with that is that you um all all of the like all the sessions that you would have uh they're trapped inside that computer. And so what we did instead is we uh we just offload everything into Postgress. So everything is centralized um from all the agent conversations that people are having.

52:45Um and then we expose those to the agent itself. So it can look at all the context that's sort of aggregating from across the system. And then the other thing we do is we start thinking about sandboxes less as this home where the uh where the agent lives and where it's kind of stuck in a lot of ways.

53:04And sandboxes become more of this thing uh more of a resource that the agent can dip into and use as needed. Um but it's it's um it's a lot less limiting. Uh the other thing that this starts to open up um is this idea of you're accumulating this large eval set of all the traces that you have from the conversations that people are having with the agent.

53:20And in principle, you can think about going and and hill climbing on that um and sort of having this automated improvement loop. Um we've had sort of mixed results with that. I would say I think typically if you're just dispatching this like torrent of agents that are supposed to um fix all of the bugs that they're encountering when you have the LLM as a judge, you start to get this kind of uh like main character syndrome where the agents are making fixes that are, you know, only seeing their their um their piece of the elephant effectively.

53:46like they're they're really um they can be sort of yeah not not seeing the whole um whole system and so having the human in the loop there has continued to be really important although we're really looking forward to this uh working uh all the way around.

54:04Um so the other major thing we do that's pretty obvious is just wire the agents to all of the resources across the company that we can. Uh we already happened to have a CLI at YC that worked really well um that wired a lot of systems together.

54:16Um uh but anything that wasn't in there uh we basically allow um adding just arbitrary API keys um that sort of thing. And then we also want to ensure that we have par with uh just an employee working on their laptop. So you know device code OOTH um we go ahead and ingest that into a keychain and then refresh it for you.

54:34So it's it's uh ideally supposed to imitate the experience of a person on their computer. We mostly keep this to to be read only uh in the database but we do

Letting the agent choose its own sandbox and model

54:48allow for rights um via uh human reviewed bulk upserts. So the way that works is the agent will put forward a plan to um edit the database um that a person can kind of give a once over and ensure it's not doing anything crazy before the write actually happens.

55:00Um one thing we've observed with this is that we've started just kind of rubber stamping these. It's a little bit like um I think if you guys use cloud code in the early days like you might have been reviewing the tool uses very closely and eventually um you sort of build up more trust in the agent.

55:17So this is something that we're uh looking at very closely over the over the next few months. Yeah. So sort of like I was saying um the sandboxes in this system we like to think of as a resource for the agent. So and and less where the agent actually lives.

55:30Um so in QM the agent can um basically uh by default it's going to be using a particular sandbox. So that's going to be one that's been allocated to the user that it's talking to. But in general uh there are there are environments that the agent can kind of converge on and uh and can collaborate with.

55:49Um, and then the other uh key thing is that um if the agent is working on like a um like a heavier dev workload, it can go and reach for a machine that has more resources. Um if it's working on something that's simpler, it'll just go for a sandbox that's that's less powerful.

56:01Um and so pushing that decision into the agent itself rather than the harness has been a really um really powerful thing. Similarly, um allowing the agent to uh tap into its own runtime. So basically, if it can choose the provider um that it's working with, you start to get out of situations like um I'm sure you guys have run into this with Fable.

56:22If you try to do AI research, if you try to do cyber security, anything, you'll get a bunch of refusals. Um so what we can do when the agent can control it on runtime, it can just pop out into another model uh when it needs to avoid a situation like that.

56:32Um similarly, uh like in the earlier situation, um it's often useful to pop between different sandbox providers. Um and so that's something we can do quite easily. And as a general rule, what we've tried to do is keep the harness extremely thin.

56:46Uh we think of the kind of core of the system as being these three tools um where you have execution in a remote sandbox, reading and writing from object storage and then uh publishing internal apps, a pretty simple sort of getbacked uh system.

57:05And then we have other tools for interacting with memory and crons and that sort of thing. But we really think of these as sort of temporary um papering over rough edges in the system.

The grind tool: budgets on goals

57:16And really the core of it is is these three up here. Um it's we we really try to keep it as small as we can. Yeah, we're we're sort of trying to be this like uh AGI anticipating harness. Although uh since we aren't there yet, um there are a few things that we've run into.

57:29Um, one of these is that the agents have been we've tried to put them in this really capable environment where they have all these uh tools available to them, but um they often give up way too early. Uh so one thing we've experimented with especially over the past month or so has been setting uh we call it like a grind tool or um basically we set budgets on goals.

57:50So the agent is not allowed to give up on its task before a certain amount of like walk clock time. just like a couple hours uh or a certain amount of token spend. And so um what that can accomplish is just like uh really a lot better, you know, uh research outputs, better reports, that sort of thing.

58:03Um and it's been really fun actually to see uh like OpenAI and anthropic um you know, crack some open problems in math with like a very similar technique. Uh but it also works for just you know, normal office work stuff too. Um the other thing that we've seen a lot of is so this harness is supposed to work uh it works in multiplayer.

58:22It works in Slack. Um, but because of the the artifacts of its of its training, effectively uh what we see is that uh the agent gets can get very confused about the situation that it's in even if we specify this pretty clearly in the system prompt.

58:40So having like local affordances for this has been something that's been uh that's been really important.

Agents don't understand social context

58:50So yeah, um the other thing that has been a problem is that agents really don't uh understand social contexts. Um to make this a little more concrete, like if I tell Regan a piece of information, uh he intuitively sort of knows uh or at least has like a good mental framework of where it is okay to share that information.

59:05Uh but it takes some actual work to recreate this with an agent. uh like privileged information can very easily just leak into these contexts where it should not be and so uh the information that you can put in the brain is effectively like bounded by how good your permission system is uh and so YC luckily has an existing software system with like fine grained permissioning uh that has been built over the years but a lot of people just don't have that uh and so it takes uh work to allow for knowledge sharing in an in nuanced Okay, so thanks everybody for listening.

59:45Um, you can try out QM. It's open source. Uh, and coding agents are pretty good at standing it up. If you run into any problems, feel free to put up an issue and we'll look at it. Uh, we're also hiring. So if any of this resonated with you or would be exciting, then feel free to send us an email.