How Anthropic Builds: Lessons from Labs — Mike Krieger, Anthropic

From chief product officer to IC at Labs

00:12Joining us on stage is the co-founder of Instagram and a member of technical staff at Anthropic, Mike Kger. How's everyone doing? I mean, good morning. Nice. Uh, Mike, thank you for releasing Fable just in time for us. Exactly. For the conference, we timed it.

00:44Um, we're we're so glad to have you. uh you are uh one of the preeminent builders and your leading labs at Enthropic. Um how has your model usage changed as as you've you know seen models internally grow? Yeah, I mean for me it's been like both the model shift and then my role shift.

01:05So I for like the first two years I was at anthropic I was chief product officer and then I kept seeing people build with the models and the FOMO just kept increasing because I was you use the models as much as possible but for example on product strategy I would write a strategy doc and then have cloud critique it and maybe you can use a workflow but it's not quite the same as like building in that pure way and I was like spending all my weekends trying to

01:28build with it and I realized okay I actually just need to shift it's like way too interesting a time and it's actually an interesting trend I've seen now like several people that where CTO's at other places are like now joining as IC's at anthropic in other places but I made a role shift and it was actually right around the time where we started getting sort of internal snapshots of what became mythos and fable and what was really interesting watching that sort of shift was um that kind of change between I have an idea I'm going to like sort of break it down in my head much

02:00more how I would do engineering normally and then kind of iterate through these different steps to moving to much more of the paradigm of I'm going to describe the goal like go off and work on it and then like we can talk about what trade-offs you you know surface some questions along the way but then figure out what where you landed and where we can go from there.

02:15I find it's hard. I don't know if people have this experience where and I know Babel's only been reenabled for a couple of days. Babel's definitely way way smarter than me. So sometimes it'll finish work and be like here's the trade-offs I made.

02:26I'm like can you explain it to me like I'm a little dumber than you are because I need you to like sort of break this down for me. But that's been one sort of big change is sort of moving from that task delegation to like express the end state and then have it go and and cook on it.

02:40Yeah, we're all learning how to delegate better. Uh Tariq did us a huge favor yesterday. Um we do I want to read it in the newspaper. Um it's you know that we have we have uh writeups of talks now in in like the next day's newspaper. He

Be unreasonable, and why people still ask small

02:55said be unreasonable. In what ways you know have you been more ambitious with your prompting? I love that. I mean, I love that framing. Um, we actually just hit this to like one of the labs initiatives I have is this internal product and uh, somebody was like, hey, it doesn't work the way I want it to.

03:09Um, and can you make some changes? And I realized I'm just going to go ask Claude to do this. Like, why don't you ask cloud? And this was a non-technical person. So, I actually think as an industry or even as a product team, we have to teach people to be more unreasonable in their usage.

03:26And it's sort of hard to imagine. I think that that if I can digress for a second on product design, I think right now the like kind of first generation of AI products, we put them too much in a box and constrain their their sort of access to tools or kind of degrees of freedom, which means it was much harder to be unreasonable, right?

03:42When you say do this thing for me and then it would be like, whoa, I I can't I can barely like I can write code, but I can't really run it or I can kind of introspect my environment, but not really. Um and I think as you see our own like product progression even with things like co-work like you know does every single like knowledge worker need a virtual machine that can write bash like on the face of it no but then when you realize oh actually that way it can

04:06remediate an issue where oh I tried to parse a PDF using our built-in PDF for I hit this yesterday and it was like ah I can't parse it this way well okay well I can probably write a script that can do this as well. Um so I think that's it. My most unreasonable thing though was u one of our labs projects I wrote in Python like near and dear to my heart.

04:23All of Instagram was in Python. I think they're finally converting it to PHP now that they have um like models but I know tokens. Um and uh for deployment I realized that cloud code had like figured out a better deployment story with bun and I was like okay I need to port this whole thing from Python to TypeScript.

04:39like as a you know if I put on my like 2010s engineering hat or even my early 2020 20s like that's a dumb idea like who would ever port like at that point you know a couple hundred thousands of lines of code um but I was like I think this is doable now and I basically created this dynamic workflow

Porting Python to TypeScript over a weekend

04:55setup and over the weekend had it port the whole thing like verify it double check it then read both code like basically churn and churn and churn and then came back Monday to a completed workflow that was a ported version of that thing. So that probably ranks on like the more unreasonable things like, "Yeah, just port this entire Python codebase to TypeScript.

05:09Get it working, get it deployable in, you know, a weekend." Yeah. I mean, a lot of people are talking about the the bun zigg to Rust version. I I think a lot of people are also like, well, it's a compiler. It's a it's a runtime. It's got lots of tests, easy to do.

05:26Can you port Instagram, which you would know very well, to PHP like that, like a like a product? Yeah. I mean I think the product side it's even I don't know if easier harder. One of the things we did at Instagram this is when Python 3 came out and we were able to add typants for the first time and it was people had a lot of internal conversations like are we going to run out of steam on Python and my perspective was always like I think we can take this way further than we think we can.

05:45Uh, but I think types are going to help us not sort of be in our own way. And we built this thing called monkey type where we basically like captured runtime type like basically the types that were actually getting used in production and then mapped those back to to the types in the codebase.

06:01And I think because of that sort of pattern, I think there's really interesting ways in which if you're doing sort of conversion or sort of cross-co compiling using LMS, you can also lean on production data a lot more or run sort of like segmented tests.

06:17I think that like there's a lot of uh things you can do there. But yeah, I think it's I mean the sky's is the limit there as well. I think the hardest part is always finding the boundary around where you can start doing it incrementally without trying to boil the whole ocean and like swap it overnight.

06:29Yeah, I mean your users are your test ultimately and um you know we I also read another article in the newspaper about how you can just use rollouts and sometimes you don't really know uh what you're going to need it for but when that infrastructure exists for you to experiment and to roll things out it's enables so much.

06:45Yeah. Yeah, I mean I always found this was advice we got. It's like we launched Instagram and the happened to be the first week everything melted cuz we didn't really know what we were doing on

Scaling lessons from 2010: measure everything, build knobs

06:52the backend side of things. And uh coincidentally that week there was like a lunch that one of our investors just scheduled like not even for us. It was just a infrastructure lunch and we ended up spending we totally like monopolized that conversation because everybody had their own opinion about how we could fix our scaling.

06:59Um, and like the two pieces of advice I got there is like 2010 that I will like will forever retain is like um like basically like pre-measure everything that you think you might even remotely need because the worst thing is an outage where you're like well is this like number normal or is it high and like oh I don't know because I don't have data until I just added this metric.

07:26And the other one is being like really thoughtful about knobs and feature flags. So even you know early Instagram we had like a very uh simple but really effective like way in which you could do like ramp ups and roll outs and dynamic config too where you know a lot of our runtime configurations had to be changed you know in a matter of seconds so that we could handle load and being able to like do that in a first class way was was really important.

07:41I'm seeing that definitely in in AI as well where you know we're making all sorts of different trade-offs and having that kind of runtime configuration is super key. Yeah. Uh my my favorite scaling story from Instagram by the way I think it's like your launch day when you you do yourself with email.

07:52Yes. Which people should look up that story if uh if you haven't seen it. Um I wanted to go into tags. Uh very very major ship. Uh it's it's how 60s

Tagging Claude in Slack, and multiplayer delegation

08:08something percent of your code is written today. Yeah. Um how did you square that with everything you just said where it's like very dynamic like you don't actually ship one app, you ship one app with 3,000 flags. Yeah. And like well what are you working on today?

08:19I don't know. Like it's it's for this segment of the population. Yeah. Yeah. I mean I think there's a bunch of things. So like with I was really excited. I was talking to Swigs earlier like I'm really excited that we have tag out there because it is uh how we've been working for a while and I would get up on stages and people be like how do you work at anthropic and I'd be like oh yeah we use these things like that are not quite cloud code but

08:40you know uh but it's hard to describe it but I mean if you like got to poke into anthropic like you would see u of course cloud code usage for things that are like more interactive or if you're kind of iterating on a particular uh sort of sort of specific thing where you want a lot of like high sort of bandwidth back and forth But most usage is actually much more delegating uh via tagging and via tag and you can say like here's the and the reason it's really interesting

09:04is how multiplayer it is and it reminds me sort of like um actually like midjourney like the fact that everybody was on discord seeing how other people were using it. I think actually to your earlier question really helps with that unreasonableness or ambition where the first time you see somebody tag claude and be like hey you know don't just fix this bug but like now you are responsible for this part of the codebase and I want you to monitor this feedback channel and proactively take on

09:28tasks and then fix them and then also take like you know if this API changes do that like I saw somebody do I was like oh wait I've been totally underutilizing this thing I've just been using it as like a glorified cloud code in Slack like that's definitely uh totally like uh sort new version of it, right?

09:37And the more advanced version is really trying to start thinking of it as a teammate that is actually sort of holds context has memory and can be proactive. And that's just really changed how we operate internally. It's much more like this multiplayer async proactive way than it is a you know most people off in their own CLIs.

10:01Are you bottlenecked by code review and git? Obviously there is cloud review but someone usually still looks at it. Is there a world in which you just merge it in? Yeah, we're it's a really good question. Um, we are definitely still bottlenecked on reviews, especially for things that are like touching some architecture pieces and it's actually more subtle than just being bottlenecked on review because that's, you know, okay, we can carve out time differently.

10:24It's like bottlenecked on human ability to even like fully conceptualize what we're doing. So, one of the reasons we built cloud code artifacts that we shipped a couple weeks ago was partially for that, which is uh you would send somebody a PR and then they'd be like, I don't know, man.

10:32this is like 2,000

The review bottleneck is comprehension, not time

10:40lines of code. Like it looks like code to me. Um, and what we started doing instead is sharing much more like here's a cloud code artifact. Like here's the explanation. Here's the intention of the of the change. Here's the trade-offs that were made.

10:48And like I think that's going to much more be the trend by which we communicate, which is the code is ultimately, you know, verifiable using some things, but actually like discussing intent and trade-offs and then measuring in production is I think the at least the direction of travel we've we've gone in.

11:04I don't review when I get a poll request. I wish I could say I've reviewed every line of code. I definitely do not. I like actually talk to Claude about the the code and say, "All right, like these are the questions that I would have. Can you go investigate it?"

11:12So, it is kind of cloudpowered code review, but still human driven. And and for the really important ones and for the ones that are like cosmetic visual changes, it's much more like look like we'll fix forward if we need to fix forward, you know?

11:30Yeah, totally. Um, I think a lot of people here are trying to figure that out, too. Um, I wanted to talk also a little bit about enthropic labs in general. Um, Nili Patel who you've probably met before loves to ask ask the question like draw the org chart.

11:37Yeah. Like like how like uh people you know you ship your org chart like I think it's important like everyone knows cloud code now you've got tags. Um how are you structuring the labs?

Persevere or pivot, and bet leads who manage nobody

11:54Yeah it's a good question because what we were trying to wrestle with was you want sort of people to be supported like you know I think the death of the engineering manager discipline has been greatly exaggerated. Like I think there's still a lot of coaching and interpersonal pieces and personal development that I think is still really really important but especially in a labs type group where like our whole cadence is two week reviews where every project goes up for we call it persevere or pivot.

12:17So basically every project is up for review and either it's time to you know keep going persevering or you know it's time to pivot it or even shut down and you know we shut down projects basically every single one of those cycles and it's like the more you do it the less it's just like oh no my project has shut down I failed.

12:33It's like no that is definitely the intention of the lab team is to prototype quickly try to ship internally maybe get it to early access and if it doesn't work wind it down but because of that kind of like rapid iteration it means that if you align the org chart too much to the individual projects you're going to end up like reorginging every two weeks which would be a total nightmare.

12:49And so we've actually ended up with this interesting setup where like the the pod or the team that is working on a given we call them bets within labs definitely just draws upon like all right somebody from product somebody from the team um you know I'll jump in when it's a product I'm particularly interested and I'll come in and work together with the team on it um and that's the unit for that time and there is the concept of a

13:14bet lead or a directly responsible individual but the interesting thing is that they don't manage usually any of the other people which kind of breaks the kind of previous way in which a lot of these things were done But I think it leads us to be really flexible when you say, "Okay, actually this project is not going to work out.

13:22Let's disband and keep going." And it's not a big deal. And the engine manager is much more playing the like make sure every individual is assigned to the thing that they're most excited about and that they're working in the best way possible.

13:31Now what we do sort of solidify is when there's a product that has like legs. like cloud design for example started in this sort of ad hoc sort of group way and then now that like

Where Claude Design goes next

13:47we've shipped it it's gotten traction we've done like a big second release um in June like it's becoming like we've hired people for that specific team and it has more of a of a structure so it's like loose until it gets solidified down the line what's the future of cloud design I think a lot of people are very interested in it's one of your biggest launches this year um where does this go I think for me I mean uh the things that are holding back cloud design for being even better is better interaction with our other surfaces.

14:11So, you know, I was designing something or I was talking to to claude code the other day. I'm like, I want a really much more seamless like what I'm talking about the design for it, you know, interactive design back to that. I think in general that's I mean this goes back again to uh kind of unconstraining cloud like the fact that our services don't talk to each other as well as they could.

14:35I think really holds back a lot of interesting ideas around what we could do. So, I think that's one like kind of major area that we're looking at. Um, and then the other one is people like the lines between a cloud design and an app get blurrier and blurrier over time.

14:43Like I've seen people of course there's no like persistence but build like fully functional like even games which is definitely not what we design cloud design for, but you can do it. That's just HTML and JavaScript. Um, so blurring those lines even further and thinking through like what is the path from a like fully featured design that looks really well to really good to something that is maybe more like an artifact where you're actually able to go and you know persist data and share it with others and build from there.

15:09So I think that those lines get really interesting over time too. Yeah. Uh, a big part of design is having taste. Um, I actually asked Fable what Fable wants to ask you. Uh, and this this is what Fable came up with. uh you deleted almost all of bourbon to get to Instagram which is like you had a whole you know solom mo whatever thing and you went to Instagram uh what would you delete in AI or more spicy what would you delete in claude oh I like the spice um I think I mean we

Project unhip, and what he would delete

15:42have it's interesting we have a uh one of our slack channels is like project unhip which is like what is in the product right now it's and and I mean this is hard at Instagram the Instagram we some things that had like four to 5% usage you're like oh that's really not very many, but then you have like 20 features that each have four to 5% usage.

15:59It's like the classic Microsoft Word problem of like uh everybody uses some disjoint subset of the of the functionality. So that that's always the challenge. Now I think uh we're a younger product so hopefully we have less of those things like we unshipped styles I think recently where it was like used by a small percentage of people and was not really AGI pilled in a lot of ways.

16:15It was like very sort of prescriptive in the way that it worked and skills were a much better uh application something like that. So I think you have to be willing to take the primitives of like one generation of AI and like unhip them or at least like supplement them or supplant them with the next one as well.

16:30I think the biggest thing as I look at it and I've been spending some time like outside labs on some of this is like man like we're asking people to make like code versus co-work versus like chat distinctions and like one they don't interoperate well and they can't delegate to each other and two I think the average person off the street could not explain to you why those surfaces are all different.

16:48So I think deleting some of the product complexity within our our code or our product I think is a a thing that would would serve well also because then cloud can do what it needs to do and and do well. Like there's nothing more frustrating than having a co-work session where you're like great I've mapped out exactly what I want you to build and then be like can you please like create a paragraph that I can paste into cloud code like that is some 2020 you know kind of workflow there that really shouldn't exist anymore.

17:17Yeah. Um I think drawing lines on what you don't want to do and also sort of leaving room for others is interesting. Um a lot of people today is like the startup's day for AIE are obviously very sympathetically aligned to startups. Uh but there's some anxiety in the room because tomorrow Enthropic could wake up and publish some markdown files that destroy my industry.

17:31Um so

Why start a company when the labs ship this fast

17:39uh why should we not all just give up and join Enthropic? Like why bother starting any other company? Um I I mean I actually joined one of the main reasons I joined Enthropic was because I saw how much this was like you know the models weren't that good at code yet but they were getting there like how much it would unlock like whole like next generation of startups not because it was going to solve their ideation or their taste but because like it would make experimentation way simpler and and would get you to move faster and I still like really believe that and I mean it's the reality of you

18:08know uh and we saw this like Instagram like we would get the questions in investors is like, well, what happens when Google launches a photos product? It's like Google's going to launch a very googly photos product and it's going to have to be bound by the integrations that they already have and it's going to be like it's going to play to their strengths.

18:16I think that is going to be true. enough like giving advice on how to compete with anthropic I guess in a way but like it's actually not because we're also a platform which is like there's so much I think room to be like laser obsessed with your particular vertical or your industry or group of people that you know really well in a way that like none of the labs are ever going to get to that level of uh of understanding and like therefore get that kind of adoption and user love and and build that up.

18:39Now, it's definitely harder in the age where like the models can just do a lot. And so there's, you know, some of these things can be like skillified and like maybe don't need their own dedicated product, but I think it's like the hard stuff is still hard.

18:55It's like understanding the needs of people, like figuring out how you're going to reach them, uh, listening to them and iterating on them really quickly. Like it is still the case that like a group of four or five people obsessed with a problem is going to move faster than those same people at any other kind of organization that are like, you know, subject just to the complexity.

19:10I just mentioned the like the fact that we have you know a lot of different products that kind of interoperate like that's a interesting constraint that we have to work through. it's an advantage in other ways, right? So, yeah, I'm still like very long and bullish on startups and um it's just uh it papers over the fact that like writing code was never the like the limiting part.

19:31You know, maybe it was on the timeline perspective, but it was never like the thing that was going to like make or break your startup. It's really that space and user understanding. Yeah. Uh domain knowledge. Yeah. Uh today is also our day for vertical AI.

19:43uh one of our uh returning speakers and top speakers Chris Lovejoy uh was always talking about vertical AI. He was from interior in the healthcare space and then recently I was I invited him back and turned out he you guys just hired him for your healthcare efforts.

20:01Um we also our next big one is also finance. Uh you know we have a AI and finance track. You guys just had a huge finance event in New York City. Um and

Vertical AI and finance

20:10where our next uh AIE is is sort of finance focus. What are you seeing there? any you know any potential uh for cloud obviously a lot of excel excel spreadsheets yeah no I think that there's there's a lot in there too and that's like an area where uh you could see the model get clearly better at it like sort of generation to generation and there's you know there's some good sort of vertical specific uh finance startups that have like done their own um evolves which has

20:35also been interesting to to track and it's not like we're like sort of playing to the evol but it is a useful sort of barometer on like is this actually getting better um at these finance use cases I think the interesting blend that's going to happen um is this mix of again the model having the flexibility to like dive in and create just in time analyses or dashboards or workflows with like some sense of like what is the not immutable but at least like verified sort of set of data and so like uh set having all of that be totally free form I think is a recipe for confusion and is

21:06like not what most companies in the financial services space want. So finding that right uh sort of cutline where you have verifiability and audit logging and and sort of data provenence here but not in a way that constrains the kinds of applications that you can build on top I think is a lot of the art that we're seeing in that space as well.

21:24Um and I think you know if you solve it well you can you can get the best of both worlds. The hard part is a lot of the systems that were built to do the verifiability like are kind of almost by design not super flexible in terms of agentic workloads on top.

21:32So I think there's opportunity at both sides of the stack there. Yeah. Um I think I I also agree we'll be exploring that in in New York. Um the last thing I want to end on is on mental health which we don't talk about enough in technical conferences.

21:50Um you've seen a lot of hyperrowth. People are just always refreshing their timelines and it's exhausting. Um how do you advise people who are working 996 to avoid burnout? Yeah, I mean I think this is a hard one.

Burnout, 996, and staying sane

22:04I mean and it is I'm sure you all are experiencing this because you're all working in this industry like it is you know multiples more intense and things move much more quickly like at Instagram like our two things that we were thinking about was like what is Apple going to announce at WWC and is it going to like totally mess us up or boost us right so that's like once a year um or you know we have competitor launches every three or four months right and uh it is definitely not that is a topic we we do when we do our weekly all hands it's usually on Wednesdays is and we

22:34have a slide that's like the weekend AI parenthesis and it's only Wednesday and like and inevitably like some competitor has shipped a new model and like there's been like new product and maybe there's some interesting thing happening um uh on the regulation side like it things are moving really really quickly.

22:42I think the way I try to stay at least relatively sane um one is like actually carving time off and I think the topic co-founders do a good job of like saying like look like burn out if you burn out like you're kind of done and I've seen it happen unfortunately to people who I'm really close to and then it takes a long time to recover from that.

23:00Um so actually encouraging people like there's no job that is so important that you can't be offline for a couple of days. Um so I think that's like a a big key like piece in there. So like that's strongly believe um and if it is you're probably doing something wrong and you talk to somebody who could be a mentor to figure out how you can uh unblock that.

23:19Um and then I think the other one as well is like uh I love sports and like uh this is the notion of like you're never as good as like your best game and you're never as bad as your worst game. I think that's also really true. Like I know like in AI there's like the you know it's so over we're so back thing like that like if you internalize that that cycle is always going to be at play in some way.

23:45You realize like it's never that bad. Like Ben Horowitz's uh book is the hard thing about hard things uh has this chapter on like we're effed it's over and like that feeling as a startup that probably many of you have had at startups where you're like oh I can't believe this thing happened like we're never going to like recover from this.

23:55I defin we definitely had it on Instagram a couple of times and then you get through it and like that like def like defines the company when you can actually go through that. I try to remind myself and the team here even within anthropic which is like look this is a it's a fastmoving but it's also a long game and it's like we're never it's never just about today's model launch and reaction or this product launch or something else like you're playing and you're building.

24:20You just have to trust that you're building like the team and culture that is going to get through those things and have that sense of perspective. Even if perspective is saying like look three months ago we were in the similar position. Maybe it's not a year it's just a matter of months but it's still like zooming out and not taking things not letting your internal sort of like sense of self and success be so driven by the day-to-day.

24:43Yeah. Has anyone any coach or mentor said something to you that you repeat to yourself uh that gets you through the tough times? Um, I think the biggest one was uh this like uh sense of like uh if you're feeling something, it's really often the case that other people on the team are feeling it too.

24:52So, this is like advice I got from my my coach around just being like like just verbalizing emotions. Like even saying like, "Hey, I'm feeling really stressed out about this." Or, "Yeah, I'm really

Verbalizing emotions as a leader

25:08sad that we are shutting down this labs initi." I literally had this meeting a couple months ago where I was working really hard on something and I kicked off the meeting like I I'll kick it off like I'm really sad like and frustrated like I wish this thing had worked out.

25:20And I think that holds the space for other people to be like yeah I'm pissed off too or like I'm sad too and like uh I think giving that advice around like not uh yeah I think if you can get yourself to be open and vulnerable it often like lets other people verbalize that and then you could from there you could be like great what are we going to do about it like you know it's much easier to start from that place.

25:29Yeah, we actually kicked off AIE with a session from Carol Robbins who runs Touchify at Stanford. Uh, and I can't think of a better way to end encouraging people to talk about their feelings, manage their mental health, and keep shipping. Yeah.

25:46Thanks so much, Mike. Thanks for having me.