The Death of Developer Advocates — Stephanie Jarmak, Sourcegraph

An astronomer, an agent advocate, and a borrowed title

00:12Hi everyone. Sorry for the start with technical difficulties and all of that. Uh we made it to the end of this track. Super exciting. Thank you everybody for sticking it out this long. Um are there any developer advocates or Devril people in the audience?

00:20raise your hand. Yeah. Okay. So, did you come to like throw tomatoes at me because I'm talking about the debt now. Okay. So, it's not going to be all doom and gloom like that. Um, a bit of like backstory in this. Um, I'm a research scientist.

00:38So, last year I was an astronomer. Um, and I just sort of like wound up. I didn't know what GTM was or any of that. I sort of wound up in this. Um, and I submitted like a bunch of boring sciency eval talks that were unceremoniously I assume thrown into the trash uh, for this conference.

00:53But my manager who is a developer advocate, he put in, you know, the de the death of developer advocates, which is, you know, appropriately buzzwordy and hypy and so, so that was great. But his title is developer advocate. So it didn't really necessarily make as much sense for him to be coming up here and giving his eulogy.

01:17So, we brainstormed like maybe I would dress up as like a robot and like a maul him and attack him on the stage or something like that. Um, but then it just like logistically it was going to be hard to do that. Uh, so he just went on vacation.

01:27Uh so I'm here uh as the agent advocate uh to talk about this sort of like new role and uh try to advocate for it and uh convince all of you that we should all be agent advocates to help uh in this new era. So uh zooming out a little bit and going back uh in time a bit because uh I was trying to talk about developer advocates to somebody at the conference yesterday and their eyes like glazed over.

01:51They had no idea what I was talking about. So just to sort of talk about what this thing is that I'm saying is dead. Uh so back in the 80s right it was called like software evangelism. Uh where one would go forth and speak the good word of the product and bring it out there.

02:06But then fast forward to the 2010s or so, that's when developer advoc advocacy um started to become a thing where now instead of having this single trajectory of um the communication pathway, now it's a feedback loop and a two-way street where you have these people with very deep empathy for developers who understand them and speak their language and could understand um what their needs were um and then bring that back uh to the product.

02:36And then um these developers right fast forward even more they uh have so much influence within their company and basically become these like kings makers uh and so developer experience became a very important aspect of the goto market sort of

What changes when developers stop working alone

02:51strategy. Um but now in 2026 uh the developers are no longer working alone and what it means to be a developer is completely changing. Um and so our role right as developer advocates um and developer in developer relations we're relating to developers and so as the role of developer is fundamentally changing so must then does the role of the developer advocate.

03:08Um so in this slide I'm just kind of talking about the other users right so what's happening uh with Devril uh outside of the agent so most of the talk is going to be talking about the agent as a user but I also did want to bring up right that engineers they're becoming like these orchestrators of these fleets of uh agents um babysitters and whatnot of these things um in their job like all of the job postings and whatnot there's language is continuously changing right they're um expected to have this AI fluency Um and at the same time there's also you

03:49know people like me like uh non engineers right I was a research scientist I had like zero commits on GitHub last year and now I have 12,000 and I'm like an open source maintainer for multi- aent orchestration framework like we have so much like capability now with all of these agents and now anybody with these agents can use dev tools essentially so you have this whole other persona in ICP uh to potentially be relating to and um having empathy with when you're they're using your product.

The agent as a user, and as a recommender

04:19So let's talk about now this whole new user that we have in the form of an agent. Um so an agent is somewhat unique, right? In the sense that it is both the user of your tool in a very similar way to the developer. It's going out and reading your docs, but it's just reading them differently because it's a machine.

04:36Um you know, it's calling API. it's encount it's has its own frustrations with how it's encountering errors and recovering from them right but then it's also a recommener of your tools um but somewhat similar right to developers in the way that they are also recommenders of your tools in a more organic bottomup way um so the whole you know basis for devril right is to encourage that bottom-up adoption but now the adoption the recommendation system a lot of it's being driven by the

05:03agent itself that is either you know maybe servicing your product directly through like chatgbt or cloud like directly in a Q&A sort of environment or it's as we had heard like in some of the previous talks where the speaker asked folks like how many of you have just let your agent install a library for you and like there were many hands went up right so there's this like recommener of tools where basically it's just installing these like frameworks and things um directly and embedding them into the workflow um and sort of working with the developer um in that taste.

Benchmarking agents with and without your tool

05:36So I know it's late for numbers. You don't have to read them or anything like that. Um so I have a couple different concrete examples for measuring these seats, right? Because I am a data science scientist nerd person. Um so one of my first projects when I was uh working on this um uh when I became an agent advocate was to build um a benchmark called codec scale bench.

05:51And so I developed hundreds of tasks that were reflective of the software development life cycle. And I basically unleashed these agents with and without um our products tooling. So I work at source graph and we have a code navigation MCP tool.

06:06Um and the point of that was to understand okay h how is our tool helping the agent do the work that it's you know going to be doing. Um and when it isn't working well why isn't it working well so that we can then go in and actually fix that.

06:22Um, so I have thousands and thousands of these traces and as we have heard in like the previous talks like now we have these amazing logs of data for like these really tight feedback loops where you can see exactly where it's breaking down and then go in and fix it.

06:30Uh, so this one specific example here was um when I was looking at how it was like using a read tool um and the model had these

A whole turn burned on a guessed parameter

06:46expectations based off of its like biases from how it from its training data of what it expected for a particular um command um that would be available within the tool and there's nothing in our description uh that would have like led it to believe otherwise.

06:59So it tried to use like reline instead of start line or something like that and then it ended up failing but then at least the error told it why it failed. So it was like okay that that was a good part of it. So it was able to fix itself but then it's burning right an entire turn just failing when you could just go in and fix that um aspect of like how it's interacting with the tool.

07:15And this is really important, right, to gather that feedback um and understand the friction that like now your new agent user is having with your tool because it's the way that um different organizations are going to be evaluating your tool, right?

07:23In terms of not just is it working well, but like how many tokens is the agent dealing with to work with your tool and how fast is it? Um so this is you know really an important aspect of the role is to measure um how these users are using it.

07:40The other side of it um is like the recommendation

Generative engine optimization, and a 65 to zero gap

07:47layer, right? So the uh GEO instead of SEO, so the generative engine optimization um and I didn't mention it before, but in the previous slide, um I had a GitHub repo. Like there's two different toy projects that I put together. At the end of the talk, there's like a QR code with a link that you can send your agent to to like have access to all this.

08:04So don't worry about like taking screenshots or anything. All of all of the data will be released to you. Um, so anyway, back to this. Um, I set up a little experiment, right, to see how, uh, these different chat bots and agents and whatnot were recommending our product or like mentioning it at all.

08:26Um, and so there's a, you know, process to that because you have you want to understand like what is your ICP actually doing when you would want your product to be surfaced. So there was a bit of a gap that I found. Um if I had designed some of these prompts around somebody who like was actively shopping for this sort of code intelligence sort of tooling and doing a comparative sort of thing, then our product was ending up being recommended like 65% of the time.

08:54Um, but what I found was arguably like the more typical use case and where we'd want to be showing up for people when they're encountering a specific pain or have a specific need where our product could serve them better. Uh, zero mentions, right?

09:02So, in this particular instance, um, I put in a prompt that was like, we keep breaking downstream services when we change shared libraries because we can't see all the consumers. And you know our one uh part of our product is being able to have this observability layer to like see across all the repos.

09:18So we'd want uh some level of like attribution or recognition from um an agent to say hey you could use something like this but instead it said uh you could just have your developers make a wiki page or something. Um but what this you know we wouldn't know that without running these sorts of experiments um and getting this sort of data.

09:42So what this leads to is like then you can have a hypothesis of okay maybe the messaging that we're putting out there isn't uh attributing some of these pains and use cases clearly enough for the agents to be picking it up. So we have uh like a content campaign in the works to um make changes to our website and then we can directly measure h whether that has like an actual lift and not necessarily in the form of like anything that was baked into the training data but then how uh the agents that are using those like web

10:14search tool calls how they are then interpreting um the information about your product. So, you know, there are just some um different ways that you could think about guiding the agent um to help

Why stale content compounds against you

10:29support like the surfacing the discoverability of your product and this user finding it um at their moment of need, right? Um so, for example, um this whole field is moving so fast. Uh so, I mean training data is like always going to be stale.

10:39Actually in the um GEO pilot study that I did the data that I was showing there that was using cloud set 4 it's very old um obviously and I just today this afternoon ran it with 4.6 six thinking that okay surely it's gonna it's going to be better it's going to know like improved information about our product but uh so in the previous model it kept pitching Kodi which was like one of our older products um but if I when I uh ran it again it it pitched Cody even more right because like now you have all of these like old models like uh outputting content that then is

11:19like compounding in the internet so you have to figure out like how to bury all of that uh noise with your true signal. Um, and the way that some folks are working on that is, as we've heard from other people, like these LLMs.txt uh, sort of pages, right?

11:34So, you have more authoritative sources of truth that you're hoping to direct the agent to. Uh, but they still need to be using the tools and using real-time um, information and providence to be able to give accurate answers about your product.

11:40Um, you also want to give like the agent something to quote, right? that they they want to bring something that they can really sell to the to the user, right? So, you want current examples and keep everything up to date. Like even if your stuff hasn't changed in two years, which would be shocking.

12:02Um even if it hasn't, like keep everything up to date and fresh because um that you know part of that is how they have their relevance algorithm and they also really really like charts and um FAQs and things like that. And you also want to make sure your product is where the agents are, right?

12:10You're

Be where the agents are, and cut the friction

12:18going to market. So go go go to agent market, right? So make sure you're in the marketplace and the MCP registries everywhere that you uh would expect an agent to be able to easily find you and also make sure that uh you know that whole you reduce as much friction as possible for an agent or and developer to go from finding out about your tool to embedding it in their workflow.

12:40Because if an agent realizes your tool requires like three different demos and emailing sales reps and stuff, they're never going to say, "Hey user, like here's what you should do, but FYI, you're going to have to do all this other stuff." It's just like not going to happen.

12:47Um and then also make sure that you are covering that those pains, right? Um because that's how a user is going to be mo like in their time of need, right? Um that's going to be the best opportunity for your product and your um service, right?

13:02To be surfaced to them. And so you want to make sure that there's enough content out there on the internet uh for the agent to like be aware of that and make those connections for you. Um and so right there's this like ongoing question of what even the heck is Devril and advocacy and now now this agent advocacy thing, right?

13:26Um it's like where does it fit? Where does it go? Like is it engineering? Is it product? Is it marketing? It's like yeah yes yes it's all of those things. Um, and with the rise of agents, it hasn't gotten any clearer, right? Those seams haven't gotten any clearer.

13:41If anything though, everybody's role with across the organization has gotten fuzzier. So, that actually helps in a lot of ways. Um, and but you can sort of split it up and think about it in terms of like these different flavors, right? And you can mix and match depending on whatever skills and abilities um various um employees have within your organization and whatever the product needs at a given time.

13:58Uh so you have like the engineering flavor, right? And those are folks that are partnering directly with the um engineering team to make these interfaces for how the agent is talking to your product like through the MCP server and building out these evals and the instrumentation.

14:15Then you have the product flavor. So those are folks that are going to own the endtoend agentic experience, right? And so translating these uh evals to um to bring it to the product team and like having the agent experience rubrics um and how they're um encountering all of that content.

14:23Um and then you have the marketing flavor, right? And that should be the folks that are um really owning that pipe genen and how the agents are like entering the funnel and finding out about your product and then bringing the developers along with them by surfacing those recommendations.

14:48So I know I you know said the death right of developer advocates um but the core right of devril still holds. It's just you have a change in your audience.

What still holds: enablement, community, feedback

15:00So it's still extremely important to do enablements, right? It's just the type of enablement is a bit different. You're educating developers now who are have a completely different type of job where they're orchestrating these fleets of agents and you're also educating agents, right?

15:16So you're having to put out content that is machine readable, has like agent friendly APIs, all of these things that make it as easy as possible to use your product both for human developers and for the agents that they're using. And community is also more important than ever, right?

15:25Um having that human-to- human connection um where developers can come um and uh bring their agents also into the loop, right? So that's another component um that needs to be considered uh when you're building these different communities because there's all these questions, right, of privacy and like data concern as well.

15:47if people are like bringing their claws and whatnot like into the discord and they're like recording all the conversations and everything like this just like a new thing that you have to think of as a community builder and then there's the feedback loop.

15:56So you're still uh responsible for bringing the voice of the developer who's using the agents back to the organization but then you can also uh basically spin up like thousands of these agents to perform experiments on them and experiments that you can't really like do as easily with the developers who don't want to maybe talk to you that much.

16:11Um and then credibility, right? So you need to be earning credibility both from human developers um so like don't like not using claude slop at them right then tell your AES to stop that as well nobody everybody knows what it is and nobody likes it um and but then credibility like actually claude loves its own slop uh for whatever reason so there's a bias right from agents of their own content so whenever you're making like agent-f facing content as long as it's structured you can have as many m dashes and whatever as as it wants.

16:51Um, but it's just a completely different sort of uh credibility landscape, humans versus agents. So, what I'm advocating for here, right, is like building out a curb cut. So, curb cuts were built for wheelchairs, like built for a specific user to use

The curb cut argument

17:06them. Um, but now everybody, you know, benefits from that, right? Anybody with wheels, right? Strollers and um suitcases and all of those things. So my argument is that by serving the uh agent uh the human path gets clearer too. There's just you know there's just one more user uh in the room now.

17:20Um but they are still serving the human on the other end. And we're all working together on this. Um so for you know Devril one quick thing that you could do like right away is point a coding agent at your docs and then looking through that uh transcript and start developing your agent experience report.

17:29And then if you're more on the GTM side, um start like developing some of these uh experiments with the GEO, um putting together those prompts and looking at the mentions versus recommendations. Um and I made this whole talk uh agent legible, right?

17:45So there's a QR code there. Um as well as a couple different toy repos that have some templates for you to get started. And that's it.