How We Got LLMs to Recommend Our Open Source Library — Christopher Burns, Inth

Not a scientist, just hacking on it

00:12the talk title. We'll see if it lines up by the end of it, but when we put this talk title in, just to be honest with you, so much changes in like three days at this point. We'll see how it goes. So yeah, the whole point of it was that how I got LLMs to understand my open source library and what I did to do it well.

00:30Is it some kind of scientific background? Am I from a lab? No. That's my slidey clicky thing's not working. So I just like to say again, I'm just like you. I'm just this side of the stage. I've just hacked hacking it together, figuring out what is useful, what is token efficient, these kind of things.

00:59And again, I am British. Please don't think my accent makes me an expert. So for quick context, I'm Christopher Burns. I'm the founder of INT. I created a open-source cookie banner library called C15T. That really annoying thing on the internet.

01:15That is me. Uh I spoke at Nexcom uh after it started taking off and it had 1.2,000 downloads at the time. Now it's closer to two million in terms of like statistics. So we just check that you know this is not theoretical. This is actual uh something that is succeeding.

01:37We have 3 million npm downloads, 4.5 uh 4.5 45% month-on-month growth, 2.8,000 8,000 websites using in production from mintlifi to zed to in physical and the whole concept of this talk was uh it goes back to we were doing all these things to make our library more efficient.

02:04You know we were batting upwards compared to every other tool. Every other tool was built for marketers and lawyers. We were built for the developer. So we had to make sure we had a very good developer experience and we had an onboarding form that said how did you hear about this and we started to get spikes that from April the 13th you

The spike, and where the inbound came from

02:27know now it is our number one source of inbound is claude chat codeex that is chachi gemini recommending us in I like to think of this as you know the iceberg You know, we start with the top of C15T and there's many many tools that go into it from, you know, LLMs.txt to sitemaps to RSS fees to robot.txts.

02:56So many micro optimizations that you can do from old methods of running the internet to new methods and uh how many of you have you know made these kind of tools? How many of you have really put simply said hey agents we need to this to be done and yeah it said we should install this library and you've gone okay raise your hands how many people have done this pretty much most people that's a lot of hands so what's really funny is that we went from wizards installing our software to agents installing them.

03:37And I just went through Y Combinator. And what's really interesting is if you know who these two people are, these are the co-founders of Stripe, the Collison Brothers. And they had a really classic saying of like a Collison Brothers install, and they would hand you their laptop and they would install Stripe.

04:03These days, it's kind of like just a prompt. Being in Y Combinator, we just gave people a prompt. And really what that means is that our very good developer experience primitives are now hitting agent primitives.

No single fix, so they built a docs pipeline

04:20So as we was pulling all these things together, there is no one tool that fixes everything. I like to think about these problems like, you know, Batman's utility belt. loads of really small things targeted in different areas to get it done.

04:38And we built all of these things into C15T because we wanted C15T to be the best developer framework in this tool. Think of it like Stripe Docs. And as we was building more and more tools, more and more documentation websites, we actually started abstracting these tools into a side quest that we call lead type.

04:54So all of the things that we're going to talk about now are things that we have already to solved with this open-source framework. We have our friends at other developer companies implementing it and seeing similar results about how to like optimize for the agent experience.

05:19So again, this isn't a magic SEO tool. It's actually a very nonsexy title, but it's a framework neutral docs pipeline. complex, but really all it basically does is take your MDX files, you run lead type generate, and it will spit out everything for um optimized agent experience for your websites.

05:45And the rest of this talk is going to look a bit like a Buzzfeed uh list to put simply of these problems because

Write llms.txt by hand, not generated

05:52again, not everybody knows even how to put an LLM.txt on their website. So you know that comes to the first problem of if your docs have hundreds of pages and how can it navigate them to find the right questions. The first solution is obviously in llms.txt.

06:14What we found in our research is that it's much better not to just generate this. It is much better to write your LLMs.txt from hand. Obviously our tool wraps it but write it as you are trying to get the answers across to the LLMs for about 40 good lines beats a thousand lines of noise from our testing and that comes to the second issue of agents don't know how to browse they know how to fetch so you then need the second part of the solution of the LLM's full again think of this as a site map where It takes the actual uh page and the links and a short

06:59uh description of what each page is for the LLMs to reference. Again, most people have heard these two solutions.

Ship markdown instead of HTML

07:07But where things are starting to get very complicated and we're seeing a lot of optimizations right now is that HTML is expensive and why can't we just ship markdown to the agents? And we can and you have seen that everybody has started creating twin MDs.

07:28So that's taking the normal website such as Nex.js quick start and then having a MD on the end of it. And when you load that it goes to the markdown version. But what's really important here and it's really worth noting is this line at the bottom.

07:43If you look at all the best documentation websites, Mintify, Versel, C15T, pat myself on the back. Um, they all have this in the header. This is saying to the agents whenever they visit the website that there is an alternative version of this in mark markdown.

08:05Again, who actually supports it? Don't ask me. Perplexity, some of the agents, it's all up in the air. And then the second thing as well is that taking the MDs you need to make sure that they're available through multiple methods. So one of them is like the MD.

08:26So as you like copy it to an agent you say MD. Another one is just taking the normal um link and then adding a uh redirect into your like your next.js JS config so that if it detects an agent has the header of accepting markdown instead of returning the HTML it will return the markdown.

08:55And then the third one is that not all agents can append header tags. So there's also a URL query of mode equals agent. So they're the ones that pretty much everybody knows. Um, and it's pretty basic internet knowledge at this point. Um, but one of the really interesting ones is where we're going next and our tooling is also helping this is that an agent can't ask your website anything.

Web MCP: letting an agent ask your docs

09:28So we need to think about the web MCP and this is still very early but our tool is already um exposing three different tools to web MCP search docs, get pages and ask docs. Again um our library lead type is pulling all of that context together so an agent can easily ask it the right questions.

09:52I think we'll even see a future where communication happens over email and there's companies in San Francisco building that today. But this is actually the most interesting one and I think the most important one that anybody who has any

Agents never visit your site, they read node_modules

10:12type of developer module surface, npm modules, cargo, python, whatever is that the uncomfortable truth is that coding agents are actually never visiting the website if you have a library. They're actually visiting the node modules. They read the repo and they read the node modules.

10:28They they have previous stale training data and they're trying to work it out on what it can do from the the compiled source. So again following what people like Versel are doing and people who are thought leaders in this industry is that we take the bundled markdown documents and then we also put them in the node modules with an agents.md file and the agents.mmd file basically says if you've got a problem if you've got a question all the documents are here gp And we actually see that this has surprisingly real effects.

11:16We can see that between many different models almost 50% token saving on instead of trying to search the web find the right tools pulling the markdown files from your codebase. So if you have a library that's forever changing then having the node modules built in is a very effective solution.

11:43This is also working without any skills but if you want as well you can add skills to it to say look at the node modules and go from there. And again just uh doubling down into this point looking at like the agents.mmd file you can say like when working with z15t nextjs library read the bundles and verify that they match and go from there.

12:11So that's really like how we've done it. I don't want to say this is like prescriptive that I know the answers. If you have documentation websites or if you have any type of markdown, if you're running your own blog, you know, I've been using our package as well on our marketing website.

12:23Every part of our marketing website also has a markdown file. It can be something that's used for many things. We're currently just um most people are just using it for documentation, but you can literally run it and it will pull out all of these extra files.

12:48And one of the big things was when I put this talk together, you know, we were seeing the results that Claude was

Testing whether your site is agent ready

12:56recommending, but there was not really any like test suites yet or test harnesses on like is your site agent ready and Cloudflare brought one of them out. But my favorite is actually one called Aura AI. Um, this is brand new and it tests a lot.

13:11I'm happy to show our score of 59 because it's constantly changing. Three weeks ago, it was a lot higher. And again, this is a forever changing area. So, aura.ai, put in your website, and it will start giving you recommendations. It's forever changing.

13:34Again, we can just stay on top of it. And yeah, this is like one of my final slides is that the SL the the market, agents, LLMs, everything is forever changing. There is no such thing as perfection. When I started making these slides, I got so caught up of like everyone expects me to be the expert here, but I've just been hacking on this problem a little more than you guys have so far.

13:59So, never get caught with being perfect. Every small little increase really does matter. Every small little thing you add really does matter. Thank you so much. You can find me on X burn Chris and LinkedIn and everywhere. I think we have time for one or two questions.

14:21Yeah, of course. So if you were building um we're a website agency. We work with a lot of startups building like their own websites.

Q&A: where to start on a plain website

14:40If you were just building a website, not necessarily like developer tool, but just a website to be found. Which of these methods like would you concentrate on if you're starting from scratch? Yeah, I think the most important ones and we're starting to see this more and more is trying to provide a MD file for every single page.

14:59A lot of CMSs are not built in this way. Um, and we see this optimization happening more and more where I didn't put in this slide, but we're seeing more and more websites being visited by agents instead of real humans. So in terms of even like trying to be proactive and token efficient, you should provide a markdown file if you can.

15:24Again, a lot of CMS's are not built this way. I actually built my own CMS. Uh my name is Chris and I built Chris CMS, short for Christmas. It's a whole it's a whole thing my team wishes I never built, but it does work and it does bring this like token efficiency uh up.

15:37So yeah, I would say llm.txt txt is your first shout. Uh, llms.txt full full.txt. Second, if you if you can just do them manually, say you're not even working on systems that have uh markdown, I still recommend them. Um, but you can always get creative with creating these um files on the on on the go.