AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash

The GenAI platform team, and its three forces

00:13Good afternoon everyone. Thanks for uh coming for a post lunch uh talk. Always appreciate that. Um my name is Farup and here's my teammate Nachiket. Uh we are uh here behalf of the Door Dash Genai platform team. Um and we kind of wanted to share our eval journey.

00:31Uh it started as uh uh you know eval is another engineering thing but then it slowly we realized it evolved into a cross functional effort and we kind of want to share our story here. So what is this team? This team is a gen platform team. Uh we are a horizontal team that helps all other product teams.

00:50So product teams at Door Dash build on top of the infrastructure and the primitives that we provide. Um and we see our uh USP and the value that we provide is that we help product teams balance these three forces which is accuracy, latency and cost.

01:05Um initially we applied this in terms of models but if you think about it it also applies to agents. Um and the way we achieve this is we have uh primitives and building blocks. Um so for example we have an LLM gateway where you can easily switch between different models uh and try the latest and greatest.

01:29Uh we have an agent gateway where you can connect to tools uh and other agents uh and we help solve authentication uh agent identity and other things in a central place uh which our security team can bless. Um similarly we pair the LLM gateway with open weights models hosting.

01:45Uh, of course cost is a number one concern these days. Uh, and we uh kind of invested in open weights models uh and have seen significant impact uh already. Um, and maybe we'll talk about that in a future conference. Uh, the fourth pillar is eval and that's the part that we

Why eval became the fourth pillar

02:07would want to share today. Um when we started talking to product teams internally at Door Dash uh there were varying uh distinct needs across teams. We had a consumer discovery and shopping assistant team. Uh for those who attended Ragago talk earlier today uh you will uh see the need for session level quality judgments.

02:29um uh then the personalization ML then you needed uh a way to scale up human judgment and with multi- aent systems we needed trajectory based evals now the question is how do you cater to all these different needs under a common platform and uh as we spoke to these teams we realized like um we needed to empower the people who are the domain experts and in our case that was strategy and operations folks it as product managers uh it was even labeling partners uh and not only engineers so we kind of started

UI first, then API first, then workflow first

03:07with like okay we have to be UI first and this was the guidance we had from Andy Fang our co-founder as well um so we had UIs for non-engineers to contribute uh then we kind of evolved to also being API first so that engineers can also build and not be blocked on the central platform and they can build their own uh uh uh systems uh and Then of course with the coding agents now we have become workflow first where we kind of empower SNO and PMs to also being able to uh navigate uh the platform and uh run operations as well.

03:41Um so with that context I'll hand it off to Nachig to talk about uh how we went about delivering this. Cool. Thanks Harup. Um and thanks everyone for joining us. I know France is playing right now and I promise you this will be better than that.

03:56I'm kidding. Um so as Faroo was saying uh Evals is not just an engineering harness it is a cross

Evals as a team sport, not an engineering harness

04:05functional effort across different pillars across different uh teams uh that actually helps us add all the domain specific knowledge into our uh into the quality of the AI itself. So from your traces to your data sets uh from you know scoring mechanisms uh this is all basically a team sport.

04:21we all have to play uh and help improve the quality of AI. So going a little bit deeper into the same aspect uh we have different uh teams uh at Door Dash who help us actually improve the quality of AI. So you're going to have your strategy and operations folks who are going to set priorities, set the quality bar that you want to aim for.

04:42You're going to have your product people who are going to translate uh these requirements into rubrics workflows. You're going to have your operations teams running uh annotations. You're going to have your

Who owns which part of quality

04:58engineering teams like us uh providing APIs, telemetry, data sets, judges, all you know the the cool things. Um and combining all these together is is what a recipe is for actually making sure that you are shipping quality AI products through an eval platform.

05:19So we've tried to boil this down uh into sort of you know like a a continuous iteration loop. Uh so right from tracing uh you know having a tracing solution viewing your sessions your traces to sampling them down you uh you know to a very small uh set that you actually want to look at uh annotating these with the domain specific expertise that you bring in with the different teams I mentioned reviewing those uh then creating those golden data sets which are going to be

05:50you you know your uh golden data sets and that that you want to measure or calibrate against uh and then of course like you know monitoring this over a

The continuous loop: trace, sample, annotate, calibrate

05:58period of time and then you know rinse and repeat uh go through the whole loop again. So this is in our experience has been you know like a good sort of continuous loop uh for you know shipping quality AI at the plat on on the platform level uh we we have two surfaces uh so we have the telemetry layer uh where we have all our traces our scores uh observations that is also sort of the plane where users are able to access these traces using an MCP using an SDK uh using our APIs and then we have the workflow This is where a lot of our strat ops, our product teams operate on the platform.

06:37So this is where all the annotation tasks are set. Uh you know this is where they review their golden data sets, uh create their judges, calibrate their judges and so on. So maybe today we'll go through you know these sort of four different uh modules or pillars of our platform uh step by step.

06:55Uh so again first one uh tracing and sampling uh which is actually capturing what your agents what your LLMs are actually uh you know outputting for the lack of better words uh and actually viewing those. Now in order to also power this uh whole platform we have I think as far mentioned we have gone in an API first uh approach.

07:18Uh what that has allowed us to do is have these table APIs that actually uh you know and then you know build UIs uh on top of that. Uh so all our scores our data sets uh these are all powered by very stable APIs uh that our team owns. Uh so all your API access uh including you know like an SDK access is basically powered by this single uh plane.

Telemetry and workflow as two surfaces

07:44Um again going back uh and you know like just refreshing your memory. Uh step one capture your traces uh capture your sessions uh measure your scores. Uh then you want to start uh almost you know like adding all your judgment your context your domain knowledge uh and then calibrating your judges is what we have seen as the whole uh life cycle.

08:13Step two is on the annotation side. Uh so you obviously are capturing a lot of your uh agentic behavior, your sessions, your traces, but you actually want to see what are some places where things went well and what are some places where things did not go well.

08:21This is where you can actually titrate your your you know and actually look in inside what's actually happening uh at the session level and annotate these data sets. Um and as Surup mentioned, we have a lot of use cases. we have we we talked to multiple different teams who have uh various uh ways of annotating uh their data sets uh and it's it's almost hard for a platform team to you know build like a UI specific uh for each use case uh and and you know to give you an example uh it's usually going to be an

09:02annotator who's going to annotate these data sets so the platform team is you know in charge of the APIs we have a strategy and of person who's actually deciding what to annotate and then you have an annotator who's actually going to annotate uh your data set.

09:09So we took this approach uh everybody uh has uh you know access to coding agents uh and we actually doubled down on that API first approach. So because we had these APIs we were actually uh able to enable our statops teams to use something like a

Operators vibe coding their own annotation UIs

09:32codeex or a claw code and v code their own annotation UIs. Uh so we had different use cases. Uh I think we had a talk from Ragav before. Uh we had image annotation use cases. We had some uh you know manual testing use cases. What stood out to us was the underlying patterns were similar.

09:49So if we are are API first uh we can actually enable our our our partners to simply v code these UIs for annotation. So it's it's like a very simple example then you know of of a vibe coded UI looks pretty clean does the job uh and you get you know the annotation that you eat this is basically like a menu from a restaurant uh it's it's you know nothing crazy uh but the point I want to make here is that what helped us was to give this workflow in the hands of the operators so that they can actually build their own vcoded annotation UIs.

10:22Uh so moving on once you have these annotation UIs you obviously want to you know calibrate your your your judge prompts you obviously have some LM as a judge uh metric that you're tracking you want to now start improving that with these golden data sets u in order to do that uh you know we have a pretty simple process uh you you're going to start with you know some judge prompt take a look at you know what exactly do you want to measure from the output uh g you know have have us have something simple you're going to have your baseline scores uh where you're going to simply run those LLM

11:02judges on your traces and then you're going to have that optimization loop. Uh so we use uh the JPEA library which is a pretty commonly used library out there for prompt optimization. Uh and once you know the the iteration loop is complete uh our partner teams are happy they're going to then elevate that judge prompt as their LLM as a judge.

11:16Now even while

Calibrating judge prompts, self serve

11:24doing that uh LLM as a judge as a concept the whole prompt calibration concept might be uh straightforward to a lot of folks but it is still like a pretty new and evolving field. Uh and what we wanted to do was really reduce the friction of back and forth with an engineering team.

11:39So we tried to really remove all the complicated logic and make this into a self-s serve UI. So the screenshot that you actually see is what actually exists. uh so uh you know like a product manager or an operator is going to come to our UI.

11:47They're going to set some of these configs uh on the platform and then actually run the calibration loop themselves. So they don't have to worry about the different settings that they need to worry about what are the different uh you know tweaks that they need to do and they can actually like you know run a calibration loop using any model of their choice.

12:10I think in this example I have Gemini they can use run it using uh you know any of the claude or the openi models too. The other important piece was actually uh making this reviewable. Uh you know again a lot of this uh is a closed box where you can't really it's hard to see what's actually happening.

12:26Uh so the second piece that we built was actually giving them vis visualization and visibility into what's actually happening. So on the left you can see we and this is like one of the good examples where we saw like a significant amount of improvement in the judge prompt.

12:41uh and we actually show the you know the the previous the original system prompt and the calibrated prompt to our partners so that they are also able to gain that trust uh why as as as we build this yeah just want to add to that is this enables different configurations in different teams in some teams you have seen the strategy and operations folks own the prompt uh you have seen some teams where the product manager owns the prompt you have seen some teams where engineering owns the prompt so this gives gives the flexibility for teams to

Different teams, different prompt owners

13:16design and evolve because we are all learning. So the even the org uh design is improving and we are enabling that. Yeah, that that's a good point. I think the overall idea was to you know build something which is as self-s served as possible so that uh you know people aren't always necessarily blocked by our team helping them out.

13:31Um and then finally you know uh the quality loop in practice. you know as we've been going through this exercise we've seen a lot of improvements happening to our product as well. So you know for example we sort mentioned we started with the UIs we are you know now API and workflow first uh we're trying to reuse a lot of the existing infrastructure that already existed at Door Dash uh and that's helped us uh get a long way.

13:57Now some of uh we we we've seen obviously like you know really good results. I think a very

What it did to annotation cost

14:04good result that we we do like to call out is we actually did see a lot of reduction in the spend uh at per annotation cost as you as you all can imagine we do have you know thousands of rows that need to get annotated every week uh and it can get pretty expensive at doash scale uh and having this selfserve uh annotation platform really helped us reduce increase the velocity and reduce the cost that we were actually spending with these annotators.

14:33to to annotate the data for us. Uh obviously uh this resulted in faster loops. Uh teams were able to iterate faster. They were able to uh you know calibrate their own judges in a completely self-s served way. Uh and thus it has resulted us in in in moving with a very very high velocity.

14:54So uh finally I just wanted to you know quickly touch on this slide again uh the eight steps you know continuous loop uh which is you know you you have your traces you want to look at your traces your sessions you want to sample it down to a size which is which you are comfortable with uh you want to start annotating your data sets you really want to start uh making the data better with the human knowledge that exists and the domain knowledge that exists and then calibrate your workflows was calibrate your agents, calibrate your

15:24LLM judges with this golden data set and then repeat this whole cycle uh over a period of time to you know to ship reliably and ship with high quality. Um yeah, we have 4 minutes left. Thank you once again. I think that was the last slide. Uh thanks for attending and if there's any questions, we'd be happy to hang out after the talk or even happy to answer them now.