Agents Are Where Microservices Were in 2015 — Roberto Milev & Uday Kanagala, Navan

The microservices bandwagon, and what it taught

00:13Right. Hello everybody. Um, welcome to our talk. My name is Roberto Milv. I am the chief architect at Navan. And I have Ude here who's also part of the architecture team. uh Navan is a travel and expense uh management company and we'll share with you some of our learnings around uh how we run an AI and what have we uh discovered.

00:38So if you've been long enough u in this industry you remember that over time there are a few paradigm shifts and we all tend to jump on a bandwagon and try to kind of do things right. Last time was when we all jumped on the microservices bandwagon and out of that a lot of good things came out like container orchestration, Kubernetes, then we had service mesh uh uh circuit breakers, all of those good things but it didn't happen overnight like it took a long time.

01:08It took some time for us to learn how to do these things. So one of the quotes from there is if you can build a well structured monolith why even try to build microservices.

A reference architecture starting to crystallize

01:25Uh it kind of translates today because if you can build a single agentic loop why go in and try to build a multi- aent orchestrated system. So over time just like previously uh a reference architecture is emerging. So we have learned a few things by by doing in production we have a lot of agents lot of tokens per day being used and as I said there are a few layers that have standardized that have crystallized around what do we need to run agentic flows reliably in production runtime memory context management all around operational cross crosscutting concerns around orchestration as well.

02:11So today we'll go over some of these layers all of these layers actually and we will show kind of where the industry is what we have done what we have learned uh and and so on. So starting at the runtime layer, uh we've talked a lot and we've built a lot of services in order to scale them uh

Runtime, and agents being stateful by nature

02:38statelessly before and now we're in a new world where you know agents are stateful by nature. They need to have uh persistent sessions. They need to have isolation. Their life cycle is different than the life cycle of a traditional API service and so on.

02:50So the cloud providers have jumped in and try to fill this gap. Um you know uh AWS, GCP, Azure, they all have a some incarnation of a aentic runtime. If you scan the QR code for this slide and for the following slides, you will see a comparison of some of the features and how different cloud providers try to try to uh approach this.

03:17At Navan we run everything on AWS. AWS has an agent core runtime. We heavily use that but we have filled some gaps around that like the uh session persistence and rehydration is something that we have built and we also run a bunch of other uh bunch of

Memory, from RAG to episodic

03:45SDKs for writing agents and part of um these runtimes is typically they are framework agnostic although they all prefer their native framework in a way. Um the next uh layer in the stack is around memory. Um we started with rag. Rag was kind of big thing for a while.

04:04Uh we were kind of driven to that out of necessity because you cannot fit an unlimited amount of context into an agent. And over time u all of these cloud providers and the industry has implemented a pipeline where memory is kind of automatically generated by following a workflow of ingestion, extraction and then consolidation and retrieval.

04:28And there are parts of rack that are built in things like a long-term memory that inherently has some semantic characteristics. But memories build up over time from short-term conversational memory to long-term memory that you kind of manage yourself.

04:44Uh then episodic memories about uh kind of instances that worked well and didn't work well uh and so on. We at Navan again being AWS shop um utilize their agent core memory but we are also kind of doing it in a way that uh matches matches our our use case.

05:14And then the next thing is context management. You know, it's a hot topic. It was a hot topic and it's still a hot topic. Context windows are growing bigger, but there's never enough context. Or if there is too much context, again, agents struggle with that because you lose focus and so on.

05:31Um, what we found working is that uh focusing on skills as a unit of context and I'll explain what I mean by that. Uh we look at skills as both having context meaning instructions and u setup about a certain domain or a task and there's also the uh the second part of the skill which is the tool execution and you know the agentic part and we compose context dynamically out of skills that we uh use

Skills as the unit of context

06:04as units of works that are pluggable that we can test independently and that we and reuse. So for example when we are we have an agent we have skills that are uh that are specific to a domain and based of that we compose them and we rely on the u uh you know the progressive disclosure which is a feature of the skills itself to start with a limited scope of context and then uh expand by included metad data uh further down the the line.

06:39Uh I'll hand it over to Ube U now to kind of walk us through the rest of this. Thanks Rud. Right. Uh can I have a quick show of hands? Um here who have who had built an agent u which failed halfway through multi- 20 step or 30 step process and be able to figure out quickly uh or reason about why the agent failed.

07:05So again logs we've generally been traditionally with microservices we all are familiar with logs there's logs out there and then we go check out the logs but this changes everything the moment we switch to agents agents output a lot of thinking there's too much to consume

Why logs stop working for agents

07:22so that's not the right way to do it right so traditionally that was the way but our thought has to be changed right now in the in the way the claude as an example when we take claim for an agent there's hooks and we can intercept everything that claude as an agent that does at that level.

07:39So what kind of tool it calls right what kind of decision it's making. So before pre-tool and post tool call or a pre session or a post session. So all of that are a point point in time for us to intercept and make a decision and either block to do a blocking operation or to log a metric or emit a metric.

08:00Right? So this is a critical u place where we can emit traces. At non we use uh one of our provider brain trust to emit these uh traces and through these traces we should be able to figure out the spans the traces and at what point in time where the agent is stuck which gives much more confidence into um how we

Hooks, traces, and confidence scores

08:24operate and build the agent. This is a dayto operational challenge. Um building agent these days there's so many frameworks but how do you navigate building and operating an agent later is a primary concern now and moreover uh the reasoning chain uh the thought process and uh critical signals that we emit um here as part of the trace uh captures we emit a few primary signals uh here what is the current goal the agent is going through uh the reasons behind its operations and the belief status and uh the tool calls that it's making so this kind of gives us a judgment pointers um uh in the traces

09:03and when we make uh when the agent makes a decision um there is a confidence score how confident it is when it makes this judgment right so whether there are multiple paths that it leads to this uh choice or whether this is an inferred answer.

09:18So basically these are signals that gives us uh confidence later to review if this is an inferred answer there could be a human in the loop to guide through uh and tweak the agent to perform a little better

Testing something nondeterministic

09:32again. Um can I have a raise of hands again to see uh how confident are you like 100% confident in testing pipelines with your agents right so this is one of the other um critical aspect uh today um because agents are nondeterministic we've all been used to program and write much more deterministic flows and we know how it works the when I ask an engineer engineer can come and tell me how this the algorithm the sequence of operations everything is programmed in

10:06our mind everything is expectations but now the agents come into a nondeterministic way and how do we test them right so that is very criticality here and yeah we are also struggling uh we've uh started doing building agents we the day operations was challenging and then we failed uh in lot of steps how do we course correct the moment we change something something else broke breaks right so how do we do that um one one approach that we took uh this is from uh research papers around the traje

10:36concept of trajectory like in a multi-step uh orchestration when an agent makes uh 30 steps or decisions to

Scoring trajectories instead of outputs

10:44make to reach to a goal if that is a program or that's a different story but this is not a program this is nondeterministic way of it makes up its own uh steps every time uh differently so how can we chart a deterministic graph have here. Is it possible?

10:58No. Can we have a trajectory of it starting from an end to a goal and then see how much how far it went in the trajectory and how far it went from the source to the destination is what we can compute to evaluate the efficiency or the completeness of the um agent uh agent evaluation.

11:26So we we heavily rely on um trajectory valves. Um and u this there are few other signals uh as I briefly spoke around uh in the previous slide around the inferred signal. Um if the answer is from an in inferred answer uh how can we um loop that into u and make a signals around how can we classify that this is a regression and make fixes towards the agent.

11:48Right. So the next is the um guardrails. Um where is this the one? Yeah. So guardrails and authorization. Um this is a critical this plays a critical role in enterprise AI. Lot of information is being piped to models. Um there could be sensitive information that goes into it uh without our knowledge and we as uh a leaders how can we put in this governance layer um to stop this um is very uh critical here and and the concept of uh authentication and authorization um is taking up a different approach here.

12:45Um traditionally we've seen um a user or a service account but now what is an agent? agent can be acting as on behalf of users. There is so much of things uh so many of use cases there. Hey, book me a flight whenever it's cheaper than $200, right?

13:01So, we just tell this

Who actually bought the flight

13:08assertion and then agent go figures out and does this action on behalf of me. So, is it me making this purchase or is it agent me making on behalf of me? So, there is agent acts as a on behalf of user or agent use a service account as well.

13:20So the line is being blurred here and we need to make fine grain authorization decisions here and the policy layer. Um that's where the guards and authentication authorization plays a critical role. Um and in Navan uh what we employ here is um before every tool call pre-tool and post tool we have this guardrails uh to check and block u and make uh informed decisions and this uh single agent versus multi- aent again this is kind of a orchestration wars you can think of whether to build a single agent or a multi- aent um again as Roboto briefly hinted If you can't perfect and build um single

14:09agent, why go towards uh multi- aent right? So learn from our uh failures, experiences and build towards that. At

One master agent, or many

14:18Navan um yeah what the approach that we have taken is um single master uh and then we adopted uh subsklls um there are sub agents within it. Um so it's a single agent that can um progressively load the skills and understand decisively what needs to be loaded into the context and then um make this um navigation um through the use case.

14:44But there are other patterns that are also emerging. There are different class of use cases here. One is um agent to agent communication. So there are if you take a large scale organization and there are so many of these teams that are uh that are acting as a boundaries and they don't talk to each other let's say how do we communicate there are two agents on either of the side right how do we do it so there is a protocol which can uh help us establish the contracts in terms of skills and we can use A2A as a protocol there uh which kind of uh is a boundary between the teams Yeah.

15:19Over to you.

What is solved, and what is not

15:29All right. So, as we went through the ST, it's obvious that um some components of the stack are in a more mature state and we already have good answers for them. As I said, the runtime I think it's pretty much solved. We are so advanced in orchestration and we are running LLMs in kind of a a very uh brute force way.

15:46So scaling is not a not a problem. Also memory I think uh as uh the frontier LLMs get better and as our practices get better we will uh find a way to cover the majority of the use cases and there is good m maturity around the cloud providers.

16:07Uh MCP has emerged as a de facto protocol and tool calling is now a feature that everybody supports. So we are seeing some industry convergence around that as well and MCP as a standard is also evolving now it's becoming stateless. It's uh we are reaching a point where kind of we know how to invoke services and and and tools with agents.

16:30Uh in some areas things are happening but you know there's still a lot of unknown around observability. There is a push towards hotel but does really work for aentic calls. Yeah, you can make it work as UD was saying. Um also we are getting more comfortable around the the testing patterns.

16:57It's very hard to test but we have found a way to give customers um quality experiences even with the unreliability of agentic system. And I think that's kind of uh getting in a in a state that is uh that is more better defined. Orchestration is another one um where you know we have a p we have patterns uh we can build you know bigger agents smaller agents.

17:21Uh as we said previously probably the right answer is to not overengineer. Uh so we are learning there and and and apparent as school thought is also emerging. Uh we're we're all struggling with and the previous talk was about this for the developer um AI assisted development perspective but also we're seeing these issues from our production

Cost, replay, and emerging standards

17:53agents. It's very hard to predict cost and it's very hard to manage cost uh and put guard rails and solve this in a way where there is reliable maybe fallback or have agents be uh using cheaper models for certain tasks. Uh this is all driven by kind of the big AI vendors who I think their interest is for us all to spend more tokens.

18:16um replay and debugging who they talked about that that's also a big big issue. It's very hard to understand but I think this is also something that that is going to be solved because we can now use agents to uh get over the cognitive overload of trying to debug what they do.

18:40And then standards um standards are emerging uh by you know the community hotel as I mentioned agent to agent is young it's kind of pushed by certain vendors but I think over time we will we will get uh there uh with all of this said you know we know what we need and it's up to us to go ahead and build it.

19:03Thank you everybody.