Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind

Agents that behave like a wrapper around the search bar

00:13Okay, good morning folks. Thank you for showing up. Uh, I'm Nidi. I'm a product person at Google DeepMine. And today I'll be talking about uh multimodal collaborative agents. Typically, these are agents that work with fuzzy intent. intent when the intent when it's not clear, the user hasn't gotten the right keywords to specify, and when they come in for a shopping intent, how can you have the agents still guide the user towards their goal with high agency execution, proactive elicitation, and a lot of handholding?

00:42The frameworks that we'll be discussing today uh are grounded in shopping or commerce because I wanted to show you a few patterns that are much easier to see in commerce, but they are quite applicable in other consumer vertical as well. finance, education or whatever you guys work on.

00:59Um, yeah. So before we even start into the frameworks, I wanted to discuss about why this is an existing problem. Currently, a lot of the agents that we have act more like a wrapper to the search bar. They assume that the user

The articulation gap: users arrive with a vibe

01:13has a well- definfined intent, has the right keywords, already knows what they're looking for, and so when they come in, they just have the right vocabulary to interact with the agent. However, there is quite a huge articulation gap when the users come in.

01:29Rarely they have their intent well formed. Rather, they kind of have a fuzzy feeling or a vibe when they are kind of looking to shop. So, the agent needs to play quite a huge role in handholding them. They need to work with the user in first understanding their preferences or even elicitating these preferences more proactively and then helping the user uh kind of show them different possibilities of what could be possible that they can shop for and then eventually move towards uh kind of recommendations that work with these constraints that they have in mind.

02:01So what we're going to be discussing is this kind of a flywheel or a loop that goes from very very fuzzy intent all the

The loop: discovery, research, response

02:11way towards uh kind of achieving the user's goal. So this is how the shopping loop looks like right now. The very first thing that the agent prepares for when the user comes in is what we call the discovery phase. This is where the you where the agent takes all the different uh contextual information that they have about the user.

02:23This could be past conversations, what the user has specified in their query. It could be information present in their personal context or even the references that the user have provided and then comes up with a collaborative strategy. And we'll be going into the details of this, but it comes up with a collaborative strategy on what more does the agent need to elicitate back from the user in order to help get the intent in a better shape to get clarity on what exactly the user is looking for.

02:49And then it moves to the second phase which is what we call the research phase which is again a two-step process. First is where it learns what is the best way to elicitate this preference. So for example a lot of times the text the textbased elicitation may not be the best way to get the preferences from the user because sometimes the user don't know what they're looking for.

03:12So then how can the agent start being more creative in terms of elicitating these preferences? Can they start using some kind of visual references or visual visual inspiration boards for grounding and speaking a common language with the user?

03:29And then the second phase to this is also coming up uh going into the background and doing the heavy lifting for the user. So taking the burden away from the user to describing what they want and rather going into the background and doing all the comparisons, trade-offs, uh summarization of all the information that they're looking for and coming back with the right set of potential options for the user.

03:44And then the third phase once um the user and the agent is ready to go into some kind of a uh is ready to go into the last phase which is where um the agent is ready to give out a response. This is where the agent needs to start adapting the response in a way that is most useful for the user.

04:01So typically a lot of systems fail here where they just give out a text heavy response. What the agent should rather be doing is adapting to the query that the user had in mind. So should it be using uh some kind of a bulleted list? Should it be using comparison tables?

04:22Should it be using um visual boards? So, this is where the there's adaptive response happening as well where the uh agent starts to develop uh more more of a uh smarter uh presentation skills uh for the user to really find the answer that they're looking for.

04:36So, we'll be diving into all these topics as I u kind of work through the presentation. So, first and foremost, we're looking at discovery. Like I was mentioning, this is where uh the the the agent needs to

Building a working state from images and context

04:50remember what exactly matters. Um you the agent starts to look at the context across a bunch of different signals. It starts to look at past conversations. It looks at some of the reference images or reference links that the user might have provided.

04:59It also starts to look at personal context and starts to build out a working state. So if you look at the sample code that we have here like there is there is a goal one of the let's let's work with this query where the user is trying to redo their living room with a certain budget in mind and some of the um some of the things that the agent develops as part of the working state is the session history.

05:21It has a user context. It also kind of extracts out the hard constraint that the user might have provided in the query. But things start getting getting interesting when we come to the softer constraints. So this is where the user may not be able to describe what they're looking for and may might have provided like a reference image on you know an inspiration that they had in mind or some kind of a layout design that they really liked and is trying to get to the agent in terms of uh this is what speaks more to me.

05:49So this is where the agent starts to be more proactive and pulls out some of the salient signals from the from the reference images and starts to develop a mental model of what the user might really be looking to get at. So here the user is uh so the agent has specified has identified that maybe the style that is working out well for them is of a certain kind.

06:06It also is also is also starting to work towards a confidence score on like how confident it is in terms of pulling out some of this information present in the multimodal inputs provided. And then the last thing uh that happens as part of developing this working state is also figuring out what are the variables that the agent needs to pull out almost in real time because these variables uh dep these variables do affect how the results will be displayed back to the user this could be variables that need

Variables that have to refresh in real time

06:36to be refreshed in real time like um inventory because if if what you're providing back to the user is stale information then it's kind of a mood point. So these are the these are the variables that you want to refresh in real time and is is and becomes a part of the agent's working state.

06:44Um few ways that you can evaluate this state. Uh the way we have developed our autoators we we do make sure that all facts are retained meaning that whatever was mentioned in the part of uh whatever was mentioned in part of the context is properly represented in the working state.

07:03We also capture if the confidence collaboration was within a certain error bound because uh the agent needs to be able to confidently pull out these signals from the input. We also focus on getting out um the counterfactual sensitivity uh counterfactual sensitivity.

07:21So we do this by flipping some parts of the queries and making

Grading the state: facts, calibration, counterfactuals

07:28sure that when the query changes the underlying constraints pulled out by the agent the those also change and the ones that are not relevant stay the same. So we kind of want to measure the sensitivity both ways. And then the second part that happens in the discovery phase as well is once the agent knows what they have what they already know about the user, what more do they need to know about the user.

07:44So what we call is this what we call this is kind of discovering the intent gap. Um there's a lot of times that there are lot of unknown variables before the agent can provide the best answer. And amongst all these unknown variables, the agent doesn't need to go and find all the unknown variables up front.

08:01So what I mean by that is in this our unknown in in our working example, some of the unknown variables that the agent said they need to find out more about is maybe knowing what the room width would be for the user or even kind of working on improving the confidence for the style before they can recommend back the results.

08:16Now uh once this unknown

The intent gap, and picking the highest value question

08:25variables are figured out, the agent also needs to work on a collaborative strategy. compare all the different moves possible and then figure out what is that one unknown variable that it should prioritize such that it has the maximal information gain at that point.

08:38So in this case um I mean one could argue that maybe finding out the room width is the best next move for the agent because if the products that the the agent is recommending doesn't fit into the room then again it's a mood point and that is one variable that is going to meaningfully change the uh the direction of the conversation and that's what the agent works on in this step kind of figuring out what's the best next thing to ask to the user and what's

09:02uh why is that the best next thing to ask as well once it has uh okay so I'll get into the autoator section on like why how do we evaluate this collaborative strategy? We first focus on making sure that the agent is able to identify all the different blockers that are needed to be answered before the agent can come back with uh meaningful responses.

09:17We also work towards making sure that the agent is um optimal in trying to get some of these responses. So we also don't want to have the agent constantly going into the loop and continually continuously asking these questions. So over asking is definitely something we flag.

09:35We also work on uh question utility. So again, how optimally is the question being asked? Is the question indeed useful to get the right or elicitate the right preference from the user and many more. Okay. Uh moving on the the second part that I was mentioning is the multimmodal elicitation.

09:51This is where the research phase happens. So play along but like let's say the the agent has asked about the room width. the user has provided the response and then the agent goes to the next step which is where the agent is asking about uh the next constraint that it needs to know about is what was

Bridging a constraint to the product ontology

10:15the style preference of the user. Now the first and the first thing that the agent needs to do is kind of form this temporary bridge between the constraint itself and how that maps back to the um the product catalog and the ontology in your knowledge knowledge database.

10:24And this is going to be important because when you start retrieving these products, you want to have a way to map these constraints back to your knowledge base. So this almost happens in real time where we map the uh known constraint or the constraint that the agent is exploring back to the knowledge database.

10:47We also work towards um figuring out we also work on like the agent knowing what is the best way to get the response for this constraint as well. So in this case the agent has decided that maybe since this is this is kind of a subjective constraint the textual textual elicitation is not the best way to do this.

11:02So one of the ways that the agent thinks this could uh this

When a visual board beats a text question

11:10could be uh elicitated back from the user is using some kind of a visual preference um board. So the agent then goes back to determining what is the best form of options to show to the user. This could be a combination of figuring out from the existing constraints the past conversation and then the temporary mapping that you have created from the constraints back to your product ontology.

11:27So in this case the the agent thinks that maybe coming up with a few styles that are most similar to what the user had provided as reference image could be a good way to start thinking or guiding the user towards a common language on what could be uh something that the user is interested in.

11:42And then the the uh then the agent also goes into kind of um observing the space of what kind of reactions the user is giving. So it could start looking at these micro signals of if there was a hover or a click in a certain direction and starts updating its confidence model on what kind of signals uh what kind of signals can be used to improve the confidence and like what could be the style preference for the user.

12:07Um some of the autoators we use here we do look at how efficient the a how efficient the agent is in discovering hidden preferences. So typically we would use a user simulator feed it with some constraints and then we'll see how how efficient the agent was in kind of elicitating some of these constraints.

12:31We also focus on turn efficiency. So how efficient was the user in how many turns did it take for the user to be able to elicit to be able to uncover all these hidden preferences. Ideally, we don't want the user to go into this loop and keep asking the same questions and again again and again or also do we want to go into the loop of asking some um some questions which may not give you the best uh which may not give you the best information required to proceed the conversation and then we also look at format format selection

Choosing the response format, and grading it

12:58accuracy. So we typically also look for if the agent is asking the right right question in the right format. So for example, if the if the question was something that was easily stable, the right format could be a textual elicitation. But if if this was more of a fuzzy question where the user is clearly having an articulation gap and is not able to describe their preference, maybe the best way to do this is speak a common language and come up with some visual uh anchor points for the user to say what speaks more to them.

13:29Um yeah. So then the last tip in this process once the so to recap like basically the now now the agent knows exactly what they what the user is looking for. The agent has identified a collaboration strategy has figured out all the preferences for the user and knows what is going to be most optimal in terms of um the different priorities for the user.

13:44The last step in this process is to also use the model intelligence to figure out what is the best way to provide this response back to the user. So in this working example, we found out that the agent knows what the style preferences, has figured out that the user is looking to buy products under a certain budget and knows what are the different dimensions along which it needs to find uh these products for because through the conversation they figured out some of the different um different metadata information that is

14:17going to be relevant to surface when they're giving out these product information back to the user. So one one important step that happens at this this stage is figuring out what's the best way to surface back this response. So for example like if the user was looking for a particular policy or review information about a particular product maybe the best way to do this is to go give out a summary or a bulleted list.

14:39But if the if the user was looking more towards comparing two different products maybe the best response is to give out a trade-off table or a comparison table across the different axis that the user cares about. Um, and in case that in our example where the user was looking more towards kind of style inspiration or ideas on how they can kind of redo their room, then maybe the best way is to give out some visual references and product inspiration photos on like how what are the different options and possibilities uh for the user to uncover.

15:04So some of the ways that we focus on uh evaluating uh this stage is focusing on the format accuracy. So we want to make sure that the response format is kind of optimal for the user query so that they can find out exactly what they're looking for.

15:20Uh ideally the information that they're looking for should not be buried in the response but should be easy for the user to spot so they can commit to the next stage in the intend journey which is to basically buy the product. uh we also look at uh data fidelity which is to make sure that the model is not hallucinating and it's really capturing the information in the correct format in the you know the the information is just accurate and is captured across um the response.

15:45We also look at user actionability. So this is ensuring that the response format is such that uh the user is very confident and commit and commits to the next action which is like I said the action to purchase the product. So just to recap so far what we have is um you want to design the product um you want to you want to design the product such that you're prepared to accept wives like I say so that is to say that users will come with fuzzy intent users

16:14will not have a well- definfined goal so you want to make sure that your system is able to work through queries that are not clean um the second takeaway is you want to focus on showing and asking rather than always asking with textual pref uh text with textual Elicitations again like visuals and comparisons do reveal preferences much much faster.

16:31It allows you the agent and the user to speak a common language. Third one I would say is uh shape the answer. So do

Four takeaways

16:40focus on making sure that the presentation format is ideal for the user being able to find the right information. Um the the way you have the model response structure is also very much part of the uh intelligence. And then the last one is make sure that you grade the loop.

16:49you have the right autoators set up on every step of the process. Um, and honestly developing these autoerators uh is a is is almost like an evolving system. It starts very simple but as in when you start the the system starts evolving you want the operators to kind of gradually grow with your system and start um um yeah it should just gradually grow with your system.

17:12So that's all I had. Um I can take a few questions uh but hopefully the learnings we shared are useful for whatever you folks are building. Yeah. Yeah.

Questions: merchant ontologies, UCP, and agent buyers

17:38What about structure shift in the retrieval should be also structured in certain way that it can also fair. Yeah. Yeah. Great question. So the question is about what should the how should the ontology be structured on the merchant side so that it's fair both for the agent and the merchants.

18:12So yes, we do take a lot of advantage on the uh domain expertise of the merchant on as to where what they're trying to sell and like we do work towards creating. So you remember how I was mentioning about the bridge between uh the constraints that the user might have specified and then what the agent uh kind of understands that is where we do expect a lot of intelligence to flow from the merchant side where the ontology on how that constraint could map to the different metadata that the agent has sorry what the merchant has maps in.

18:45So yes, we do partner a lot and then there's also like the UCP stuff that we launched recently which allows all the merchants to kind of start speaking the common language with the agent as well. for example like a sign. Yeah. So ideally I mean honestly right now we focus on making sure all of this flows back to the agent and the agent makes the decisions because we want to build like a horizontal common layer across all the different merchants.

19:29So and also like it should be a seamless experience for the uh for the user who's interacting with our apps. So right now the response format is very much part of uh the agents intelligence. It is not something that the merchant gets to uh decide.

19:45Yeah. Yes. Just curious your opinion what happens when your user is not a user anymore. It's an agent. That's a great question. Um in the early stages of building this out but I think I mean there could be a case where um Yeah. Yeah. Yeah. Exactly.

20:02So, I would I would expect like an MCP to be the interface between the two for sure. Uh, honestly, we haven't gotten to a point where we have agents interacting with our agents just yet. Also, like what we've realized at least from our user studies is uh users really like to be more involved in the process of choosing or even exploring the different possibilities.

20:21So, during the upper funnel journeys where users is looking more towards discovery, inspiration, that is where they would rather be interacting with the system than in their agent. I think where the agent typically comes in or even where uh what we've heard is like towards the lower end of the journey where they're just looking to compare or negotiate or compare prices across different merchants but very much upper in the funnel it's the users who kind of interact more with our systems.

20:36So yeah, I think um yeah, I can take questions outside, but thank you folks for coming.