Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax

Introduction

00:12Joining us on stage is the co-founder and chief science officer at Hugging Face, Thomas Wolf. Hello everyone. Hello Oliv. Nice to have you on stage. Hi. Nice to thanks for having Yeah. So I think you're on for a treat today because you just saw uh GLM which is current number two on the

The race between the leading open-source models

00:47artificial analysis table. I take fab fable out because nobody can use it. And now we have number four. So basically you will have all the top models at least the top open source model in a row and we're very lucky to have a leave who has a pretty amazing path in life.

00:55Uh so she came to the US Pennsylvania she was studying doing PhD at NYU in the lab of Yand working on JPA but we decided we won't talk about JPA today right something for another day. Um and then uh instead of joining hugging face which was uh in New York also at that time she uh decided to go join uh Mini Max.

01:28So uh for those who who maybe don't know all the all the nail labs around the world and you're you're forgiven because I think there's like 64 nails right now. Minimax is one of the top of what we call the AI dragons in China. So these are the new there's uh there's deepseek which is very well known now.

01:45Moonshot who does Kimmy Z and GLM that you just saw. And now we have Miniax. They're all extremely good, extremely talented team fighting for the first spot. Uh so the the the latest release of Minax was M3 uh just earlier earlier in June which was the the top model at the time top open source model.

02:03Uh very impressive. There's a lot of very interesting things about this model. So we'll quickly dive in them and then talk a little bit about uh what's what's what's specific about Miniaax, what's what's great there. So uh maybe Olive to to start a little bit.

02:22Can you can you give us you know a little bit of your your view of of M3? What you like about this model? How is the release? Mhm. Yeah, M3. We released M3 earlier this month and it is a

MiniMax M3: coding, vision and one million tokens

02:39smaller model with 400 around 400 billion total parameters and 20 billion activated. Um but it is very capable in terms of both coding performances and also it understands vision. So um that's uh what open source models don't usually have is that they can the model can only deal with coding but it can also understands videos um images and it has a super uh long context of 1 million um with our new architecture called MSA minax bars attention.

03:12So we you we really put these three things together uh because we know that they are they will be very important in future AI applications coding capabilities agentic capabilities longer context and multimodel understanding. Um yeah I think that would be very interesting about the model.

03:35Yeah. So, so there's a lot to unpack unpack in this model and it's um it's it's still I think the only top five model open source model that is actually multimodel. So, we need to talk about that. But maybe first about the long context because there was also the first one that really had this real 1 million

Why agents need longer context windows

03:51token long context that's actually functional and you guys had also the the minimax pass attention which is this one technique to to make that efficient that you also published and and share extensively. So can can you talk a little bit about this maybe how the project went from from the attention how to make this long context?

04:07Yeah, I would say the story about long context went back to even Miniax M1 and Miniax01 where the model was actually was able to perform tax of 10 million um token context 10 million 10 million yes um but then it was not an agentic model right it was just um 10 for example you dump in a book it would be able to give reviews on it stuff like that so uh what we realized was that you know longer context actually unlocks a lot of capabilities especially when interacting with users and now when you know the agents is interacting with the whole environment and getting all the

04:53tool responses um getting multi- rounds the like shorter context wouldn't be enough to uh perform the complex tasks. So for this version we said oh we have to have our longer context backs. Um so what we pursued was with our minia max sparse attention um which you know was the architecture that was scalable and had a simple design.

05:20So I would say from a higher level right it has an index branch um that you know selects on a higher level what is what matters more in the context and then we have a sparse attention branch that calculates performs the calculation on the selected blocks uh to actually performs the tasks.

05:44Um and so yeah like that we really designed um an elegant architecture so that we can scale the length and then scale the model size in the future with that. That's beautiful. I like how for for those who've been in the field for quite some time we we had a lot of work on attention right this n square and there was a lot of linear attention.

06:04Yeah. And then some that somehow all of this disappeared at some point when flash attention came around. We discovered we just needed more efficient kernel and now I like how we come back to thinking you know first principle what is attention how can we make that more efficient so 1 million token is crazy right GPT2 was24 and and everyone was like oh that's really big we we we never

From GPT-2’s 1,024 tokens to million-token models

06:28need more where do you see this coming like going in the future like Jeff was pitching me the other day a trillion token attention you think we should go token attention that's definitely something we can explore towards right ultra length of the context definitely that's something that's very exciting to explore with and something that architecture design along with hardware um would require a lot of research on to that yeah you think there's still a lot of lowhanging fruits so typically today we saw open really reducing I mean we don't

07:01know how as a phone but like reducing their their inference bill by half by probably having some more efficient processing around tension something like that do you think there is still a lot of flow low hanging fruit that can be get in how we can process that.

07:08So, so one one thing still very interesting about M3 is how cheap it is in particular because of this part attention or in part because of it's small one but it's also very efficient right you think we can go even way further maybe how did you guys invented uh minmax pass attention was it an agent

The intern who designed MiniMax’s sparse attention

07:31coming up with the idea was it a human still coming up with the idea tell us a little bit about yeah um so we do think there's still a lot of work that can get into architecture and inference optimization so that the model can be more efficient especially if there are tasks that are very task sensitive but require very strong capabilities right and for that those kind of task we really want the model to be efficient um and who came up with partic I think an intern from our team worked on that that yeah an intern uh that doesn't usually happen in a lot

08:08of labs u because I think in some labs interns don't have access to the data, the work and stuff. Uh but yeah, we are open to anyone who would like to contribute to our model. So um the architecture was actually designed by an intern. It's really good.

08:24Still some work for interns here. Good. Good news. Um that's also a good segue to also how Minmax is

How research projects work inside MiniMax

08:31working internally. So So we were discussing before coming on stage, they were saying everyone can propose a project. Can you tell us a little bit about how you are organized, how you do research? I think that is very different from um even in school or even in earlier you know the earlier tech companies is pretty pretty different is that um what we what we make sure is that we have good foundation and good um infrastructure so that anyone can play with the model and can think of what they can improve with the model and then after model releases when they are free Right?

09:09They can play with the model. They can think of their own evaluations. They can find their own weaknesses and propose a thing that they want to improve on the model. And then other people who are interested in that would you know propose to join the project and they will work on for a couple of weeks or even a couple of months and when they work out the final thing is shipped to our model.

09:33It it is you know we use that in our final training and it's shipped out to the audience. Interesting. So you can have people working for a really long time on project. When you say a couple of months, it can be like really deep exploration of possible.

09:47Yes, I would say for example architecture might require longer time of investigation, research experiments, even redoing the evaluations for treat pre-training. Yes. So it might require longer time. It's really nice. Yeah. And I know you're

Training a multimodal model from the first step

10:02also very big on evaluation. I agree. We could talk about that. I think one one thing probably related to that is this unique specificity that M3 and your team has um around multimodality. So not just text but this model can also understand image and video and as I understand but but please expla explain better when we read the model card on hugging face it say the model was trained from the first step as a multimodel not just as a like user one as after salt right can you tell us a little bit more about that and why you think it's important and and and why starting from the first step on

10:34multimodel training and not just training um so we call it native motor modality um And so it is somehow typical for model labs to train the multimodel let's say vision understanding capabilities after the text pre-training is done. Um they put adapters and then train that part.

10:56But what we found out was that that would actually harm the text performance and the vision vision understanding performance wouldn't converge that well because the model is kind of converges towards the model uh the text understanding. Um and it's just not the most optimal and also not the most scalable if you think about it.

11:14We want to scale the data, right? And also we can also some labs um train this capability from halfway through the pre-training for example continued pre-training but what we found that this would be very you know uh recipe sensitive it is different for the recipe would be different for different architectures different uh you know data mixtures different learning rates um it's hard to control hard to you know scale to you uh you can't really scale your experiment results and conclusions to a larger model.

11:53And so for you know what we thought was why not just training from the very first step that comes the most natural. We know that a lot of labs run into problems doing that. Um the model would collapse after a couple of steps of training you know both text and vision understanding.

12:10But we managed to solve that problem. We did a lot of work on um VIT and we did a lot of work on the data that we actually training. Um for example, we do interleaf data. What we call interled data um it's actually natural data but we keep the um images and videos in instead of masking it out.

12:36And we do some pretty good cleaning and masking on the data and we do very good reward modeling so that we train it from the first step and scales up a lot. Yeah, it does does not collapse. That's really impressive. Impressive. Should do should we expect much larger model in the future?

12:55So this one is still fairly small, right? It's it's 428 billion parameters, 23 active billion. Um well do do you think you will go past the trillion?

Going beyond one trillion parameters

13:09Definitely. Yeah definitely in the future uh there are many tasks that wouldn't be able to the more model wouldn't be able to perform or good at with smaller parameters. We are definitely going more ambitious than this.

Building AI products for 300 million users

13:25That's great. Looking forward. Um, another interesting thing I always find fascinating about Miniaax is how how you also have this whole range of of apps and product, right? So I remember already so so Miniax started to open source things on the on the hugging face platform in January last year.

13:41So like 18 months ago and and we were chatting a little bit about the team to understand what you were doing and I remember so you were already having a huge usage on some of these uh of some of these apps. Um can you tell us a little bit how how this started right so was it basically you had a lot of apps and then you thought we have we have all these data why not training a model and then they build up research team how is how is the story there um our our story is model from the first

14:11day so um I believe that multimodality model a model that can understands all visions and outputs all modalities was the first thing that our um CEO planned on the first day even before the company even started. So that was the dream of AGI.

14:27I think that was very very early even before Chad GBT came out. Wow. Um yeah and then apps were something that comes along because you have some model capabilities you want people to experience it well not many people can use it with API right we can't expect everyone to experience with API so we need good um user interaction you know interfaces good apps good scenarios that people can can you know experience experience the model with I think actually those apps covered more than 300 million people around 200 countries globally and I think over a million companies as well.

15:09Yeah, this was uh mind-blowing when I heard about the size and we we don't often realize the size of of of this type of usage already and and that that kind of brings me to the question around um opensource business model and and all of that which

Why MiniMax plans to keep open-sourcing its models

15:25is the always existing question which is right now it's nice to open source model but you you also need to have some revenue stream right and so I guess M3 is something you you decided for instance to give for free and I'm think it's It's it's great for the world.

15:39Um, how do you see this? Do do you also have some specific models you use for the app? Do you think about do you think in the future you'll keep it's probably hard to say for sure, but do you think you'll keep open sourcing models? How is the culture around open sourcing right now?

15:55Personally and also for the model research team, we always hope to open source models. Um that is our plan because we really see how the open source community together can help the model build better. For example, we receive a lot of um feedbacks on the model performance from the great community and we receive PRs on um whatever we open source right and those are very very valuable and come comes to our later versions.

16:18So definitely open sourcing is great. That's great. And actually do you have some ask for the audience people who are using uh M3 or minimax? Is there something you would love them to send back to you as feedback? Do you do you for instance do you read when people try to modify the models or play around you know tweaks or what is the best thing you you think you can take from the community uh for the future models for instance?

16:52I would say whatever um issues that people are running into especially with multimodality right this is the first time that we're combining it together we are definitely going more ambitious on that in the future it might have some flush right now but we are improving on that so whatever that's uh feedback that model is not good doing that great we will definitely improve that in future versions and also whatever features that uh people want say you know for example um thinking effort right some people ask for that um like everyone can ask and we

17:26will try to accomplish that in the future models yeah do you see a lot of usage right now already in multimodality in terms of

Agents that understand presentations and long videos

17:33coding agents I feel like it's it's a little bit underexplored it is it is but um it can actually unlocks a lot of capabilities and a lot of uh agent applications say that for example you want the model to read the PP PowerPoints, right? Or to read some reports that is not very structured um and you wanted to understand a very long video.

17:58Say that you dump in a long playing video and then you want the model to act uh using some tools uh after understanding it and it unlocks a wide variety of um agent use cases. So like the agent could finally watch my YouTube tutorial and understand how to use my coding tools how I described it.

18:14Is it something like that? Huh? Could the agent finally watch YouTube tutorials and understand things from them? Yeah. Yeah. I think so.

Automating AI research with agent harnesses

18:27Do you use a lot of uh agent coding tools internally? Is it like I mean coding for sure but like is it also already in terms of research? Is it automated part or not? How does this Yes. Um, we have our own research harnesses. Um, we build our own research harnesses that automate our workflows.

18:44I would say a lot of our workflows are automated. You can see how um the latest frontier models all pursues capability like kernel optimization, right? Like let the model pulse train other models. Um, let the model build data, auto data, stuff like that.

19:01Um you can see how more and more models are capable of doing those including M3. Actually we were very good at those cases longer horizons and kernel optimizations. Um and so we can use that model capability harness it together and help with our um daily routine and make our iterations even faster.

M3 is already helping build M3.1

19:27Is M3 building M4 already? Um building M3.1. M3.1. Okay. Let's hit the gem. already. Um I would love to finish on what what you find exciting in the coming month. What do you think? It can be Asia in terms of feature or or things you want to see happening in in AI or or more generally in terms of whatever whatever really is top of your mind.

19:49I would say it's going to happen.

Why multi-agent systems come next

19:55A lot of things are very exciting. Um but what I recently find the most exciting would be a multi- aents that I think a lot of AI applications are using model routing multi- aents um that unlocks even more capabilities even comp more complex tasks and also it tells us what the models are capable and not capable of and you can you know do a lot of things with that is pretty exciting.

20:24Thanks a lot. Pleasure to have you. Thanks for having me. Thanks everyone.