This Insane AI Video Search Technology Selected by NVIDIA and Snowflake | Twelve Labs, Jae Lee

Intro

00:00If you try to accommodate all of the advices that you get from your mentors, your company will most likely become like an underwear company totally different from what you wanted to build. Having some fundamental like foundation for yourself and for the company, and be able to say thank you for your advice, but no thank you.

00:18So having that the gut to say no to someone that you you respect as a founder, I think I grew a lot. Hi yo my name is Jae. I'm one of the co-founders and CEO of Twelve Labs. Twelve Labs is an AI research and product company based here in San Francisco and Seoul.

00:34We are building video foundation models for developers and enterprises building video centric products. We basically build humongous AI models that can understand videos like humans, and we serve it to developers via APIs that are looking into building really powerful semantic search or classification or summarization into their products.

00:52We currently have a little over 20,000 developers that are actively using our search API. We have largest creators of the world adopting Twelve Labs as well as media, entertainment, and large sports organizations and law enforcement.

Chapter 1. A Startup Founded during Military Service

01:10So I was born in Seoul. I spent about ten years in Seoul. I had a chance to move to the States. I moved when I was 11, so I went through elementary and middle school in Knoxville, Tennessee. I was able to pick up a lot of the culture and things like that.

01:26I was very interested in expanding my perspective and exploring the new world. My first experience with software engineering, or at least coding, was Matlab. My uncle was actually getting his PhD at the University of Tennessee, so I see him kind of like plotting this distribution graphs and things like that, and which made me curious what what he was doing.

01:47That's how I got into playing around with like, small data sets and doing the same thing that he was doing, because I wanted to be relevant and I wanted to talk to him about a bunch of things. He probably thought what I was doing was pretty cute, which sparked my interest in learning more about how do we capture all this data and be able to create a system that can really understand the distributions of the things of the world.

02:10And it just kind of felt like if you had the understanding of that, it gives you power to predict anything. And I went to Berkeley for college and studied computer science, so I really geeked out on AI and software engineering. So I spent about 15 years in the United States.

02:25So half of my life in Seoul and in the States, I was drafted into this organization called Korean Cyber Command, where like minded people are already there, armed with incredible knowledge in software engineering and AI. I joke about it I served the country with keyboards rather than a rifle.

02:44I was really fortunate to have met my chief architect, SJ, and then Aiden kind of also joined in. He was, you know, I still clearly remember Aiden with his buzzcut coming in from from boot camp. But from day one, we knew we had this common interest in AI.

03:01And what can we do as a as a young scientist, really like push the frontier of AI development. And we've spent a lot of times reading papers, discussing, arguing. And it turned out there were two clear paths. One was after military, we go pursue a career in academia, go become a professor, or start something of our own.

03:23And looking back, what we realized is that we were spending so much time together and military is this really special setting where, you know, you're basically jamming in 50 to 100 of 20 year olds, right? ith Testosterones what we thought was, you know, if we're having this much fun in military, imagine what we can do when we go out.

03:42So it was pretty clear to us that we're going to start. I think we've spent about a year and a half really thinking about what is the next frontier for AI, and how can we contribute to pushing that boundary? There's this seminal paper called Attention is All You Need, which some people call it the transformer paper, which is making a lot of impact now.

04:02But when it was published in 2017, at least for text and image based, you know, foundation models, probably capital was going to be a mode whoever raises the most amount of money. What we realized was there's still a lot of unexplored research areas for multimodal video understanding there, rather than capital is probably going to be really passionate, smart, but also slightly dumb people.

04:26That's like dumb enough to start. It will have really good chance of succeeding. And also we've realized that with the explosion of that video and, and other complex multimedia data is going to be the infrastructural data for the internet. And we realized that probably developers and enterprises need something better than object detection or transcription to make sense of all, all this video data that the humanity is creating.

04:48So it was like a no brainer to start building for video Understanding the model that Twelve Labs is building, it's basically trying to map human language into whatever that's happening in video content. So if you can map precise human language to whatever that's happening within video content, that gives you this emergence capabilities, like being able to search for things really well or being able to classify things or summarize, you know, we didn't all join the Korean Cyber Command at the same time.

05:16So SJ was already like six months ahead of me, and then Aiden was six months behind. So like, okay, we decided we're gonna, we're gonna start this company. But then SJ is like leaving like next year and then I'm leaving the year after, and then Aiden is like six months after.

05:32So how do we do it? So it was genuinely very scary. So we had our ideas. I remember SJ was discharged on Thursday and he came back to the military base Saturday of that week with our laptops, and he took us out in front of our military base.

05:46There was a bagel shop called Last Bagel and that was like our office where SJ would bring all of our laptops and we would do our research and do some little bit of prototyping. And we did that for six months. And then I got this charge, and then I did the whole laptop carrying and and taking out Aiden.

06:03So we did that for like a good like year so that everyone is out. And then we had a bunch of friends that were working in AI and crypto and blockchain and Web3 was just booming. And we had a mutual friend, like the founders had a mutual friend that had like a really nice office in Seoul, and it was like, oh, you guys can come in and use our office space.

06:22So that's what we did. And then after like three weeks, that company went bankrupt. And then really scary, like people started coming in and we had like our desktops and our GPUs all set up there. And we got really scared. So we brought everything back out and we found like really tiny, tiny office, probably like a size of a dressing room.

06:41That's where, like, all five of us kind of spent the next six months before we raised our proper seed round. Looking back, if we were to do it again, I don't know if we'll be able to do it, but some people say ignorance is bliss. And I think we were just like, really naive and just really excited about building this company.

07:00And I guess not knowing what was ahead allowed us to kind of do what we did.

Sponsor

Chapter 2. What a startup with only $2000 can achieve

08:10When we first started the company and we hired our first employees, they had a hard time explaining what Twelve Labs does to their parents. Really new thing. What is foundation model? What is video understanding? It's a new concept that's hard to understand for I guess, people that are not in this space.

08:26Nowadays people talk about the foundation layer, the tooling layer and the application layer. It all. Everyone's very familiar. When we started Twelve Labs, I think technologically it made total sense. We knew it was going to happen, but what was uncertain about it was will market accept?

08:42But knowing that we are at the verge of breakthrough in building an AI that can at least, you know, get to a certain level of human understanding of videos. So we were betting on markets acceptance of foundation models. The founders are pretty pretty much broke, right?

08:58Because we spent two years at a military, and I think we had $2,000 to start with, so barely, barely enough to do anything but figuring out what was going to be impactful that we can do, given our current resources that will put us in the map, or at least, you know, let the world know that what we're doing is relevant.

09:18So our tactic here was, okay, we're going to talk to a bunch of customers. And there were early believers in Twelve Labs who took our APIs and built awesome things with us, but we needed more exposure basically. So as a team, we've decided to participate in ICCV it's International Conference in Computer Vision.

09:38They're putting this like awesome competition for video understanding. So we talked to Aiden. We have nothing to lose and only to gain. The team was extremely supportive of of Aiden spearheading that effort with the team. All I can do to support is.

09:53You know, there was some ideas and and directional kind of feedback that I gave to Aiden, but we needed compute and we needed determination to put some serious cash behind And back then for Twelve Labs like $200,000 in in compute wise, it was a lot of money for us.

10:08So and just thinking that, okay, we're going to blow through $200,000 in ten days in compute was really scary. But the team was able to use that capital, that that precious capital and build something incredible helped us win the competition.

10:24So I think the important thing is, if you're building something really impactful and you think that it's going to significantly change the industry that you're in, there will always be someone that has very similar thesis. It's just a matter of how do you get yourself out there?

10:39How do you let the people know that you exist. For us, that was the competition. After winning the competition, companies like Index Ventures and Radical Ventures had very strong thesis around multimodal AI. The next idea would be what's next?

10:54And video happens to be the most relatable multimodal data. These amazing companies actually came in inbound, so they reached out to us and we started jamming. The conversations turned into next conversation, and then we talked about technology and and it just happened very serendipitously.

11:14The first pitch deck I was in Seoul, and my first call with Index Ventures was at like, it was like 3:30 a.m. Seoul time, and we didn't have pitch deck, and we just felt like this is our first meeting and we knew nothing about fundraising then.

11:32We didn't even know this was going to be like a friendly introduction, but I just kind of felt the need to like, oh, we don't have pitch deck, we need to build one. So I remember staying up till like 3 a.m. Building it. I think the storyline was like quite simple.

11:45We didn't have much too much to show for it. The idea was, hey, the problem that we're solving is massive. 80% of the world's data is in video, and there's no adequate solution out there or technology out there for developers and enterprises to make sense of it all.

11:58That is the market that we're we're tackling. We want to index all of that, like 80% of the world's data. And this is the technology underlying research work that we've done. And then that was it. VCs asked a lot of hard questions. The most memorable one, if I bring you TikTok as a customer and they want to index like billion hours of content, how long does it take?

12:18And I think we were we were thinking about maybe like 1,000,000 hours. It's going to take like ten years, right? That's when we realized we should never be comfortable with what we've built, this whole new, incredibly large world out there.

12:32You know that maybe some people are impressed with our system being able to index like 1,000,000 hours. But there is others that are thinking about billion, 10 billion, 100,000,000,000 hours. And that was like a really challenging question because like what we said during our pitch is we want to index all of the world's videos.

12:49And that question kind of made me stunned. And to think about all of the technical issues that that we had at the time, he probably find it like, funny, right? I was trying to give my best answer as a founder. It's like, so the company has raised about $30 million in seed funding.

Chapter 3. Lessons from Acquiring Early Customers

13:08My name is Soyoung Lee. I'm one of the co-founders of Twelve Labs, and I currently lead our business development and go to market. We have a customer who was paying for our product, but they weren't actually using it. But we had gone through a lot of work to actually get them as a customer, through a lot of sales and kind of relationship building and so on.

13:27But I think they were, you know, they were extremely early and we had almost pushed the sales to happen. And I think what we learned from that experience was we have to optimize even early on, it might have been better for us to not actually make the sale because the customer probably wasn't ready.

13:43They didn't have the passion or the innovative drive that our other customers had had. But we were optimizing for kind of hearing the Yes We tried so hard to make that no into a yes. And we had succeeded. But at the at the end, I think it turned out that we probably should have kept it out No and focused on all the other customers where the yes was more clear, and they had a very clear vision of how they can build their new experiences with the technology, and especially for earlier products and earlier technologies where resources are limited and you want to build for your best customers

14:19and the innovators in every field, you should probably start to optimize for hearing the no than the yes, because that will help you find the right direction faster. We quickly. I think we learned through trial and error that the types of companies that we need to early customers that we need to find and work with are true innovators in their field, whether they come from like the content creation space, law enforcement space, e-learning and so on.

14:45You know, we've had instances of trying to oversell. We were going through sales, you know, sales 101 books or like sales methodologies. And we were trying to pitch to the customer, hey, here's the use case you could build out with our, you know, with our technology, you know, we'll improve your ROI by x percent with our technology.

15:03And we did make some sales from those, but I don't think it was something that we probably should have spent so much time on doing. Because if you find the right customer who's innovative, you don't need to explain anything to them. You show them a use case based demo of, you know, for us it would be we would index some videos that resemble the customers, and then we show them how you can search, or you can generate text very easily, like, like just like a person would.

15:28Had they been watching the video, they can draw out the full map of, okay, this is what I want to provide to my customers or my users. This is a technology that you have and I can they can fill that gap in pretty easily. And they know already what the return on that or what the opportunity of that experience would be for them, for the kind of really large, high profile customers that we have right now.

15:51It was all the same process. We actually learned all of these from the customers because they had, you know, seeing our demo and seeing the early kind of hints of the technology, they were able to teach us about how they could utilize the technology.

16:04And even today, it's not just a single use case that they want to power. They come to us with 4 or 5 different ideas of how the technology can impact different units, different business units and different, you know, optimize different workflows or build new experiences.

16:18I think we we make stupid mistakes probably every day. The one that I regret the most is we knew we had like this conviction around building a foundation model, but then we didn't have any data point as to how do you build that company. And I think I blindly kind of believed that the startup mantra like, identified a narrow problem and building a very narrow solution for it.

16:41I think we've spent a lot of early days thinking about, okay, if we have this really powerful AI that can understand videos, what do we do with it? And we know, we know from the from the get go that we wanted to serve it to developers and enterprises.

16:54But I think, you know, we've had mentors and, and other founders like, oh, you should build TikTok 2.0 or you should build like YouTube 2.0 or you should build that. And I think we've spent a lot of time thinking, okay, maybe like TikTok 2.0 makes sense, or maybe like Gong 2.0 like sales call analysis.

17:12But that didn't really like excite us because we knew, like, we're good at building the infrastructure and helping developers build the next thing. And that's probably the stupidest thing that we've done. It's like spending time on thinking about things that we're not excited about.

17:27As a founder and CEO, it's really hard to get distracted. If you are a first time founder or a young founder, your mentor's advice means a lot to you. But having your own grounding, having your own, also really like relying a little bit on your own gut feeling is very important because what we've realized, if you try to accommodate all of the advices that you get from your mentors, your company will most likely become like an underwear company, totally different from what you wanted to build.

17:55So my key takeaway is having some fundamental like foundation for yourself and for the company, and be able to say thank you for your advice, but no thank you. So having that the gut to say no to someone that you respect as a founder I think I grew a lot right.

18:13Twelve Labs has multi-year compute partnership with Oracle Cloud Infrastructure, where we get all of the state of the art Nvidia chips. And Oracle have put together this small event for their key partners that are building foundation models for.

18:28And Aidan and I had chance to meet with Jenson because Jenson was at that event and we had, I think, 5 to 10 minutes to talk about Twelve Labs and I think it seems like he had he has a special place in his heart about computer vision and video understanding is, you know, that was one of the first use cases that Nvidia chips powered.

18:47So we got to meet with like Nvidia folks from that event. And then Twelve Labs was featured in Nvidia's 2023 GTC. And then I think that kind of sparked other people from Nvidia to be interested in Twelve Labs. And the Nvidia's venture team reached out to us.

19:03It was quite casual. We were talking about Twelve Labs and the future that we're drawing and the future of multimodal video understanding. And I think the venture team also had an idea of how Nvidia and Twelve Labs can partner up more than just financial investment, but also think about really robust product partnership from then on, what Twelve Labs is doing and what Nvidia wants in vision and video understanding was just like a perfect match.

19:28It happened quite naturally from from conversing. Our technology, our roadmap and Nvidia's future in terms of like producing really powerful chips for edge devices for smart cities, right? So there's that natural fit of two companies product really creating synergy.

Chapter 4. The Best Engineer is not the Best Coder

19:50Nowadays I am focusing mostly on hiring. So I think Twelve Labs is a group of great people. I spend a lot of time meeting great people. I want to be able to recognize greatness when he or she comes in good engineers or even like just good people in general have this core values.

20:09I would go near their places and get together at a cafe, and we would speak for three four hours. My way of deciding whether this person is a good fit for Twelve Labs is I'm able to learn from their core values. Everyone's really good at coding nowadays, but great engineers can apply their core values and their skill sets, and is able to talk about the company that they're excited about and how they want to impact it.

20:34How do you see the product evolving? How do you see our interfaces evolving? Some of the best engineers are not the best coder, but having that perspective, really strong perspective and and groundedness is very important, and I try to look for that.

20:49Twelve Labs vision in the next two years is really becoming horizontal. Video understanding infrastructures for all of the businesses and developers that are working with video data, we want to enter into streaming as well. So real time video data and really become a visual cortex for modern video applications.