AI-Native Organisations Run on Skills: How to Structure and Scale Them — Imad Touil, QuantumBlack

Three questions, and the hands that drop

00:13Thank you for joining me today. My name is Imatel. I'm a distinguished engineer at Quantum Black. And today talk I want to really cover the uh how AI native organizations runs on skills. But before I really get started, um I just want to do like a quick exercise to show of hands.

00:30Can you raise your hand if you have already created and using skills? Amazing. Now can you keep your hand up if you are using it and sharing it within your teams? Great. Now keep your hands up if you have governed maintained skills across your organization.

00:50Amazing. I see a few hands. But that is what this talk is about. Today I'm really looking to break down why this is really critical, how it's important and how actually you can adopt it across your organization. But before I get started, what I really want to cover is actually the agentic software stack.

00:58So the agentic software stack has two loops. So the first loop is all you know, right? It's the code and agents or the code agents harness right at this you have some core components. You will have your context manager, the tool and MCPS, memories and states and skills loader, right?

01:16But then there's an outer loop which is your workflows, right? Those has skills, sub aents, MCP servers that you use and some hooks. Sometimes you need them to have this running properly. You will need some enablement components at the bottom.

01:31So what you would have is an environment

The agentic stack, inner loop and outer

01:39sandbox. You would have your MCP gateway to manage and simplify all of the MCP tools across your organization. a model gateway to again manage and optimize for all of your LMS either like you know open source like running locally or or or frontier models and also a graph knowledge graph that abstracts your IT core systems your codebase your skills registry and at the end workflow marketplace and then you have your context layer right the context layer will bring all of what is needed to get the task done so this is the project

02:12instruction think about it as the cloud code MD file the agency file your tools and MCP schema is actually to understand which tool to use and when your memory right conversation history with the end user the human is in the loop and finally the retrieved contents that's you can pull from either like files your codebase etc now what I want to really focus on today is the workflow right and I think this is where we kind of like think um it's kind of a pretty simple um

02:41uh workflow in day-to-day when you're trying to actually create an end to-end product software delivery life cycle in the organization. In reality, we all have seen the four steps. specify to define what you want to build the design or plan to plan you know what what are

Why specify, plan, task, implement is one step

02:56you looking to build then you go to the tasks you break it down into tasks and then finally you start implementing right I think this looks familiar I think this is how most of our coding agents actually are kind of shaped today in reality that is not how it is composed and at scale when you look at the organization complexity this is just one step in the journey this is like building a product increment When you really look at the overall end to end life cycle of something that a business want to build to capture the value out of it all the way to ship it

03:28to the client, you will start first of all by defining your product strategy. What to build and how to build it, right? You set the define the success metrics. You identify and you break down your plan road map for your products. And to do this, you may need a lot of insights, right?

03:42So then you do like your market research. You do a competitive analysis. You bring this as an input with some customer interviews. Then you go to the discovery side, right? So then you start discovering. Okay, now I understand what to build.

03:56I'm going to break this down into like some problem statements, find the solution, validate the solution, and then probably like experiment and then create user stories. And before we start building in reality actually need to prepare your data, right?

04:03In some cases, you will need actually to clean up your data catalog that will support the build of your products or maybe adjust some of the endpoint connection and integration

What the full lifecycle actually contains

04:19to your core systems that will help you actually build your products. And this is where the data product delivery comes. So you build your data pipeline, you validate your data quality and you put your catalog your catalog data assets ready for development.

04:30Then we kind of like go back to the product increment. That is where we will start. But then building a product in every organization that I have been uh serving for the past 18 years in my career. I can see that in one organization you will find like different DLC iss kind of like uh you know scatterers across organization.

04:54Some of it is actually for a mobile application other is a different department or different platform. Some of it is internal platform that is for your um employees. Other is actually customerf facing. So it is not like a one workflow that can actually build anything you want for your organization.

05:09When you figure out like what to build and how to build it, you need to run it. Then we come to the platform engineering ops, right? That is again when you have your provisioned infrastructure, thinking about how you you build your your infrastructure as code modules etc.

05:22And then you launch your products and the moment you launch it then you start kind of like the journey of optimizing the performance of your products trying to look forward like any incidents to resolve and then you start the loop again. Right?

05:30So at scale when you look at really building a digital platform not like a very simple products that you can you know solo build and deploy the landscape is way complex that is

Organizations run many lifecycles, not one

05:46expected and what you're looking at here is literally like probably 10 20% of what is it and is really different from organization to organization now going back to the uh the stack right when you look at the workflows there's like four core components the first one is hooks MCP servers and sub aents those are kind of given but it doesn't really bring the right kind of like structured value to your workflows right that's why scales is one of the critical components in your workflows hooks basically what it does it just kind of like pre kind of

06:17like trigon events to to do something along your workflow the MCP service we all know that you know you may need an MCP tool but tell me like who actually build a lot of MCPS we just use MCPS tools that is actually provided by the tool that we used to use before right so we don't really own The sub agents is just to minimize the context window.

06:33We just delegate to sub agent to execute a specific task what is needed. So at the end of the day you will find all of your knowhow is actually at the skills level. And if you don't have the right structure of your skills then you're not really having a deterministic workflow.

06:49And one thing to mention is workflows think about them as harness blueprints that actually shape the behavior of your coding harness for example in the runtime.

Where the know how actually lives

07:03Now looking at the rise of skills adoption right so just eight months ago entropic published the first article about skills right two month later we had this standard an open standard that is adopted and starts a lot of agent harnesses starts adopting this new standard around February um this year we have seen most of the actually agents adopted this even though you don't see them right if you pay attention when the the uh the agent is thinking And you can see that he's pulling skills um as he's going and doing the task.

07:32The other implication here that you need to pay attention to is actually the number of skills created. Right? This is just like literally a snapshot that I did across some public GitHub repost, some public skills registries, right? There's way more than this publicly and within your organizations, right?

07:47So the de the creation of skills and the demand is rising. But we need to understand why in the latest skills bench comparing the latest models against like you know running the same task against auto engineering and cyber security without skills it did well right because that's what expected and it's going to continue to be improving day after day right but then when we applied skills that's our more deterministics the outcome was clearly higher than than That's it.

08:24Now if you think about now like how like the anatomy of skills and how you actually design and implement skills in your organization. This is not like a new problem that we are trying to solve

Designing skills like microservices

08:33here, right? We have solved this with the microser kind of movements, right? So your microservices need to have all of these design principles, right? This is like a software kind of like you know problem that we use to solve is is similar.

08:43So skills need to be reusable, right? Need to be modular, need to be disco like you can discover your skills. So if you are sitting in one team and you need the skills you actually can automatically discover and capture the skills it's portable that you can actually use skills across workflows but also you can use skills across harnesses again everyone adopted the same standard.

08:59So if I'm having a skill on cloud code and I want to move it to cursor, it's going to just work. Specialized skills that is where the value you should not build like a one skill like a monom or again it should be specialized to define one tasks specifically.

09:15Composible uh skills need to be designed in a way that's actually can compose. So you don't have duplication across your skills when you're trying to run them conflicts each other. Consistence that is actually what one of the key items of skills is actually consistency and deterministic.

09:30And finally cost efficience and for cost I can go for another hour talk but it's basically the skills comes to solve a key problem around the context window right is actually putting with the disclosure pro progressive disclosure pattern the right skills the right amount of skills in the right time to solve the right problem and that's reduce the token usage.

09:45This define a new unit right that makes your knowhow

Progressive disclosure and token cost

09:53in your organization executable portable and cheap. On the right hand side, you just a very simple example of data retention policy. Right? When it came to regulation, you need to understand and make sure to instruct your agents while you manipulating.

10:01For example, um your customer data, you should make sure that this data manipulates according to the regulation, right? And that bring me to the next example. On the left hand side, you can see that there's like a think about like a a catalog of skills.

10:20And in the right hand side is your harness and the output is on the right hand side. Right? So the composible skills at the regulation level. You have the skill that I just showed earlier which is the retention policy. But you will need disclosure standards.

10:36You will need the GDPR rules that need to be respected. You will need the fill in templates. Right? So all of this kind of think about it is like defining how any data any feature that is built across your web, mobile um like you know different applications across your organizations is really respecting these rules and this gets pulled automatically by on the runtime by the regulatory disclosure review workflow and the outcome is expected is deterministic.

10:59You have an audit uh audit reports that you can actually store. you have um specific uh identification of if there's anything to improve and that this kind of like loop back to improve your your codebase.

Composing regulation skills into an audit trail

11:21However, if we don't govern skills, we will start creating a new class of technical depths, right? First of all, you will find out that you are having a lot of duplication in your organization. So if teams are not collaborating and everyone is think about it using the same technology stack the same infrastructure you're for sure building the same skills over and over again without sharing them quality.

11:39If you don't test and make sure that you're maintaining and you're validating your skills not against your task but also against the latest models that comes right then the quality starts degragating over time. You should be able to discover your skills in in reality without a governance you cannot really discover it.

11:56Think about it as the the backstage the IDP right it's come to solve a problem where okay I need to understand who owns this microser back in the days right I don't need to talk to anyone I just need to tap into the service catalog and immediately find who actually own it that is brings ownership part if you don't have an owner then no one will be able to maintain scale those skills composibility is not something that comes by default you need to have a governance way to to align what to build and how to design it is think about the

Ungoverned skills as technical debt

12:30domain driven approach that we have been taken also for for many years, right? Is similar to how you shape your skills catalog. Security again some of uh all of us like we experiment with the public skills, right? But when you think about it, some skills may have some prompts injection and skills actually does have scripts because that is the deterministic part of it because it can run a specific script for a specific task.

12:45So if you don't have again a pipeline that check your security, you may be pulling something that is insecure and permissions. Not every skills is actually something that anyone in the organization should access. Some skills may have some business logic that is very sensitive, right?

13:01So the access control is is also crucial at this stage. Now how to bring this to your organization? First of all, you need to allow at the individual level to create, test, improve and use those skills. Again, it shouldn't be random. It should be structured way.

13:22There's a different tools out there. You just need to decide which tool actually you want to agree on and you use that mechanism at the individual level. The moment you create a skills, you need to be sharing it with your team, right? Your team starts collab collaborating to improve the skills and think about it.

13:38You build the same technology stack, building the same products. So, it's going to evolve really quickly. But then you move on to a very critical point which is the centralized platform. That is where all of what I've been covering so far come

What a central skills platform needs

13:52to play. You need a centralized platform that have a catalog with metadata in it that's actually can discover skills and could be searchable. You can have an MCP that's actually plugged to this cat catalog, search for the skill and a CLI to pull the skills back to your either your IDE if you're locally or to your um sandbox in your factory.

14:09Then you have the dependencies. So you need to understand the dependencies between the skills as well. You have the versioning uh and and life cycle. So you understand which version of the skills is actually the latest. Um and a very good example when I'm using for example building a functionality I can the agents automatically up capture that there is a latest version of the skill and pull it right.

14:25So this versioning help also to pull the the the right latest changes from the skills registry access control again as I said if you don't know who is accessing what that is a huge uh uh gap and finally evaluation observ observability and then all of this is actually play around a governance and this is where technology stop solving the problem right so you figure out all of this all good now who's going to govern this and that is where dep you know it really depend how your organization is structured today that is where you should have your architects,

15:02your engineer leads, infra leads etc and cyber leads actually sitting down owning part of those domains and making sure that the skills we need to get updated is actually according to the policies you want to adhere within your organization and drive this change.

Governance, and who owns which domain

15:17And finally when you get this right what you will have you would have at the organization level all of your teams pulling from one centralized place high quality skills executing them and pulling them back um to to the centralized platform if it is improved.

15:32Now what I want to bring this because it's a little bit of a inclear view. So what I created I created a simulation right. So think about this. This is your organization today, right? And what I have here, I have um just a random teams. I have 15 teams created um five to 12 like per team.

15:48Uh you have skills per engineers contribution. You have the average skills utilization kind of like on average like how much time uh skills are being pulled per day, the duplication across the team kind of as a ratio and the skills quality and security ratio.

16:04Now if I run this across six months what's really happening and think about it this is already happening within your organization is teams are creating and using the skills right but we don't have visibility and skills again they're tightly coupled to your productivity uplift if for example the example that I shared earlier on the regulation if we don't have a skill about the regulation that is someone is vip coding back and forth and trying to figure out exactly how to steer the agent to implement it properly right

Simulating fifteen teams over six months

16:34that is burning more tokens from one side costwise but also the productivity is spending more time rather than giving in one shot the right answer and the quality and security is similar if you don't have clear you know skills defined and maintained you will have a low quality in your implementation because then it's up to the human to decide this and different the m in the maturity from team to team you can see the difference and that is why for example if I look uh

16:59just randomly at this this is like you can see the productivity of this team is a kind of a medium right If I look at this one is a is a bit of like I don't know it's low medium productivity quality and security also medium but when it came to the cost is really high.

17:15Okay, now let's actually say okay how this looks like if I governed all of the my skills in my in my organizations. What's going to happen is of course some of them will split right and this is reality isn't going to be perfect um as we expect but at least what you will see you will see actually some common ground across all of your teams.

17:32The moment you govern, you publish one skill, the next engineer trying to build a new skill, the coding agent harness will identify this skill that is already available and pull it. Right? So you you almost solve all of the issues that that I covered about the governance.

17:53And one last point is when it came to skills, um it's just one component of your workflows as I said, right? So that

Applying the same idea to whole workflows

18:00doesn't mean uh if you figure out skills that says you're good. No, you need to apply the same kind of like approach and solution for your whole workflows. And you may think to apply this again if you think about it like if you have a a centralized platform that have all of your workflows, right?

18:09From one side you're centralizing the workflows which is also having the skills but also if the next engineer came and want to I don't know like provision infrastructure they can tap into a workflow and build that workflow with the required skills and run it and test it again.

18:24And if it's something you need to be improved in the workflow, you can easily push it back um to the centralized platform for your organization to use. Now before I wrap up, um what I want to leave you with is this is just the start the beginning.

18:40You see like this just we're talking about six to eight months. What's coming next and I would invite you to already explore is skills registry, right? You should have one if not already. And the good news is all of the players that's been solving the IDP problem like um internal developer portal they already start centralizing this capability right so if you don't have it today maybe in a couple of months you will see it coming but also there's a lot of tools that's actually solving this specific problem

19:13second is skills eval evaluation there's still kind of like a discussion on what is the right approach to to to you know to to evaluate skills the easy thing that I found so far very valuable is actually test like you you l static test your or evaluate your skills against the entropic best practices.

19:21Right? If the

Registries, evaluation and auto evolving skills

19:30skill is not invoked properly, if the skill is not structured properly, there's a high chance that it's not going to be high quality. And finally is auto evolving. And again, this is what everyone kind of like is is the next hype right now.

19:37Like yeah, I can create like a an a closed loop that can evolve automatically my skills. So what right? If you automatically start this machine, the impact will be way more than it is today because what I shared earlier is going to be just maintaining auto evolving skills without that governance in place that actually put the guardrails for your organization.

20:03And at this point I would leave you here. Thank you so much for your listening and looking forward if you have any question I will be in the leadership lounge. Feel free to grab me. Thank you so much.