Episode 382 Deep Dive: Quinton Anderson | Trust the System, Not the Agent – Zero Trust for the Agentic Enterprise

Quinton Anderson [00:00:00]: If you rely on individuals, it's brittle. Things will go wrong eventually. If you create a system, a sociotechnical system where agents are checking agents, humans are checking humans in a way that scales, the system becomes resilient to those kinds of failures over time. VO [00:00:17]: From KBI Media, I'm Karissa Breen, and this is KBKast. KB [00:00:24]: My guest today is Quinton Anderson, co-founder and CEO of AigentSphere. Quinton spent years running large technology teams and now builds the control systems that keep AI agents honest. We get into what it really takes to build that kind of resilient system and why human-in-the-loop quietly trains people to rub a stamp, plus why the agent is never the thing you trust and where the blame lands when one gets it wrong. VO: If you find these conversations useful, Hit follow. It's the single best way to make sure the next one lands right into your feed, and it helps other execs find the show. Alrighty, let's get into it. KB [00:01:07]: So, Quinton, I want to start with your view on human in the loop. Now, a lot of people I've been interviewing lately talk about we have to have humans in the loop, but your view is it's counterintuitive in the sense that Human beings are naturally creatures of habit, and then we're conditioned to just accept certain things, perhaps not challenge it as much. So walk me through the thinking here, because so many people are saying we need the human in the loop, and you're sort of saying an opposed view. So I'm really keen to start there. Quinton Anderson [00:01:39]: Yeah, it depends on how you define it and what you mean by it. And I guess this is the problem with terms like this, is that everyone starts to project their own meaning into it. But we prefer the term oversight, human oversight, as opposed to human in the loop. In the loop tends to imply a per-transaction and repeated action. And so if you've got a human being asked frequently to review small tasks, what they learn, like you say, humans are creatures of habit. What humans tend to do is learn to say yes and get worse and worse at review. This is also true. It's not just with agents, it's just with any repetitive task. Quinton Anderson [00:02:17]: This is just the nature of the human condition. So oversight is more about saying, do you really understand what this thing is intended to be doing? Can you check? Will you know when something is starting to go wrong and how often do you do periodic checking? That periodic checking is specific and intentional. So oversight is different than being involved in every little thing and more Taking the time to understand how to apply oversight. And then obviously there are going to be cases where you have to be in the loop. So when you've, when the agent needs to escalate, say the value of the transaction breaches a particular level and the agent no longer has a mandate to conduct that action. KB [00:02:58]: So I have a question then on that, because of how AI is now, like people are just getting lazier with knowing certain things and getting the answers are relatively easy. to obtain. Do you think if the human in the loop was just there in isolation and there was no other AI element, do you still think people would just be like, yes, yes, yes, yes, yes? Or do you think because of AI and how people are moving now with just people not necessarily struggling with like reading and writing, like even younger children apparently in places like in Europe are doing studies on this. Do you think that they're connected or do you still think regardless people would just still— accept and not think through it in any fidelity? Quinton Anderson [00:03:37]: I don't think it has anything to do with agents per se. Obviously, there's the broader psychological effects of social media and use of AI where attention spans are reduced, et cetera. Maybe that plays into it. I don't know. I'm not aware of any particular study that has created that linkage. It's more to say when humans do repetitive things, we learn to just say yes, or we get trained. to take particular actions repetitively and it almost becomes subconscious or routine. That's from a control perspective is dangerous. KB [00:04:08]: Yeah, this is a good point because I mean, like even go back to, I don't know, like 10 years ago, we used to do things so manually, right? And so do you still think that people were the same level perhaps of accepting things? Because again, people, people will then argue and say, yeah, but Quinton, people making mistakes because they were fatigued or they hungover that day and therefore they still accepted the thing versus We can sort of get a machine now to do it. And yes, there's some intervention, but perhaps they're not as fatigued because they're not doing everything from like the ground up, for example. Or do you still think that regardless, it's the same no matter which way you look at it? Quinton Anderson [00:04:45]: So maybe use an analogy. So if you think about a production line, so go back into the 1930s and think about a production line and have someone check every single item coming off the line versus sampling. What you find is the people who are doing it from a sampling perspective will tend to zoom out and look at the whole system. They'll start to go and check the process by which people come to work. They'll start to check the maintenance on the machines. They'll start to check the behavioral and communication aspects between team members, and then they'll do periodic sampling. And what they're looking for is how well the entire system behaves. And if you contrast that to checking every single item coming off the line, And doing that as a human, our ability to spot differences, our ability to apply a critical lens just degrades over time. Quinton Anderson [00:05:36]: So again, I don't, I don't think it's anything to do with AI necessarily and more to do with the human condition. And we also need to recognize that we've gotten good at making things safe by applying a systems thinking approach to it and not just direct rote interactions. KB [00:05:54]: Okay, so you made a great point around people's ability to spot the difference is going to degrade. So obviously, is that going to get worse now? Because like even now, like things are getting worse for people and understanding certain things or becoming lazier in their thinking and critical thinking. So I'm assuming then like maybe in the next like couple of months, like it's going to be worse than you and I talking today from what you're sort of saying. Quinton Anderson [00:06:17]: I think there might be a temporary effect like that. I don't know, you might've heard There's a lot of the rhetoric around AI is coming for your job, but I think what the data really shows is people who know how to use AI are coming for your job. So you're going to have a period of time potentially where people are getting lazy, not really understanding how to interact with these new types of entities within their day-to-day work. And people who don't know how to apply judgment, communicate well, learn quickly, and therefore be productive in terms of oversight with agents will slowly find their way out of the workforce or will slowly learn better routines and patterns. So, I think there's a risk of it degrading, but not sustained. Organizations will continue to have to be productive and increase productivity and deliver services safely. And large complex organizations are pretty resilient systems. They'll find a way through. KB [00:07:12]: Okay. Operative word, productivity. So, We went through this sort of phase around, oh yes, we're gonna leverage AI, we're gonna be more productive, which I understand. But like now I'm talking to people and saying, yeah, but we've been looking at our tokenomics and perhaps the person that we thought a machine could do is actually creating, is actually seeming to be more expensive to use the machine now. So when it comes to productivity, do you think that again, going back to the human in the loop, is it doing perhaps a disservice? Because just say someone accepted something they weren't supposed to and then they have to go, we have to roll the thing back, or we have to go and fix the thing versus perhaps of just doing a little bit more manual work, it would have been a better outcome. So I'm just more curious on that because now it's like everyone's like, we've increased our productivity, but now it's costing us more money. Other people are now saying, well, it's costing us a lot more time to set this stuff up, whereas we should just have kept a human there to begin with. So I'm sort of seeing like people saying that now in the market. Quinton Anderson [00:08:12]: And I think that's part of a learning curve, right? If you think about normal hype cycles, We are going through that trough of disillusionment now where it's not that there was anything wrong intrinsically with the technology, it's that we are learning where the value is gonna come from. And it's a pretty normal pattern. I'll talk a little bit later about this concept of a sociotechnical system. The first time the term came on the scene was back in 1951 where they'd done a range of industrialization around a coal mine. Having added all the machinery in and industrial automation in, They weren't getting efficiencies through the mine. And they did a study off the back of that and kind of came to the conclusion that if the technology gets too far ahead of the rest of the organization, you actually remove value. So we're potentially at that stage right now where the technology's gotten ahead. And we will now start to figure out where does it generate, where doesn't it? And we'll be a bit more nuanced around which sorts of use cases are valuable for an agent to do on its own, which sort of use cases Is it just augmenting me, you know, knowledge management, search, et cetera, versus where we can delegate end-to-end outcomes? And then we'll start to see the actual outcomes as opposed to just use cases. KB [00:09:29]: So then on that note, I just want to pivot and shift gears just slightly and going back just to the human in the loop, just so we can really explore this in detail. Now let's start with that and then perhaps the loop for, you know, argument's sake. Okay. Companies are already struggling though to govern their human workforce. We've seen this over time, you know, all these issues, whether you're a large enterprise or not. But now with everything that you're sort of discussing and what's happening, because everyone's like, oh, we gotta put guardrails in, we gotta do this and deterministic and all this sort of stuff, right? What do you realistically think people should do now, given everything that you're sort of seeing in what you do day to day, et cetera, and what you're saying around Well, is the human in the loop as effective as we thought? Where do you sort of see things moving from here? Quinton Anderson [00:10:18]: I think organizations, so I'll talk about larger, more complex organizations first. Organizations have to focus on putting in scalable mechanisms. And what I mean by that is safeguards are important. And if you implement them a particular way, You don't think about emergent risk over time, control planes atrophy, the controls that you have atrophy, and slowly risk creeps in and the organization is impacted or customers impacted as a result. What do you need to do to make the mechanism scalable? So, control planes are good ways for technology that you basically delegate to the control plane to make sure that the desired state of the system remains in place. You've got to have management systems, so you can't think about technology and the social constructs independently of each other anymore. Back to the point about sociotechnical systems, you've got to think about the whole organization because it's effectively the humans in the organization that are dictating agent behavior through prompts, through knowledge management, but also through agent buildout. And so having the management systems in place that help humans understand their accountability, help them understand how to bound authority for the agents that they're accountable for. Quinton Anderson [00:11:34]: Help them understand how to manage agents. It's similar in principle to when you go from an individual contributor into a management role for the first time. It's not always intuitive. Like, you need to be taught what performance management looks like, what oversight looks like, how to trust your team and create a psychologically safe environment, but also verify their results and strike that balance. And so we've got to think about our management systems and how the humans effectively project, broadcast, cast a shadow of their culture into the agents. And then I guess the third thing is we really have to fix the basics. I've been always amused by, like in my career, I've done a lot of work as leading large technology teams and getting organizations to care about patching has been one of these big bugbears of mine. And I know many CISOs feel the same way. Quinton Anderson [00:12:28]: But in the last 6 months, you know, things like the Anthropic announcements around Mythos, et cetera, listening to CEOs now talk about patching as being important. And that's just one example of where you need to get the basics right. All of your recovery mechanisms, so crisis management, incident management, et cetera, you need to have those basics in place and not in a way that's tantamount to hope. It's not about doing it once and saying, yep, we've ticked the box. It's about actually exercising those mechanisms and And convincing yourself that you can recover quickly. KB [00:13:01]: Yeah, and you're right about the patch stuff because we've been talking about this for 20 years. If you read theory books on it, it looks easy, right? But it's hard to do it when, especially when you've got big bureaucratic companies with so many different, very old systems, for example, as well. So it's not as easy. It's easier said for you and I to talk about it than to do. So do you think it had to be something like Mythos to become the catalyst now for people to care? Because like, we've been talking about this stuff for years, like, oh, we could do better patch management. And like, while some people understand that, like, no one's really taken it seriously. Do you think that now, because of the Mythos stuff, people are freaking out more than before? And it's like, well, we gotta take this a bit more seriously now because there's no real answer to this problem. Quinton Anderson [00:13:45]: Yeah, I guess the patching stuff never got to the place it needed to because The data didn't support it in terms of impact. So, you know, as an industry, as engineers, we care about patching because we know how vulnerabilities ladder up over time to open up particular attack vectors. And we've been, you know, calling out loudly, as you say, for 20 years about the importance, but organizations haven't seen continuous floods of cyber attacks. And so, it always happens to someone else, right? It doesn't happen to me. is one, another one of those human behaviors. But by and large, organizations haven't been impacted at an individual basis. So they've seen lots of other companies get impacted by cyber, but you talk to a lot of CEOs, they haven't been through a cyber event themselves. And so what Mythos has done is basically said the likelihood of an attack has gone from, you know, 20% in a given year. Quinton Anderson [00:14:41]: Now, whichever bad actors are out there have the tools to basically guarantee that they're going to get in at least once a year. And that's a very different prospect for a CEO to be sitting on, right? If they're looking at it and saying, I can spend $30 mil on patching, but the probability of a cyber event is quite low in the next year, I can reduce that a little bit. If you say to a CEO, the probability of a cyber event in the next year is almost certain, they'll go, what do I need to spend? Because the brand reputation alone is potentially ending. KB [00:15:13]: You're right. So do you think, and I've spoken about this on the show before, do you think unfortunately we had to, you word before around impact. We have to see that impact of companies constantly just getting hit, hit, hit, hit, hit, data breaches left, right, and center. Now the whole Methos thing, that's really added to it because, you know, unless there was people were severely like dying from these breaches and like, I don't know, you and I worked in a bank together. It's like one thing for someone to get their money stolen, you get it back, but you can't really get someone's life back, for example. And there's people that I've spoken to over the years that have said, until we get to that point, and this was way before Methos was ever on the radar. They're like, we probably will have to get to that point, KB, until people start taking it seriously. But now because of this, is it more that they can see it now and practitioners are saying, well, there's no real fix to this at the moment. KB [00:16:01]: We're just going to have to deal with what's coming. So do you think it's more that unfortunately we could have, like you said, could have spent $30 million on this sort of stuff, but until people can really see it, they're probably not going to move as fast. Quinton Anderson [00:16:14]: There's a whole bunch of like industry waves that get created around these things where impact is not necessarily assessed. Single thing. So to give you an example, for a lot of jurisdictions, the government has been very clear with critical infrastructure that they can't take using agents for attack vectors. They have to take it seriously. So if an actor's got access to a Mythos-style model and the open weight models are getting really, really powerful too, then you need to be able to withstand an attack from such an actor. And if you can't show us that you can do that, you're effectively representing systemic risk. And so you start off with your critical industries, you know, military, banking, food security, so on and so forth. And they employ a huge number of staff within their organizations, but they also have a huge impact on the supply chain upstream of them. Quinton Anderson [00:17:08]: They will start to put constraints on organizations that supply to them. And so you get this wave of pressure to do the right thing or to fix some of these basics or to take these things seriously. You know, maybe in safety industries, events do occur. I hope not. But you can also get systemic risk emerge in other ways. So, like the financial system is a classic one. You look at the impact of algorithmic trading and how quickly we went to highly regulated postures around algorithmic trading because of the systemic risk that it introduces. And once people start delegating payments to agents, Think about the range of ways in which the fiduciary system can get unstable. Quinton Anderson [00:17:46]: So systemic risk will start to get realized or lead indicators will start to crop up on that front. And so if you think about traditional payments and programmable money moving at the speed at which agents can interact, there's quite a few ways in which the systemic risk is going to rear its ugly head shortly. KB [00:18:04]: Okay. I want to know when you said we have to take it seriously, like companies have to take it seriously, what does that actually mean? Now, what I mean by that question is, for example, companies out there will say, we take it seriously, we invest all this money, we get all the top people in there, all these, you know, expensive people working here, and then something happens and they go, but we took it seriously though. So do you think that, like, what is that? How do people, like, know and define that? Because to the outside world, it'd be like, well, you didn't take it seriously because you had a breach. Or some people will say, well, I've been in now in 3 Australian breaches thanks to X, Y, and Z companies. So do you think that this whole, and that sort of ties into the whole trust thing? Oh, you know, we have to engender trust and all that sort of stuff. Do you think though, there's a little bit of virtue signaling in that? Yes. Whilst we can say people invest in $100 million into this sort of stuff, whatever, but does that then define a company as taking it seriously in the eyes of a consumer, for example? Quinton Anderson [00:19:00]: So I guess it's one of these complex things, right? Because you can protect against thousands of attack vectors and it only, it's only the one that you didn't think through or didn't cater for that succeeds. So it's, it's one of those classic game type scenarios where the definition of good enough constantly lifts. And so people taking it seriously, like regulators and boards have always taken it seriously. The minimum level that they had to meet higher now than it was a year ago. And so they've taken accountability when they, when a lawyer applies a reasonableness test to them. So say a breach has occurred and they're being prosecuted, the reasonableness test is applied at a particular level. Did you hold the right people to account? Did you define controls? Did you assess risks? Show me evidence that you did all of those things. Then events like this occur, or the, basically the water level raises and people are asking for more specific controls. Quinton Anderson [00:20:04]: So, it's not that you put a vulnerability management program in place, it's that you can show that your vulnerability management program patches 100% of vulnerabilities within 7 days. And so, historically, the expectation of reasonable would've been you have done something and you have checked it. And the new definition of reasonable becomes, and you can show that 100% are done within a particular point in time. And so that's what I mean by people taking it more seriously is that the level, the test for reasonableness goes up. KB [00:20:39]: Yeah, got it. And I think in contract law, the phraseology is reasonable endeavors. So did the company take reasonable endeavors to make sure, you know, whilst it wasn't successful or was unsuccessful and they got breached, they took the right measures at least. So that's part of the audit and all the things that companies then have to go through. Quinton Anderson [00:20:59]: Correct. And if you go back 5 years ago, the reasonableness test would have been at a particular level. Today it's at a different level and smart boards know that and regulators are certainly pushing it. KB [00:21:11]: We'll come back to that after a quick word from our sponsor. I'm known for being direct. Let's be honest. Nobody gets into technology leadership for the compliance paperwork headache, but if you're building or scaling a tech company, Security frameworks like ISO 27001, SOC 2, Essential 8, CPS 234, or GDPR aren't just tick-box exercises. They are business critical. That's where Vanta comes in. Vanta automates up to 90% of the work for security and compliance, helping you get audit ready in weeks, not months. It integrates seamlessly with your tech stack so you can spend less time chasing documentation and more time leading innovation. If you're a CTO, CISO, or head of security, it's worth taking a closer look. Visit Vanta.com/KBKast, Vanta.com/KBKast, to learn more. KB [00:22:11]: So I want to then ask you about— you advocate zero trust for agents. So if an organization then cannot trust an agent, for example, and can't fully test it and can't always explain the decisions. I know we touched on that a little bit before. Why should people even bother to delegate meaningful authority like at all then? Because again, it's like, then it goes back into that trust piece. Do I trust this agent to do the thing? And then, you know, I'm hearing compromise agents because agents are like, I need to do this thing as fast as possible. And if I, you know, railroad the other one to get the thing, but then that causes other issues like. KB [00:22:48]: I really want to get into this because I found this quite interesting. Quinton Anderson [00:22:51]: So I guess you've got to treat the question with the right amount of nuance. So if you ask a question like, or abstractly, we ever trust an agent? My answer to that would be, you don't trust agents. You shouldn't, but you can trust the system. Complex systems are quite resilient. And if an agent exists within a complex system, and I'll come to in a minute how you need to think about that. Then the agent itself doesn't need to be trusted, but the system can. Versus if I look at it in with nuance at a particular use case, you should ask the question for a particular use case, should I use an agent here? And that is part of the system being healthy, is looking at a particular use case and saying, is it absolutely necessary or is it even in principle necessary to use an agent here? And the answer might be. Well, no, I can use a coding agent to build some code and I don't need an agent ongoing here. Quinton Anderson [00:23:50]: I can just have a workflow that a coding agent has helped me build fairly quickly. And that is now a lot cheaper than it used to be or easier to do than it used to be. So at a micro level, in a particular use case, you should always ask the question, but in the abstract, the answer is absolutely, we should embrace agents because we can make resilient systems, not individually resilient agents. KB [00:24:14]: So do you think, given your experience, and you know, when you've got 50,000 people in enterprises, there's average people in there. Do you expect that people have the capability to ask those intelligent questions on their own, to really think through scenarios and do we need an agent here and all that sort of stuff and walk through sort of the risks of this? Or do you think people, to your point at the start, are just on autopilot, not we just need to roll out all of them all the time? You know, there's companies that I'm speaking to, they started with a few and now there's thousands being rolled out because they think it's good. Whether that's good or not, time will tell, but it's just, again, going back into the thinking. Quinton Anderson [00:24:53]: I think a lot of organizations talk about how the most important thing is culture. Culture eats strategy for breakfast every day of the week. And if you lower your hiring standards, then you'll end up with a particular type of culture. But as organizations scale and mature, that They also always need to accept that they have to put assurance and oversight mechanisms in place. The average quality of the people inside the organization goes through ebbs and flows over time. Humans are fallible, make mistakes, and they're now interacting with other fallible entities in the form of agents. So safety for an organization, if you're sitting in a director's seat and being asked the question, how do you know? Can you give me some degree that you've, you know, have you taken reasonable steps? The only way a director can answer that is to say, we've put controls in place to hire the best people, to give them the best agent platforms, to train them on how to use them. And we've done oversight. Quinton Anderson [00:25:55]: So we've got platforms to oversee how the agents are meeting controls. We're doing independent testing. We've got humans checking the other humans. And then finally, we've got auditing or assurance functions, which check the checkers. And those layers of defense are the things that help you protect against emergent risk over time. So the quality of hiring goes up and down. That's okay. The system can withstand that because you've got multiple layers of checks. Quinton Anderson [00:26:26]: The controls atrophy over time. They always do. Even strong and well-engineered controls atrophy over time. But if you've got 3 lines of defense, you know that and you can take corrective action. KB [00:26:38]: So going back to your comments around trusting the system, what if you don't own the system and using like third-party vendor, for example, and whilst you have every intention, reasonable endeavors, all the hoo-ha, something goes wrong, which inevitably does. And I know there's the whole conversation in the market around the whole accountability thing, et cetera, still falls on the company at the end of the day. But talk to me about that, because then you have a lot less control of what an external vendor does, whether you agree with it or not, and it's not as easy to just rip it out and replace it. There's a bit of a process to it. It's complicated. So, I'm just curious to understand how does that look in your eyes? Quinton Anderson [00:27:13]: So, it might be worthwhile clarifying what I mean by system there, because what I'm trying to get at is systems thinking. And the question of what is the bounds of the system that we're discussing is always an important question. So you can think about system as in AWS Bedrock or a particular agent that does supply chain optimization or order handling. That might be the bounds by which you think about system. When I talk about system, I'm meaning the whole organization, including all of its suppliers, its people, its agents, and its technical platforms. That is a sociotechnical system. So think about the social constructs, oversight, accountability, incentives, culture. You've got to think about the technical aspects. Quinton Anderson [00:27:58]: And before agents, those 2 things were relatively separate. You had a requirements document in the middle, or you had a procurement process in the middle, or you had an agile ceremony in the middle, and people would implement technology and the technology would largely behave the way they intended it to. Effectively, work is being done by technology and people together. And so you've got to think about the system as a whole, the people and the technology together, not just system as in the technical artifacts. And so how you think about incentives for the humans, how you set up your organizational structure to ensure that humans are applying oversight is part of the system design. If you don't design it the right way and you don't give the right members of the organization the tools and platforms they need to do assurance and oversight and red teaming and pen testing and all those good things, then you are going to have failures over time. You haven't designed the system to be resilient. That's what I mean by a system. KB [00:28:56]: Got it. So it's like a zoomed out view and sort of collectively looking at everything. So where are people at now at this stage? Are they like, okay, well now to your point, we have to look at everything holistically and start to look at the people and all this sort of stuff and the architecture, perhaps the suppliers that we're using. Maybe we don't need certain things anymore. We can scale it down, decreases our risk, et cetera. So do you think people are in this stage of redefining what their companies look like. But then if you look at like a bank, like look how old some of their systems still are. So like, is it a bit of horses for courses because people are at different stages, other people are a little bit more mature, et cetera? Quinton Anderson [00:29:33]: So I guess there's 2 questions in there and I'll pick them up separately. First of all, a lot of organizations do have legacy technology and it's been a long pain point of end of life, end of support. You know, the disconnect between support costs versus fully amortized, et cetera. But the key thing has always been a prioritization one. It's really hard to justify patching something that, or replacing something that is working well enough when you've got other priorities. What agent-based development has done is allow us to do those kinds of patching, upgrade, and migration cycles faster. So we do have a really powerful tool in the toolbox to deal with legacy technology that we didn't 5 years ago. On the upside, that's a real opportunity for us collectively is to go lifecycle management doesn't need to be as difficult as it was before. Quinton Anderson [00:30:29]: So that was the first part. The second part of the question, are people starting to come to the realization? So we've been, so we formed this organization in July of last year. Our view has been that you've got to think about the total system. You effectively need to have an HR system. System and a risk management system that works for agents and creates a strong accountability link between agents, agent oversight, and the human organization. So think about it as a control plane for the sociotechnical system. And when we first started talking to customers about this with our early prototypes, there was a lot of, oh, that is a great idea, but we're way too early. And it was the similar sort of reflections as we went through the funding process. Quinton Anderson [00:31:13]: The first wave of venture capitalist firms that we spoke to said, great idea, too soon. And so what we found from a customer perspective was some early forward-thinking customers who recognized that they should consider these things from the very first agent. They need to think about not only how they build agents and what the technical platforms look like, but how they're going to assure these things on an ongoing basis. And they wanted to basically prove compliance before going live, prove that they could do continuous compliance before going live. What we found over the last about 6 weeks is the rest of the organization now starting to participate in this. It's not just led out of technology. It's not just led out of the AI team. Product managers are starting to step back and go, well, this gives us an opportunity to rethink the unit economics. Quinton Anderson [00:32:06]: How do we lean into that? The risk people looking at it going, I need the tools now to be able to show these things. I've got a security, a security of critical infrastructure obligation that's being added. How does that work with agents as I've got this growing agent workforce? And so, we're seeing that tipping point where technology moves out of technology teams and into the broader organization. And that creates that trough of disillusionment, but it also allows the organization to start to get real value. And then we, and then we start to see the benefits flow through. And so we're seeing more and more of those conversations now. And that's heartening for us as a business, but also I think a reflection on the industry necessarily maturing utilization of this technology. KB [00:32:52]: Okay. That's interesting. You said continuous compliance, because that would then eradicate the whole tick box exercise because it's continuous. It's forever evolving versus, oh, we get to the end of the year. We've got, you know, the compliance people coming in to have a look at everything. And then obviously we know how that goes. So. Is that going to change the whole conversation perhaps around the innovation towards AI? Because they know that like there's all these other checks and balances that are being done. KB [00:33:16]: Because I've been speaking to people and they're like, oh, we're hesitant because can't do these other things perhaps. And then it stifles innovation, but we don't want to be stifled by the innovation because then our competitors get a head start. So I know it's a conundrum, but it's just more that component of it changes the game then for people and how they approach this. Quinton Anderson [00:33:35]: Completely. Like the path to adoption at scale is understanding where you need to introduce brakes and controls. Confidence, trust, et cetera, is built by investing in and spending time on the safeguards, the controls, the system resilience around it. And then you feel confident to scale. And so it's actually an accelerant. It's the old analogy around Formula 1 cars. You need to invest a lot in the brakes, and that's what allows you to go fast, right? If you can brake late and brake confidently, you can improve your lap times. And it's similar in terms of this landscape. KB [00:34:13]: So then to extend your metaphor, why do you think historically companies haven't really invested in the brakes? Or again, was it, we tried reasonable endeavors to do that and weren't successful? Quinton Anderson [00:34:22]: So I'll stop short at saying organizations haven't, like there are obviously organizations that do invest in the brakes. That tends to be in places where there's a regulatory obligation. So it's kind of their right to play, or it tends to be in places where trust is an important part of their brand value and a loss of trust immediately impacts the bottom line and overall brand valuation. So you will find organizations that have taken it seriously. And obviously governments and defense would fit into that category of They're just obliged to take control of active management of risk seriously. And so again, I think we'll see those sorts of organizations leaning into the space, thinking about the brakes upfront, and some of those will move too slowly. We'll see organizations not thinking about the brakes moving too quickly and we'll normalize somewhere in the middle at a pragmatic level. KB [00:35:19]: Okay. So, Quinton, talk to me a little bit more how agents can't be fully tested then before Deployment. So that then sort of reads as potential behavior and attack space is infinite, for example. So then what do businesses need to do to have that level of assurance? And I know we touched on it before, the continuous compliance, but talk to me a little bit more now through that. Yeah. Quinton Anderson [00:35:43]: Traditional software systems, you test them and, you know, you can argue about what your test coverage levels are, but they're the majority of the state space for testing. So if you think about all of your test cases and all of your attack vectors as some kind of dimensional space that you can visualize, you can probably cover 70 to 80% of that space with traditional testing mechanisms and negative testing mechanisms. When it comes to models, the behavioral space is practically infinite. And so you can do red teaming, you can do functional testing, you can use evals, And you're only going to cover a small percentage of what the behavior might be. And when I say a small percentage, like less than 1, right? It's a tiny portion of the state space. And so, you have to think about it as what will keep me safe effectively testing in production? Because that's what you are practically doing. You can't assure everything before you go live. You need to take a stance of zero trust, so bound authority, Think very carefully about your operating envelope, minimize exposure in terms of access, et cetera. Quinton Anderson [00:36:52]: Then you have to be able to test the entire system. So that is not only the agent and its integration points, but also the humans that are providing oversight for that. How well do they provide the oversight? Like when last did they go in and check how it's behaving? Are there alerts being generated? Do they have an incident management approach that they're using? Are they producing evidence that can be used in an audit or an assurance process? So when I say test the system, you've got to think about the system, not just the agent and its technical integration points. Then you need to do independent verification. That again, coming back to that same point, and then correct quickly. So understand that things are going to go wrong, block where you can block through preventative controls. know when something has gone wrong because the right humans have been brought in at the right time to provide oversight and they've taken corrective action. And if you keep going through that cycle, you converge on a good enough state over time. Quinton Anderson [00:37:53]: Okay. KB [00:37:54]: There's a couple of things I want to ask you. So when you say test the system in totality, not just the agent, do you think companies get that though? Or do you think they're starting to get that concept? Quinton Anderson [00:38:04]: Starting to get that concept. And again, as it moves out of purely a technology play, like the mind trick here is it's not traditional software. You can't just go to a central engineering team and say, build me a solution. What you're effectively doing is saying, onboard a new type of worker. And now I have to lead, coach, and manage it on an ongoing basis. And that mindset shift is an important one. KB [00:38:28]: So then on the mind shift note, You said before, correct quickly. But as we know, in companies, things would take months, years to get sorted, looked at, corrected. So how does that— and I know we touched on before about the whole cultural thing, you know, even companies doing that risk review, all the tech risks, you know, line 2 risks, business risk, all these people not agreeing or agreeing or whatever this outcome was. So how— that's like counterintuitive to what we're sort of saying now. So how can these big businesses— get to a point where we're not going to be able to do 500 meetings and take a year to make decisions, guys. We need to make a decision quick. So how do you do that in a big beast of a company that really can't make decisions fast at all? Quinton Anderson [00:39:12]: And this is where agents are actually part of the solution. So you can think about agents also being part of the oversight and assurance process. Agents are very good at doing exactly what they've been instructed to do. And if you tell an agent to do Let's say customer onboarding or outbound campaigning or whatever, write code, review code, et cetera. You can also create agents whose purpose in life is to review, and that creates a natural buffer. And similar to prior to agents, how we had separation of duty within an organization, you're allowed to do a thing, you have to check it, or you can't complete a particular action unless another person is involved. And those are system-level controls that you can put into an organization that makes sure that if there's a malicious actor, a human, it's really hard for them to get away with it. And if mistakes are going to be made, those are minimized through 4 eyes, 6 eyes review processes. Quinton Anderson [00:40:16]: And so you can think about building your organization, your agents, your digital workforce in a similar way. You can add resilience into the system. So if I've got a line 2 or an HR person, they have the ability to build, configure agents that monitor other agents and our platform specifically to do that kind of thing, provide control functions within the organization, scalable means to do assurance, to do oversight. And so while agents represent a problem, they also represent an opportunity in that it scales coders, it scales business processes, but it also allows you to scale the assurance functions and make the brakes better, coming back to the racing car analogy. KB [00:41:00]: Okay, so what's coming to my mind, maybe this sounds rudimentary, but I'm going to ask it anyway. Just so hypothetically, you're working through something, you've got these agents, they come up with something, play devil's advocate, maybe it's opposite of what I think. Do you think we're going to see instances where humans just override the agent to be like, yeah, they, For example, they wanted the wall green. I think it should be blue, but therefore I've got an ego and I'm upset about it. So I'm just going to override the agent. Are we going to start to see that sort of scenario get played out now? Quinton Anderson [00:41:30]: I'd be very surprised if that isn't playing out now. And I'm not certain it's wrong either. It's really contextual. That kind of conversation is to, like, you can't normalize over it abstractly like we are now. It's use case by use case. So take a coding agent example of, I'm taking this particular design approach and the human says, no, there's some context that you don't have and I'm making, this is the choice I want you to make. And that's important because humans still apply judgment, but also agents have finite context. And so humans being able to say, no, I'm overruling you is potentially correct. Quinton Anderson [00:42:09]: It's also potentially incorrect in the human applying bad judgment. You know, internal biases, et cetera. But again, the answer to that is the system as a whole, the individual. If you rely on individuals, it's brittle. Things will go wrong eventually. If you create a system, a sociotechnical system where agents are checking agents, humans are checking humans in a way that scales, the system becomes resilient to those kinds of failures over time. KB [00:42:37]: And do you also see the agents becoming like an adjudicator, sort of to be like, if you and I can't agree on what should happen, Do you see scenarios where people fall back on the agent? So I'll give you an example. So we're sitting historically, security issue, sitting with the, you know, service owner and the project manager and all these people. No, you won't be able to go to live with your project because of these reasons. These are too high, the risks. Then they'll call up John, who's been there for 40 years, and he'll say, no, I'm accepting all the risks. John doesn't know what he's accepting. So in the eyes of a security person, it shouldn't have happened, but then we get overridden because it's like, well, this guy wants to get his bonus for the year. He's just going to accept the risk. KB [00:43:11]: And then who cares? Someone else's problem. So in that scenario, technically it was probably right, the security team saying, no, we can't go live with it. But then you've got other people, to your point around biases and people have got their own ulterior motives, or we have to have this thing live, et cetera, KPIs. So I'm just looking at that as a real scenario. Would then potentially, would people then fall back to the agency to be like, well, what do you think as a relatively independent party on this matter? Like you would in any sort of like mediation. Quinton Anderson [00:43:40]: Yeah. KB [00:43:40]: So that's why you have a mediator to be like, we have opposing views. So like, do you see that as a potential thing playing out? Quinton Anderson [00:43:46]: Absolutely. And again, I think it's going to be use case specific as to how much you accept the mediation of an agent versus use agents and LLMs as a way to bring more facts and research rapidly into a conversation. You know, in any given mediation, quite a lot of complexity. Those biases, but ultimately if the system's working well, if people are focused on the outcome, what they want to do is find data to confirm or deny their position. And quite often disagreements occur when you can't find the information, then it's just a judgment call. And then it's more down to style than it is rational thinking. And so agents bring quite a lot to a mediation context in that You can do research quickly. You can gather information quickly. Quinton Anderson [00:44:39]: You can make better informed decisions. A simple way to think about it right now is you've got that Teams function in ChatGPT. I don't know if you've used it before, but you can add multiple people into a chat. And it's this really interesting dynamic of like grab a friend or a colleague, add them into the chat and have a debate. And then just watch ChatGPT intersperse with fact-checking. And see how much better the debate goes than if you just have a debate between a friend and a colleague directly. Because you've made this assertion, here's the independent research, and you can then build on a fact base within the conversation as opposed to just bias. KB [00:45:19]: That's interesting. So then do you envision that it's kind of like when people present, like they're in a, like in a trial, right? So it's like, okay, here's a jury, independent people. Your attorney says X and Z, and then, you know, they're basing all the facts and the evidence and the information. So these people make an informed decision versus, well, I don't like that guy's shirt. He sucks. He's definitely guilty. Um, and it could have gone the other way. So is it more like, do you envision that people will go, actually, you know what, now that I have that information in front of me and you spell it out like that, it makes sense to go with this decision. KB [00:45:51]: Whereas in the past we didn't have that as an option. Quinton Anderson [00:45:54]: Correct. The scientific process, you know, the scientific intellectual posture that we've had since circa 1700s is science works because you submit your idea to a marketplace of ideas and try and prove you wrong. And if it survives enough criticism, then it's supported by facts. If it survives enough criticism, then the idea becomes broadly accepted. And that system work, obviously there's entwined interests in that and politics can influence science. And there's that rich history from people like Kuhn and Popper that explore this space. But what's true is the easier it is for people to present facts, the easier it is for them to have a rational conversation and make a good decision. And agents are extremely useful in that context. KB [00:46:45]: So Quinton, what do you sort of think moving forward? And I know that like, obviously it's still relatively early days, people are still figuring it out, like There's no like blueprint. What do you sort of think moving forward now for the next like couple of months into sort of 2027 in terms of like how things are going to unfold? Quinton Anderson [00:47:02]: So I think that first theme of definitely moving out of technology teams and the whole organization now engaging in it is going to be a key theme. 2, there's going to be a lot of focus on making a resilient system. So the whole organization, not just a particular technology platform. And that's going to have organizations re-questioning accountability. What is HR's role in this? What is legal's role in this? What is product management's role in this? And starting to evolve hiring practices and training away from skills-based and more towards generalist experts in terms of training and hiring people who are inquisitive, learn quickly, communicate well. Know how to use the AI tooling are going to start to outperform, for want of a better word. I think beyond that, we're also going to start to see hypotheses land in terms of unit economics changing, business models changing. So we'll see a number of AI-native organizations get to real scale. Quinton Anderson [00:48:07]: And then people will start to look at those entirely novel ways of thinking about organizations and go, Actually, this is where the unit economics is changing. Productivity won't become how do we remove a person? It will be how do we rethink supply chain completely? And that is where the real productivity is going to start to come from. It's not doing the same thing for cheaper. It's rethinking the value proposition entirely and the unit economics as a result. KB [00:48:40]: That was Quinton Anderson, everybody. What's staying with me is his line that the brakes are what let a Formula 1 car go fast, and the same holds for agents. The companies that spend on oversight first are the ones that get to scale. If you're a CEO listening to this, stop asking whether you can trust the agent and start asking whether the whole system around it can catch that agent the moment it could be wrong. VO: I read every reply. If you've got some thoughts on this one, send me a message on LinkedIn. KBKast, Cyber for the C-suite.