Episode Transcript
Mark Thomas [00:00:00]:
The traditional model says, hey, did the model give me a bad answer? Okay, that's what we were asking just a few months ago. Today, I'm asking, did the system do a bad thing before I could stop it?
VO [00:00:13]:
From KBI Media, I'm Karissa Breen, and this is KBKast.
KB [00:00:16]:
My guest today is Mark Thomas, an IT governance and risk veteran and ISACA Hall of Famer who runs Escoute Consulting and sits across more boardrooms in a quarter than most of us see in a year. Adoption is sprinting. Governance is still tying its shoes. We talk about why a policy in a folder isn't a control, who actually owns the blame when AI executes, and the moment nobody has a clean answer for it yet.
VO: If you find these conversations useful, hit follow. It's the single best way to make sure the next one lands right into your feed. And it helps other execs find the show.
KB [00:01:00]:
So, Mark, welcome back to the show. Would you say organizations are adopting AI faster than they can govern it? And are boards mistaking a policy document for an actual control? Now, I know recently there was an event with ISACA. You came and you spoke to me for about 20 minutes, but now I really want to get into this a little bit more fidelity because I do think it's a big topic and one that perhaps doesn't have a lot of answers around. So, I thought, What a better person to ask than yourself, Mark Thomas.
Mark Thomas [00:01:27]:
Agreed. Agreed. And, you know, ISACA just did an AI Pulse poll recently, and I want to kind of throw a few statistics in there from that as we walk through the next couple of questions. But you asked 2 questions in there, and my answer is yes on both counts, right? Adoption. Adoption of AI is sprinting. And I believe that governance is still trying to lace its shoes because too many boards are really confusing say, a signed policy with real operating control. A policy says, this is what should happen. When governance proves it actually does.
Mark Thomas [00:02:01]:
So I think the real board test isn't, do we have a policy here? It's, would anyone notice if this was violated? So I like to say a policy without operating controls is like a gym membership. You own the card, and just owning that gym membership card doesn't make you fit. Owning a PDF doesn't make your AI well-governed. And we saw a lot of the responses in the Pulse poll that AI adoption continues to accelerate every industry. We found that many organizations are still in early stages of governance despite this widespread deployment. And I think policies, as we think about policies, is you see a lot of organizations that might be in the news. I think you and I might have talked about Air Canada last time, right? Air Canada had policies around AI. And it's a famous story about an agentic AI system making decisions that the organization said, we don't have a policy for that.
Mark Thomas [00:02:54]:
And this chatbot still invented some bereavement fare out of thin air. And a tribunal made the airline honor that whole idea. So I think you and I were talking about my own digital twin environment. And I wrote the usage rules on day one. The policy took me about an hour. The actual governance, though, is the habit afterwards. I think that's kind of a big deal. And what we always say about policy is an organization's culture to follow policy, right, is a very big deal.
Mark Thomas [00:03:22]:
You and I have both been in organizations where some say, oh, policy, that's a compliance requirement. And some say, policy schmolicy, right? So, there's a culture aspect in there as well.
KB [00:03:33]:
Okay. So, there's a couple of things I want to get into. Going back to would anyone notice? So, I'm guessing people probably don't notice until something goes wrong like the Air Canada thing. It's like, oh, actually, we've got a policy and it overrode Do you think, are we going to see more instances? Because at the end of the day, companies, if a machine makes an error in the instance of the Air Canada scenario, companies don't want to admit the guilt because also once you admit guilt, then of course they're going to be up for the money and all the stuff that goes with it. Do you envision that these businesses are going to try to obfuscate now that responsibility, trying to blame it on the machine? But it's like, well, hang on, that machine is still connected to your company. So whether it's a human being or it's not.
Mark Thomas [00:04:12]:
Yeah.
KB [00:04:13]:
Your system's made the error, therefore it's your responsibility. But are we going to start to see this come through now in terms of there may be more of these instances than breaches now? Who knows?
Mark Thomas [00:04:23]:
Yep. And I think we're going to see a lot more of it. The way I like to look at it is like this. We're right now with AI, we're driving on a dirt road, right? This dirt road doesn't have speed limit signs. This dirt road doesn't have lane divider lines and it doesn't have lines in it. there's not a lot of rules that we are aware of. Okay? But you can still get a speeding ticket on that dirt road because there's some overarching driving ethics and driving things that we need to know. So right now, we haven't seen a lot of court cases, a lot of legal types of things that say, oh, thou shalt be doing this because of this.
Mark Thomas [00:04:57]:
I think right now, we're waiting to see how organizations are navigating the dirt road with no posted speed limits on that. And if you think about the whole idea of these speed limits and kind of the idea of policies and practices around that. I think that organizations are starting to understand their policies have to be enforceable. If it's not enforceable, don't publish the policy, right? We have shells in there, and each one of these has to have an owner, right? We have to have control. There needs to be a consequence because compliance is not just meeting external rules, regulations, laws, and so on. It's also our own internal policies. And there's a reason a policy actually exists. And so, Those traffic lights, we should be able to say on that because we even saw this with the EU AI Act, right? Where we have different levels of AI risks.
Mark Thomas [00:05:45]:
So we can do the same thing. You know, there are certain AI decisions and things we can do, maybe green light stuff. Hey, this is go. This is within what we call our appetite level. I'll talk about that here in just a little bit. An amber light might be, hey, this requires some approval. And a red light, this is absolutely something we don't endorse in the organization. We talk that back into that idea called, of course, appetite tolerance.
Mark Thomas [00:06:08]:
But I will say that no AI system should enter into a production environment unless you can explain it and you have a policy that defines that. So I think you're absolutely right. We're gonna start seeing organizations see more of these types of cases because we haven't experienced it a lot yet. The legal environment, the legal industry right now is still trying to figure out where the cases are. that are going to drive decisions that are going to take place in the future.
KB [00:06:35]:
And what was going through my mind, Mark, as you were talking was it's kind of like, and again, I'm generalizing, perhaps organizations wanting their cake and eat it too. It's always like, we want to innovate by leveraging these agents, trying to leverage AI, for example. Oh, but then to your point, if there's a speeding ticket, oh, we don't want to pay the fine. How do you think people start to navigate that? Because that's going to start to annoy customers too. It's like, well, you, want these machines, but then when you don't want to be responsible for them, it's kind of like getting a puppy dog and it's really cute, but then you don't want to look after it sort of thing. So where is this line going to sit then? Because that would start to exercise the frustration and the confidence level for customers to these brands.
Mark Thomas [00:07:16]:
And it goes to the whole idea of digital trust as well, right? Because I may be able to pass my audits. I meet all of my requirements. Technically, I'm secured, I'm protecting privacy. But if for some reason my customers don't trust me, none of that matters. And I think that's a big deal. And it kind of goes into that whole idea is how fast we're moving down this path. We're trying to get— we have that, what is it called? Fear of missing out, right? And we wanna be the fastest adopters. So we even saw in the Pulse poll that early adoption moving fast is that alone is unlikely.
Mark Thomas [00:07:50]:
To create even a competitive advantage for a lot of organizations. So I think that long-term success, particularly with our users and our customers, are going to depend on if we can demonstrate responsible practice, our transparency, and keep that whole idea of digital trust. I may trust you as an organization, right? I may not trust the technology between us. I take my mother, my late mother. She trusted her bank. She loved her bank. But my mother never had a credit card because she didn't trust technology. She did everything with a check.
Mark Thomas [00:08:22]:
If she needed cash, she'd write a check. And she used to always say, I trust your organization, but not the technology. That's what we're seeing right now with AI, right? You may trust that organization. You've been doing business with them for years, maybe decades. But you're not quite sure about this AI thing because you've heard stories. You've heard stories about hiring practices. You've heard stories about bias in loan approvals, right? You've heard stories about a chatbot giving illegal advice. Again, even though you might have trusted the company, now the technology you don't trust.
Mark Thomas [00:08:52]:
But guess what? That propagates itself into the trust for the entire organization itself. So I think the organizations that can prove trust, right, without question, speed's gonna get you to market. Trust is gonna keep you there because again, you can pass your audits, you can be secured, but if nobody trusts you, none of that matters. So everyone has access to the same models now, right? Most organizations do, even if you're building your own, the same types of models. So adoption is really kind of— we're thinking about this provable control at scale. That's the differentiator. Because if we're talking customers, regulators, or even the courtroom is going to eventually come asking, And the folks who understand those pieces are the ones who will succeed in this race.
KB [00:09:38]:
And going back to the Pulse poll, you mentioned that obviously that movers and shakers first advantage, you want it with speed, you're going to obviously run into issues. But then the conundrum is if you don't go then with speed, then you're left behind and then your competitors have that advantage. So do you think people, organizations are trying to figure that out? Yes, everyone doesn't want to be left behind. They don't want to be obsolete and go out of business. But then equally, if you're moving so fast, you're going to probably run into some of these issues. Do you think businesses are starting to potentially put aside, like on their P&L, like, hey, we might have another Air Canada incident that happens. We need to be prepared for it.
Mark Thomas [00:10:13]:
Yeah, absolutely. And so we're talking, you are talking ROI, return on investment for this too, right? Because if you think about, we're trying to go into this so aggressively, we're looking for that edge, right? That will take us to that next step in the market. So the pressures that we have right now to deliver benefits quickly, they're creating what I call my governance gaps. And we, I think we'll talk about governance here. The irony is this, is the shortcut right here isn't even buying people ROI, right? Shortcutting, trying to get to this as fast as we can. And as we say, only 22%, I think in the poll, said AI has actually met their expectations, which tells me organizations are sprinting with And we're seeing a little bit of that. We'd never accept that in, say, finance or safety types of things. And so if you look at a real estate company, right, they raced towards creating an AI pricing model into the hot market to buy homes at scale.
Mark Thomas [00:11:13]:
But this systematically overpaid. The write-down ran into the hundreds of millions of dollars, right?
KB [00:11:19]:
Yeah.
Mark Thomas [00:11:20]:
And so, the ROI showed up after the discipline, not before it. So, my point here is for chasing down this ROI, making sure that, hey, we wanna be first to market. We don't wanna be last. We wanna make sure we're getting the investment. So, I would say this. I would say hold AI to the same bar that you would hold finance or safety. My CEO, when I was a CIO, it was a managed service provider. He used to always ask questions.
Mark Thomas [00:11:47]:
we were talking about risk. And his first question was, if this risk happens, could somebody be hurt? All right. And so that's the same thing is, again, we're not talking about at scale people being physically harmed by every AI type of decision, but use the same level of scrutiny. And we treat governance as the steering wheel here, not the brake pedal. Because a lot of people love to say, oh my gosh, governance is just, it's just providing brakes, right? We We don't need all those rules. I go back to the traffic analogy again is, you know what? I'm pretty comfortable driving on the streets knowing that there are speed limits. There are rules that, hey, when there's a school bus, I'm not supposed to drive past a school bus. I use my blinker and so on.
Mark Thomas [00:12:27]:
But what happens is when I'm out in the middle of nowhere, Montana, right? I may be on a highway that has no speed limit. And why? Because the risks are a little bit less of those kinds of things. So governance are those rules, and we don't have to go replace our governance structures. And I kind of mentioned that when we talked a little bit last time. But I would say, watch where the pressure leaks. And that is shadow AI for a lot of organizations. If we're trying to get ROI, we're trying to have beneficial value to our customers and our users and trying to reduce friction, which is really the whole idea of digital transformation to begin with. We need to make sure that we don't wanna buy a car with a big engine and no brakes.
Mark Thomas [00:13:10]:
And I think that's what we're seeing a little bit in the market right now. We haven't tested the brakes. And we, we can talk a little bit about kill switches later and those kinds of things, but I think that's a big thing that we're not considering right now.
KB [00:13:21]:
And then going back to the 22% that you mentioned, AI has met their expectations. What do you think people's expectations are? Do you think people had these astronomical expectations where it's like, cool, you know, I can put my feet up, I can do whatever I want. Have beers with my buddies early on Friday morning because I don't have to because I got this machine doing the work. Or is it to your point, we've got this really cool fast car, but the brakes, oh, the brakes don't even work now. We need to go back and retrofit it and look into it. Is it more that?
Mark Thomas [00:13:49]:
Yeah. So I think there's a mix. We haven't seen a huge, huge number out there where organizations have said we have increased revenue by this astronomical event just because of AI. Right now, we're seeing most of it in the cost-cutting side. We see the news all the time. Large firms are saying, hey, look at us. We cut this much cost by what? Removing humans. And we replace those humans with AI agents.
Mark Thomas [00:14:16]:
And here is the kind of challenge we're running into. I'd seen a number not too long ago that, oh gee, almost a quarter of organizations who are putting agents out in their production systems are pulling those agents back out. Why is that? It's not because of technology, which is what they're blaming it on. It's because of their governance issues. So an example is I may be talking to a chatbot, a customer-facing chatbot, and we're chatting through an issue I'm having. And for some reason, that bot cannot answer my question. It transfers me to a human. That human has no clue what we just talked about.
Mark Thomas [00:14:51]:
And I have to explain the whole thing to a human all over again. Those are pretty basic things. And also those chatbots might be giving me information that's outside of its normal registry, which kind of says, hey, it's giving me information out there. The point we're talking about here is from ROI and expectations. We're seeing it from cost cutting right now. I'm not seeing massive amounts of new revenue, but also what I'm seeing are organizations are finding themselves spending more time than they thought they were going to save. I'll give you a personal example. When I created my digital twin, the whole idea was for me to streamline things like creating blogs, things like presentations, doing my billing app, you know, those types of things.
Mark Thomas [00:15:31]:
But I found that although it's saved me time in some areas, I spend more time tuning, governing, and human in the loop now with my AI digital twin than I ever expected to because there's overhead to doing that. And that's that idea of what we call human in the loop. And I don't think we really understood what we meant by human in the loop. We love to go out and say human in the loop. But I don't know if anybody really knows what that means. I've got some thoughts on human in the loop here in just a little bit for you.
KB [00:16:02]:
Okay. I want to get to human in the loop because you're right. A lot of people have been interviewing human in the loop, on the loop. But then I spoke to a guy and I'm going to interview him soon. And he said, I'm over the loop. And I said, well, what do you mean by that? And he said, well, humans, we're just, we're getting lazier. So even if it's like, okay, Mark, do you accept Or deny this change request, for example, because it's fallen out of the deterministic way in which you've seen things. It's unusual.
KB [00:16:31]:
He said that now people are probably just going to click yes. It's like, well, it's probably yes. Like people aren't going to look into things. So therefore it is counterintuitive to having the human in the loop. So where would you say we're at now? We're saying we're trying to get the humans in the loop. Now people are too lazy to do that. We're trying to use machines to do the work, but now it's costing even more time and money, people are saying. So are we gonna roll back to where we were, Neal? What's happening?
Mark Thomas [00:16:58]:
So, you know, so when we say human in the loop, and let's say we replace a human with an agent. So from my perspective, in my governance rules, my digital twin does not communicate with customers. Every product that comes out of my digital twin requires my review. But I think the challenge we're running into is you're right. And he is right as well, saying we are getting lazier, right? Because we need to ask a couple of questions about human in the loop. This is really important for us. Number one is we need to test that loop with 3 questions. One is, do we have the expertise? Does the human have the expertise to make that decision? Okay? The second one is—
KB [00:17:39]:
Affirmative.
Mark Thomas [00:17:39]:
authority, right? So they may have the expertise, but have we identified the authority? Or have we just said, hey, Mark, we want you to be human in the loop on this because AI says that's a critical control of ours. And do they have the time? And what we see is AI moves so fast. We've seen processes take place that if there were human in the loop, the process would fail because we're still waiting on a human, right? My analogy to that is, oh, this has been decades ago. You might remember We had what was called the flash crash that took place in the market where an algorithm started making errant trades in one of the stock markets in the US. And over minutes, trillions of dollars were lost and humans couldn't stop it fast enough. Right? So that's the whole idea is when this guy says, yeah, I'm kind of over it. These are automated controls that we may have to have in place because does that human understand what they're reviewing? Do they have the power to say, No. Right? And do they have the time to actually look? So you miss one of those 3, you don't have human oversight.
Mark Thomas [00:18:43]:
You know what you are? You're a spectator in this sport. And that's the big problem that I see a lot of folks running into. Human in the loop has to really mean human in the loop. And the other piece is define where that human sits, right? That's in writing. And what we're talking about here is not every decision needs a human. I have agents making decisions for me all the time, but they're at a level of authority That they are allowed to do, and they have that level of authority. It meets certain criteria. If it meets these criteria, it has that level of authority.
Mark Thomas [00:19:14]:
If not, it then escalates through a workflow to a human that we might have to do there. And I think the last piece on that human in the loop that I really like is log what the human actually did. Not just that the human was present. Log what the human did, because we're logging it with our agents. So just by saying review and approved in a system log, that proves attendance, not oversight, right? And capture what was checked and what was challenged and possibly what was changed. Every decision that I make in my digital environment, I do this. I sometimes, I will do this as part of a test with my digital twin. I will ask it to do things that it's not allowed to do based on my governance rules.
Mark Thomas [00:19:56]:
I will ask it to do that. My twin comes back, says, hey, Mark, you've asked me to do something that exceeds My authority level that you, my governing body, assigned to me. If you'd like for me to make this decision, I need your approval to do that. And if you give me that approval to do that, I'm going to put this in the decision log to cover our bases, right? To make sure that it was documented and so on. Not enough organizations are actually doing that. A decision log to see, hey, at, on this time, this decision Mark Thomas overrode his own policy, which is okay. I can have a policy exception because I am the authority level on that and it documented that and it's auditable.
KB [00:20:39]:
I think this is really important because even like if you go back to back in the day, we'd have a policy and yes, we have policies. Sometimes people obey them, sometimes they don't. And then, oh, there's an exemption because some general manager needs to get his project out on time. So he hits his KPIs. So he goes on his European vacation for the year. Fine. So it's sort of like the problems that we've been facing the last 15 or 20 years in IT are still there in a different form though, because then if there's a human in the loop, you're not supposed to accept something, but you did. So you've overridden your own policy.
KB [00:21:12]:
Therefore, it's more of a moot point. Where do you think we go to from here then? Because are we over-indexing and over-investing now into AI to make these problems go away. If anything, it looks like the problems potentially could be getting worse.
Mark Thomas [00:21:26]:
Yeah. And so I kind of boil it down to a couple of things is that's clearly we're talking accountability, right? And we may talk about ownership here in just a few minutes. But when we're assigning accountability, we certainly want to assign accountability to roles and individuals, not committees. You know what happens when we assign accountability to a committee, right? No individual role takes accountability. So I think that's kind of the first one. So the second thing is what I call licensing decisions, right? We need to bake these things in. So no AI system, right, enters a production environment unless you can explain it, you can test it, you can monitor it, you can stop it, and put that policy verbatim in that. And like you said, we have a lot of issues today.
Mark Thomas [00:22:09]:
All we're doing is adding this new word called AI at the end of a lot of issues that we ran into. not too long back when we started seeing cybersecurity proliferate into the organizations. And before that, it was web applications and so on. We're gonna see this exact same thing with quantum computing. And that kind of goes all the way back to what is your overarching governance model and what does it actually mean to do that? So you're right, we are experiencing those same kinds of issues. And I think those basic policy practices still make a lot of sense to me. And I think Coming down to those license conditions, right, is nothing enters production unless I can explain it, I can test it, I monitor it, and I can stop it, right? That's that what we call the famous kill switch on these pieces.
KB [00:22:56]:
We'll come back to that after a quick word from our sponsor. I'm known for being direct. Let's be honest, nobody gets into technology leadership for the compliance paperwork But if you're building or scaling a tech company, security frameworks like ISO 27001, SOC 2, Essential 8, CPS 234, or GDPR aren't just tick box exercises; they are business critical. That's where Vanta comes in. Vanta automates up to 90% of the work for security and compliance, helping you get audit ready in weeks, not months. It integrates seamlessly. with your tech stack so you can spend less time chasing documentation and more time leading innovation. If you're a CTO, CISO, or head of security, it's worth taking a closer look. Visit vanta.com/KBKast, V-A-N-T-A.com/KBKast, to learn more.
KB [00:23:44]:
So I want to move now to a little bit more fidelity around the accountability part. And I know that we sort of talked about it before with the Air Canada, for example. Yes, it was their machine. It made a mistake. Again, no one really wants to pay the money if they believe that there was an oversight, et cetera. But then really at the end of the day, who do you believe is accountable then? Because this is what we're going to start to see. We're already seeing it now as well.
KB [00:24:20]:
Is it the vendor? Is it the executive? Is it the machine? Well, The person who clicked approve for the machine batch.
Mark Thomas [00:24:26]:
Oh, yeah. Because everybody wants to know who owns this thing, who's accountable for it, right? Even in the Pulse poll, where we found a lot of really interesting information that are kind of summarized in the fact that, hey, vendors provide tools, but organizations remain responsible for how their AI is implemented, how it's governed, boards establish expectations, and so on. I think everyone in that list has a role. And this is challenging now. Because we saw this to a lesser degree. Well, I shouldn't say lesser degree in cybersecurity. Who owns cybersecurity? Well, as it turns out, we go create this new committee called the Information Cybersecurity Committee. We create a new chief for cybersecurity and so on.
Mark Thomas [00:25:04]:
Now we have somebody to go to when something goes wrong. And we're seeing that whole thing replay here. But AI is also the same ideas, is AI is touching every organization. So let's say everyone has a role. But I think accountability still, I think it still lands at the top. I think the board still owns direction. Executives, business own implementation. The vendor owns that tool, right? But even if we outsource, we use a vendor, we use a tool, we're still accountable for the outcomes of what might happen if that tool misbehaves, right? So accountability doesn't transfer to the vendor.
Mark Thomas [00:25:43]:
So kind of that idea. So if you think about an organization of, oh, I'll say at Knight Capital, Okay? So Knight Capital, an algorithm fired off millions of bad trades in some time. Let's say this happens and money's lost. So regulators didn't find an algorithm, right? They actually go to an organization that does that. So that's the whole idea of that governance piece. I will say when we talk about ownership though, a lot of people love the knee-jerk reaction. We need a chief for this and we need a committee for it. I'm not saying that's wrong.
Mark Thomas [00:26:13]:
It's based on different organizations. When you visit as many boards as I do in every quarter, boards are being overwhelmed with the committees, right? We are. So a committee shouldn't own this, but a committee should be doing the heavy lifting so we understand what we're doing. Got that piece. I think the role of a chief AI officer, it depends on the organization. Because like I said, we love to create a new chief for everything that confuses us, right? And then we blame that chief or we hold that chief accountable. Look at the poor CISO, right? We started to add, and I've had organizations say, oh, well, this AI stuff, this goes to the CISO. Like, wait a second, right? Not every AI risk is associated with information security, right? And so I think that's the first thing.
Mark Thomas [00:26:57]:
So here's what I would suggest is overall, the board of directors, we get it. The executive, the business owns this idea of AI, but every AI system, that goes into your organization gets a named owner before a login is ever created, right? And somebody owns it. It's not a committee. It's not a department. It's a person or a role. That owner is answerable for that system's behavior, right? Whether it's drift, doesn't matter. That owner is responsible for that. If nobody's name's on it, no one's watching it.
Mark Thomas [00:27:30]:
You know, the old adage that if something is everyone's job, it's no one's job. So we still have to really kind of identify what that is. But I would also say this in AI. Should make sure we have this distinction between the owner and the operator, right? Because the person who runs that model day to day, they shouldn't be the only person accountable for it, right? Ownership sits at a level with authority, authority to fund, right? Authority to make prioritizations, authority to shut it down, right? So, it could be the same person, maybe a different role. I think the other thing is since a lot of organizations are very vendor-heavy, When it comes to using AI, keep the distinction that the vendor owns a tool. The vendor doesn't own that outcome. And we've kind of seen that. That's the third-party risk that we've covered quite a bit, all the way back to the Target days.
Mark Thomas [00:28:17]:
Okay? Ownership follows this whole lifecycle, not just launch. We don't put a name on it when it launches and then kind of forget about it and say, oh, call Mark for that, right? So when we deploy it, don't let ownership evaporate over time. The owner's name stays on this. And I think right now we're seeing specifically here within the last several months is agents, right? Agents need owners too, and agentic AI. So if I deploy an agent into my environment, it doesn't just need an owner, it needs a tight owner. So when AI moves, when we're talking agent, and AI has just moved from recommending to executing now. I will tell you that Singapore has done a Fantastic job. I don't recall the name of the framework off the top of my head.
Mark Thomas [00:29:05]:
They have the first agentic AI governance framework. And what they actually suggest, this is really neat, is a lot of things. But one that I really liked was every agent, right, not only has an owner, but they have an ID card, right? And that ID card, just like you or I, when we go into the organization, we've got to beep in and it lets me know, What areas of the facility I'm able to get access to, what information, what data I have access to. And I think that's a really interesting kind of idea. So as we're moving to this execution, I think every agent needs that owner that's accountable for its credentials because that agent, right, is now doing what a human used to do. It's in a role. It has access. It has decision authority, just like a human would.
Mark Thomas [00:29:52]:
And if we understand how to govern it, it will make fewer mistakes than humans actually could sometimes. But that's the issue, is being able to govern that.
KB [00:30:00]:
So now, Mark, I want to move to ask you about perhaps if companies cannot explain, test, monitor, or shut down an AI system, for example, should it be allowed in production then at all?
Mark Thomas [00:30:14]:
If I can't explain it, if I can't test it, if I— no. I would say that from the perspective of if I can't explain, test, or monitor that, No way. That's the short answer, right? So, when we're talking about explanation, testing, and monitoring this stuff, those are my license conditions I talked about a little bit earlier, right? If you can't meet all those, you're not operating a system. You're hoping at that system. Okay? The data is saying, right, that most organizations are hoping, right? They're not governing. I think the number was 12%. 12% have tested their ability to shut a system Down, right? And that's important. And we saw this in some recent headlines where organizations, again, like we said, pulling agents out, they're shutting a complete agent down.
Mark Thomas [00:31:00]:
And now they're going back to try to put a human in the spot. And the human's like, hey, you know what? We found something else to do. So now we're seeing another critical issue in terms of talent and those kinds of things. But nope, like we said before, nobody buys that car because the engine's impressive, but it has no brakes, right? I hate to kind of keep saying it that way, but So if we have an automated system that issues unlawful debt notices, right, for years, because there's no way for somebody to challenge or halt it, we haven't tested that, right? And so those are the kind of things to think about as we're thinking about the testing and the shutdown. I do have now, in fact, it's funny, is for when I first launched my digital twin, I did not have the ability to shut this down. Think about it. I've got a digital twin environment of me. I'm a one-person LLC.
Mark Thomas [00:31:47]:
If I get ill, I get sick, I pass away. That's a reality, right? That's a risk scenario that could happen. What happens to my digital twin? So we do have controls in place where my digital twin will continue to operate and be able to what we call gracefully shut itself down because there's certain information, things like that, that it knows that might be needed in those cases. Now, again, that's a very extreme case. But that's a control that I have in place. I do not want to say I retire, I'm done, but I have a twin continue to do some of this work. Same thing applies for organizations that have agentic AI, AI systems in their organizations that goes crazy, right? It starts to hallucinate and actually people may be being harmed. It's actually causing more concern and more issue with users than it's worth.
Mark Thomas [00:32:37]:
We need to have that kill switch. If you can't Prove that in a test environment. You should have never taken a look at it. Another perfect example: I have a board of directors. It's a virtual board of directors. Again, one-person company. It's an eight-person board of directors. We tested that board of directors.
Mark Thomas [00:32:52]:
When I say we, I mean me. Tested that board of directors multiple times on whether or not they had the agency to be able to make certain decisions, and they don't. They're not the board. But we also tested the ability. to remove or dismantle that group. And we were able to test all of those before my board of advisors actually became a production advisory group for me. So, all of those things, again, I'm a one-person organization. But you multiply this to the size of a multinational bank, distribution firm, consulting organization, same rules apply for all of us.
KB [00:33:30]:
So, just going back to your example around the debt notices, if the company There's no way of challenging that. It shouldn't happen, which I agree with. But how can people literally go to sleep at night thinking, you know what, I don't have any way of challenging that. Oh, well, maybe someone has a debt notice. Maybe they don't. Who knows? Maybe I'm not recovering the debt I should be. Maybe I've done the wrong thing. Are people really that way inclined though?
Mark Thomas [00:33:55]:
No, I'm not sure. And I think that's another piece that we're missing right now is that escalation piece. And when we talk about, are we testing it? And so on. I'm not sure I'd be comfortable, right, reaching out to an organization's chatbot to talk about something that I disagree with on that. I don't think the industry has that answer quite yet, to be quite frank with you. I don't think it does.
KB [00:34:16]:
Are you also seeing that companies are of the mindset that if they can't intervene, because I've even heard of instances that you've got AI agents, because it's all about efficiency, overriding other AI agents to get to the outcome, which is causing all sorts of problems. So, There's all these very unique cases I'm hearing and interviewing people like yourself, Mark, that we're seeing that obviously people, it was hard to predict because like you said, it's a dirt road. It's still being formed and all these sorts of things. Would you say the companies are still of that mindset that if we can't intervene or shut it down, we're not doing it?
Mark Thomas [00:34:52]:
We're not going to do it. We can't do that. Exactly right. Because agentic AI risks are a perfect example. of this. And I always like to use the word agency, right? There are levels of authority. So we treat these, right, the same way I would treat a human in a position that makes decisions, right? And you go back to the zero trust architecture principles, for example, right? Least privileged access that we might have on that. Also, I would say that credentials for AI decisions and AI agents should be timeboxed, right? I have to renew.
Mark Thomas [00:35:24]:
It's like the password reset. Right? I'm on a client email system. Every month I get an email that says, you need to reset your password or you don't have access to our email anymore. We can actually timebox that agent's ability to make those decisions. So it has to be renewed on a regular basis. I think that will put a really, really good pressure, kind of a pressure valve in the whole process. So human in the loop, we talked about that for the gates around those consequential, maybe those irreversible actions. And I think that having that, those kill switch procedures, because The core shift, like we said, with this is we're trying to make our lives easier.
Mark Thomas [00:35:57]:
We're trying to do 3 things: increase revenue, decrease cost, and remove friction, right, for our users and our customers. That's what we'd like to do. So these agents, right, like we said, are moving from decision risk, right, which we have now, to action risk. So the traditional model says, hey, did the model give me a bad answer?
KB [00:36:21]:
Okay.
Mark Thomas [00:36:21]:
That's what we were asking just a few months ago. Today, I'm asking, did the system do a bad thing before I could stop it? This has happened in months, right? We've gone from, hey, this is great to make a presentation. Now it's actually making decisions for me. So this is compressing. I'm kind of going back to what can humans do for this? This is compressing this, what we call this detection to impact window, right? It is now shortened. Dramatically. And an agent that may have, say, write access and a bad governance framework over it, it's not going to wait for a quarterly model to validate its lifecycle, right? It's going to make that decision. And so, these autonomous agents that are taking actions, in some cases beyond their scope, and that's the whole idea is we're trying to put these into production so fast, the fear of missing out.
Mark Thomas [00:37:13]:
that we're truly not governing. And, you know, we love to throw that word governance behind everything. We have corporate governance, finance governance, IT governance, you name it. And when we think that governance just kind of magically happens, it's the homework that has to be done before we put these into production, the intent, right, of having those checkpoints in this, understanding what those actions are, the risks. I really think we're starting to see Organizations understanding this, because when we listen to somebody that says, just spin up an agent for it. Well, we can spin up an agent. If we did not tell that agent that it cannot spawn another agent, it's going to create another agent to help it. And it's going to actually have that agent with certain skill sets and so on.
Mark Thomas [00:38:00]:
In fact, it's interesting that my own digital environment, I asked it a question. I said, What does my consulting look like in 5 years from now? Or maybe 10 years? And one of the options of this was, again, this is just saying, this is what it's telling me is, is my consulting might be my agents maybe consulting your agents in 5 years. You and I are sitting on that beach we were talking about. We're overwatching this training taking place where my agent, right, recognizes that some of your agents may have skill sets that it needs. And you're paying me for my agent to train and give you new skill sets for some of your agents. Interesting how that looks, right? So I kind of, I did a little bit of what we call drift, right? Going a little bit off question, but it was really kind of neat how we're looking at some of those things in the industry right now. And shadow IT or shadow AI, that's becoming a big concern of mine right now.
KB [00:38:54]:
So I just wanna talk a little bit more about the agentic piece because now what's coming down the pike, as we know, is people doing agentic and all this sort of stuff. Things doing stuff autonomously in the background without human intervention. So that then would signal to me that businesses are then comfortable to step away without human in the loop and all that sort of stuff. So do you think the only real solution for, to prevent potentially an agent going rogue if it's doing stuff in the background quietly is to have that timeboxing mechanism to just double check, okay, the privileges are still there. you know, how you're thinking is still there because it might start to go rogue and think, actually, no, I can start to escalate some of these things. We have to keep a check on that. So, does that then mean businesses are getting comfortable with, well, we're happy to take my foot off the accelerator and remove some of the helicoptering that we're doing because we're losing time elsewhere? So, that to me definitely means where people's minds are that they're happy to do that, to let, you know, let go of that control.
Mark Thomas [00:40:01]:
No, I think you're right. And we're trying to do that for those reasons we talked about, increase revenue, decrease cost, remove friction. And so I think the whole agentic movement is great. If we just have to keep in our minds that AI agent we created, that used to be a human, it's a role, right? And that kind of goes back to some of the things I was mentioning earlier is if that's in a role, What are its decision authorities, right? Because I may have an agent that can add, that could do routing table maintenance in my network, right? But it has to have certain levels of authority to be able to do that. It can only do it during these times. It can only be approved by a human and so on. So I think you're right. When we're looking at agents, it comes down to governing that agent, managing that agent, just like we would a human being in that role.
Mark Thomas [00:40:49]:
There's a job description, right? I love the idea of having an ID card that says, here is what your level of authority is. Now I can change that level of authority. It's interesting. I had a discussion with the technology team of a hospital not too long back. And really neat story they were explaining is in their service desk, their incident management environment, they had gone through some testing and they had agents that were helping them handle incidents. So an AI agent was monitoring a specific service. It catches the fact that there's a service that's wobbling just a little bit. It hasn't breached any thresholds, but it's wobbling.
Mark Thomas [00:41:26]:
Its performance level is changing a little bit. The agent notifies a human that says, hey, I see an incident about to happen. If we don't take action now, this incident is about to happen. The human takes, oh, just under 10 minutes to respond back. In the time it took the human to respond back, the incident happened, right? Human says, open up a ticket, follow the process. The agent creates an incident ticket with the appropriate level of priority, the appropriate categorization, and it's level 1, right? It's tier 1. It determines it doesn't have the skill set to handle this incident. So guess what it does? It wakes up agents on tier 2, says, hey, Knock, knock.
Mark Thomas [00:42:08]:
I've got an incident I need your help with. And these agents are communicating, right? And these agents say, oh yeah. So they're referring back to vendor information, to product details. And between these agents, the human sees this happening, right? And between all these agents, they come up with a solution, go back to the human. And they say, human, we have a solution that we can put in place. It'll fix this incident right now. And by the way, This fix we want to do, we've already tested it in a virtual environment for you, and it's prepared to deploy. We can deploy this.
Mark Thomas [00:42:42]:
We have the level of authority to do this with your human approval. And the human verified that it was tested. Human said, you may deploy. It was deployed, and this incident ticket was closed, right? All this was minutes.
KB [00:42:55]:
Okay.
Mark Thomas [00:42:56]:
What the CIO was telling me was that process, if it required humans in every step of that process, That entire cycle probably would have been around 3 to 5 hours. Okay. And it was done in minutes. And again, this was just a test that they were doing and they were prepared to deploy this. So we were talking about, hey, what could it do for us? It could do some great stuff. That human now is in the position to make those decisions. That human could have said, no, I do not want this deployed into a live environment for these reasons. And then we would have come up with another solution.
Mark Thomas [00:43:31]:
Seeing this with auditors right now, with international auditors. So I'm working with an audit organization for a country and they're really starting to use AI agents to monitor transactions, financial transactions, right? To look for things like fraud, things like money laundering and those things. And initially they were worried that they're gonna replace humans. Well, as it turns out, right, instead of doing this quarterly sampling of financial transaction data, The agents are monitoring real time. And what it's doing, it's pushing out those little anomalies saying, I am monitoring 100% of all these transactions instead of, say, 30% every 3 months. And it's getting the humans involved. And they didn't have any loss or any removals of any humans because now the humans are taking these issues that are called out and determining if they're false positives, right? Being able to move and actually do an auditor's work. So, so kind of to your point is it's making things better.
Mark Thomas [00:44:29]:
And I will tell you that, that humans, right, we cause more mistakes than machines do if those machines are tuned perfectly. If you've ever been in a driverless car before, we have a lot of them in Phoenix. Waymo is what we call. First time I was in a driverless car, I'm gonna be honest with you, I was very nervous. I was very nervous. Statistically, right, there are fewer issues than there are with humans driving. But all it takes, the problem we run into with AI right now is all it takes is one incident where a driverless car hurts somebody, then all driverless cars are bad.
KB [00:45:02]:
Okay.
Mark Thomas [00:45:02]:
So that's kind of what we're seeing is, is there's a lot of scrutiny. We're looking for that great one story where that same story happens with humans all the time. All we have is that one story for AI and all of it is dangerous.
Mark Thomas [00:45:17]:
Yeah, there are still things that we need to look at. Goes back to what you and I talked about in our last call, our governance models, our governance models.
KB [00:45:25]:
So, Mark, final question, closing comments, final thoughts. I know we didn't quite get to all the pieces today, so I'll have to bring you back. But what would you like to leave our audience with today?
Mark Thomas [00:45:35]:
So I would say that for anybody looking at AI, this is risk, right? We need to understand what our risk scenarios are. You have model-level risks that are talking about bias, fairness, those things. Put controls in place for this. And you can find this information out at, obviously, we have information on the Pulse poll from ISACA that has information about what you're experiencing. But we also have some certifications and some courses around that. My favorite, I think we talked about last time, AAIR, Advanced in AI Risk, which talks about controls to put in place. So think about things like your adversarial threats. Your AI may not perform properly Because of an internal governance issue you have.
Mark Thomas [00:46:16]:
But there's still threats. There's adversarial things that are taking place out there, attacks like data poisoning, the famous prompt injections we always hear about where somebody's trying to trick an AI agent out there. But of course, the governance and accountability risk, the ownership, those things. And keep an eye on the regulatory environment. Folks, right now we're on that dirt road with no speed limit signs. And no guardrails on it other than the EU AI Act right now, which is now partially enforceable. But I think that the industry is waiting to see what those next landmark decisions are going to be. So don't wait, right? Don't wait until you have to be compliant.
Mark Thomas [00:46:57]:
Assume there's some compliance coming and have your own risk scenarios identified and respond to those.
KB [00:47:06]:
That was Mark Thomas, everybody. What I'm sitting with is his board test. Would anyone even notice if the policy was violated? For most companies right now, the honest answer is no. If you're a CEO or board director listening, go find out who owns each AI system in your business by name and whether anyone has tested the off switch. If you can't answer that today, you don't have a control, you have hope.
KB [00:47:30]:
I read every reply. If you've got some thoughts on this one, send me a message on LinkedIn.
VO:
KBKast, Cyber for The C-suite.
