[00:00:00] Speaker A: What we do with information is important. It's not just about ticking a box. It's not just about us being able to find it later so we can more conveniently access it, or we can do our job better. We have to think about the person, the community or the environment at the end of the line of that information.
This is Katie. How they complete the site as a primary target for ransomware campaigns, security and testing and performance risk and compliance. We can actually automate that, take that data and use it.
[00:00:38] Speaker B: Joining me today is Rachel Greaves, CEO and co founder of Castle Point Systems. And today we're discussing the state of play regarding records management. So, Rachel, thanks for joining me back on the show, and welcome.
[00:00:49] Speaker A: Thank you very much. Great to be back.
[00:00:51] Speaker B: So, Rachel, let's sort of start with your view on the current state of play with records management. Perhaps, maybe give everyone a bit of a background. I know it's not something that I often talk about on the show. I think we may have touched on it when you were last on, which was a while back, so it'd be great to get a bit of background from yourself.
[00:01:08] Speaker A: Yeah, look, records management is kind of the new frontier in cybersecurity. You will have seen a lot of changes coming from central governments and regulators around a data centric approach, and a big reason for that is because of something called data minimization. So you'll know, if you're familiar with privacy regulations, that this concept of holding a minimum amount of sensitive data is really important. And that doesn't just mean only collecting what you need to have, it also means only keeping it for as long as you require it for your business purposes and not keeping it any longer. And the reason for that is because these days, it's broadly considered that cyber breaches and spills of data are inevitable. So even though we might have spent a long time since the beginning of the digital age focusing on reducing the likelihood of a breach, we now need to really focus on reducing the impact of a breach, which means holding the least amount of data that we can. Of course, we have other obligations for keeping information. If we get rid of stuff too soon, then we lose the value of that information. We don't meet our obligations to our stakeholders or to regulators. And of course, we want to exploit the data that we do have. We want to get the best value out of it for business reasons as well as cybersecurity reasons. So there's a really difficult trade off between keeping stuff and destroying stuff, and the only function and the only people and the only systems that can solve that trade off are records management.
[00:02:39] Speaker B: So you mentioned before holding the least amount of data, would you say now, because of all those major breaches in Australia? Yes, of course. I've harped on about it again, but I think it's given definitely a wake up call to the industry, even more broadly, outside of Australia as well. Do you think that people are a bit more cognizant now of what data they are keeping, or are they starting to pay more attention, would you say, in your sort of profession?
[00:03:02] Speaker A: Yeah, they are, because they have to be. So it used to be, the only people that were worried about getting rid of information that was accumulating were people managing storage. They wanted to clean all this stuff up, let's get rid of it. And of course, records teams were very resistant to that, because everything in there could be a record, it could be valuable. And the commercial sides and the executive sides of the business were also resistant to that, because that's a gold mine, that stuff. We spend most of our waking hours at work collecting data. It's kind of the oil or the uranium of our business. It's this really valuable resource. But just like uranium, the longer we hold it and the more it starts to decay, the more dangerous it becomes. So as storage got cheaper, that kind of noise in the ear of let's get rid of stuff, kind of went away because there was always more storage. But then a new voice came in the other ear, which is about cybersecurity. So a good example is me, actually. So, in 2018, my university, the ANU, was hacked by a foreign state actor, and they took 19 years worth of staff and student records from the ANU systems, which had just been accumulated over time and never disposed of. And the problem with that is that those records didn't need to be kept for 19 years. Nothing like it. They had a seven year retention under the ANU's own policy and legislation. So my data was in that spill. And now that foreign state actor has all my information that the ANU had at the time, even though I'd graduated 15 years before that breach, and my record should have only been kept for seven. So 60% of the people affected by that breach shouldn't have been, by rights, and that would have been a 60% lower impact, obviously, on the community, but also a lower impact on the cost of remediation, of insurance, of communications. And the reputational harm would have been a little bit smaller as well.
[00:05:02] Speaker B: So they've effectively kept data for twelve additional years than what they needed to from what you're saying, okay, how do you get to that state where, I mean, they're bigger, your organization? It's not like they've got no budget for these types of things. I'm just curious to hear from your point of view, how do companies get to the stage where they're like, wow, we're actually holding data that we don't really need to. Twelve additional years. How come no one's asking why this is a thing?
[00:05:30] Speaker A: Well, it's because it's really hard to get rid of it, and it has always been really hard to get rid of it. And it's been so hard to get rid of it that now the australian federal government has only 7% of their information formally records managed and sentenced. So sentenced means there's an expiry applied to that information. We know when we have to dispose of it, and the disposal action could be just destroying it because it's no longer valuable, or it could be transferring it to the national archives, for example. So 93% of the records that are held across australian federal government or central government are not sentenced at all. And that's up 30% from a few years ago. So the problem is getting worse, not better. And the reason is because the kind of traditional ways of sentencing of managing records don't scale and have never been feasible, really, since the transition from the paper paradigm of managing a file and putting a stamp on it, to the digital paradigm where information changes all the time. There's a really high volume of information, a big variety of information, a rapid velocity of change of information, and still only two or three records professionals in the organization to try to keep across it all. So this is a problem that's been around for a long time, and this is the problem that made us develop our solution. Because my background is audit. I used to be an auditor and I audited mostly federal government for their non compliances against these rules. And what became really clear is that there were best intentions. Sentencing has always been taken quite seriously, but it just wasn't possible to succeed with the technology that was available.
[00:07:15] Speaker B: What does best intentions look like?
[00:07:17] Speaker A: Well, it looks like people wasting a lot of time on low value work, frankly. So if you're a records person, you have to decide how long something should be kept. And to do that, first you need to understand what it's about, which means you need to read it all. So you would have to take a record, which is an aggregation of stuff, a bunch of emails and documents, all in one file share might be a record, and you would have to read it all, not just the titles, but all of the content, and effectively take out your mental highlighter and highlight all the topics and things that that record is about. And then what you'd have to do is match all those topics to hundreds of different rules. So there'll be a retention rule for final approved versions of asbestos remediation policy, for example. Well, do you have one of those? And there'll be another rule for information about community advocacy for children's health, not including crime prevention. So now you've got to find, is there anything in here about that? And also, is there anything that would exclude it from matching this? There are so many rules and there's so much content. And even if you could do that for the one file that you have, then someone will come in and edit that tomorrow. They'll add a new document to it, they'll take something out, and then that classification is no longer relevant. You've got to redo it all again. So you can see that it's really impossible. And the only way to do it has been to do it like that, to do it the hard way, which means we have people who aren't machines spending all their time just trying to play catch up, reading and matching and reading and matching and not using their time doing things that humans are good at, which is making decisions and planning and strategy. So the reason that we're able to move the needle on this now is because of artificial intelligence. We've come to the point, and five years ago is where we developed this new AI paradigm to do this work. So it's well established now, but it's only been five years that it's really been possible. We get the machine to do that hard work, read all the words in everything in the whole environment, read them constantly, read them all the time, and match them automatically to these hundreds or thousands of rules, and then calculate and roll up the retention that applies. When we get the machine to do that, we free up those humans to actually then go and do the important thing, which is make the decision to dispose if appropriate.
[00:09:41] Speaker B: So you're saying historically, people were doing this manually, and if there's one edit, they have to do it all again? I mean, this could take months.
[00:09:50] Speaker A: Well, it's just impossible.
A couple of stats. So it would take you, based on my maths, it would take you about 130 years to read 1 Microsoft Word documents. So it's clearly not possible to read all of that stuff. And we've seen that in practice, too. So we had one client who had spent two years. They hired a records manager to go through their legacy shared drive, which wasn't changing at all. So that's a head start. Like, it's static. To go through the legacy shared drive, which had about 4 million items in it, 4.2, I think, and classify it all and sentence it to work out what needed to be kept and what could go. And in two years, she got through, I think, 1.2 million items in that period out of the fourish. And then she quit. Like, fair enough. Like, I would quit, too. I wouldn't have made it two years, that's for sure. So what we did then is we deployed our. We deployed Castlepoint in that environment, and we read all the 4 million items in the drive, and we classified them all against the rules. And then that organization did testing and benchmarking, and what they found was that Castlepoint had been effectively 100% accurate with the classification. But on review, on detailed review, the records manager had only been accurate 25% of the time. And that's not because of any lack of skill of the records manager. It's because 1.2 million items in two years works out, again, by my math, to be something like one document every 7 seconds making a decision about that document. And, of course, in that case, there is no time to read it. There's no time to even open it. You're just making a value judgment based on what you can tell about the name of that item at the time, and that's not enough. The names of our documents don't really tell us anything about what's inside them and what they relate to, not at the fidelity that we need to match to all these regulated obligations.
People, literally, it has been their job to just try to read this stuff and match it to rules. And not only has that been very inefficient, it's been very ineffective as well.
[00:11:59] Speaker B: And also, there's a lot of complexity there. When you talk about all the rules, it depends on how many there are. And you're the expert in this, not me. But I'm just curious to know that's when things could take even longer to map it all against that. And if someone's doing that manually, 130 years, forget it.
[00:12:16] Speaker A: I know the rules are really complex because, of course, there are retention rules, and they come from different acts and regulations. So everything from administrative acts to criminal acts, any kind of legislation, will often have retention rules in it about information. So, for example, you'd know that under australian law, corporations act, et cetera, you need to keep your financial records for seven years, right. Pretty straightforward. But then a lot of organizations also have what's called a records disposal schedule. And for government at all levels, that will be issued by the government. And that's the one that gets into the real nitty gritty about, well, something like this, but not like that, needs to be kept for 15 years, but something like this, but also like that is 20 years. So they're very complex. But then there's other obligations too. So there are secrecy provisions, regulations that make it a civil or a criminal offense to release information to unauthorized users. And there are dozens of those across jurisdictions. There's information handling rules as well that say any information of this type must be able to be accessed by the CFO at all times. There's lots of rules about information handling, and they all need to be applied in a layer. You can't just focus on retention and keeping stuff and not focus on risk and protecting stuff. You have to do both, which means you have to know and apply all the rules, which, again, is something that one records manager doesn't usually have the skill set to do, let alone the time it needs to be SMEs from every part of the business, including legal. And that's, again, really hard to coordinate that collaboration. So that's, again, where AI becomes really valuable, because it can do all of that mapping much faster than people can and more accurately without ever getting decision fatigue, without needing to log off on the weekend.
[00:14:09] Speaker B: So would you say, Rachel, conceptually, people at a high level understand information handling, all those little things you referenced before, like, yes, there's the seven years, but then if it's this, it could be that, it could be 15 years, it could be 20 years, there's all these little nuanced things perhaps that could create that decision fatigue.
[00:14:26] Speaker A: No, they don't understand, and they shouldn't realistically need to understand that. And in fact, the international standard changed recently to say that they shouldn't even know about that. So it's quite clear in the international standard for records management and governance that people in the organization, punters, everyone who works there who isn't an expert in retention, should not be allowed to apply retention rules and should not be involved in the process because it is a specialist activity, it's complex and it's a legislated activity. So what that means is that not only shouldn't we, from a business point of view, be putting the burden on users to understand these rules and apply them, we also shouldn't be doing that from a technical point of view. So it systems that are requiring users to go and pick a classification or set a function or set a class on a record, not only are very inefficient, and again, even more likely to be inaccurate than the centralized control model, they're also non compliant with the standard. We don't want users worrying about this stuff. We want them focused on their productivity, getting the value out of the information they need to know more about. I would argue more about the sensitivity and risk of that data than about the value, because that is something people can and should grapple with and start understanding their threat environment and understanding the kinds of impacts that could come from unauthorized access to the data that they create and capture. But no, this isn't something that general users should understand or should need to be involved in applying.
[00:16:02] Speaker B: So going back to your example before about the ANU, so after everything you sort of said before around the classification, the complexity, the rules, sometimes they override, sometimes they don't. How long it takes, 130 years. Are you probably not surprised then, from what happened with everything that you've just mentioned?
[00:16:18] Speaker A: No, and I'm not even sure they've really even started a sentencing program since then, and that was five years ago.
And it's hard. It's hard to say. Okay, well, Rachel's record can be disposed of according to whom? Like, who is tracking that? I graduated seven years ago and who's going, oh, great, today's the day I can destroy Rachel's record. And then even if they were, even if there was some kind of trigger, notifying the records team to say, hey, these people graduated seven years ago, it's time to clear up their stuff. Their stuff is everywhere. There's emails in there about me. There's stuff from the student society, social clubs about me floating around. There'd be stuff on the file shares and the drives of all the different academics that I interacted with in that time, there would be my thesis is there somewhere, there'd be all my marking information somewhere. It's all over the enterprise, and no one person has access to all that information. And even if they could access all those different systems, and that includes people's inboxes and their sent mail and their onedrives, even if they could access that, they don't have time to go and search for me across all those systems. So now when we deploy record systems like Castlepoint, they include ediscovery just by default, because the first thing is just finding the stuff, like where is it? And then the second thing is alerting on the stuff. So where is the stuff? What do we have who's doing what to it, what activities are happening on it? Because remember, even if I graduated seven years ago, if I have interacted with the university in any way since then, that's going to reset the clock on that seven years. So someone has to be tracking what happens in the lifecycle of this information all the time as well. So we need discovery, we need audit, we need risk. Because maybe they can't destroy my stuff because maybe I was a troublemaker, right? And maybe I was involved in some kind of controversial activity and they need to keep my record a bit longer to manage their own risk. Or maybe I was involved in, maybe I was taught by an academic who later was audited and they have a different obligation to retain my stuff for an extra 15 years. Maybe there's a court case currently, maybe there's some other activity, maybe there's a freeze or a hold on record because of a royal commission or another inquiry. Like there's so many rules, so you can see why they would just default to little, just keep it. And that's what every organization realistically has done and has been doing. And that's why the big lever is getting pulled by changes to the privacy act to say, well, guess what? There's no more just minimum retention. You must keep it for seven years and then who cares? It's going to be maximum retention. And if you don't destroy something when it reaches its maximum retention, you need to justify why you've kept it. And that's completely flipping the paradigm and that has to completely flip the approach to management. And again, without technology, there is no way to understand or apply that maximum. So this is the real forward surge of AI being involved in records governance at this time.
[00:19:24] Speaker B: Yeah, and I guess going back to your point before, around we'll just keep it all. Well, it's like, well, if you destroy it too early and then some auditor comes in, it's like, well, where's the stuff? Oh, we destroyed it. Then that's another problem on its own.
[00:19:36] Speaker A: That's a big problem. In fact, that can be a really damaging problem. So when we get information wrong, it's not just, oh, we get a slap on the wrist or we don't get five stars on our report, not even that. We might get sanctioned by the government or by a regulator. People can be really significantly, you know, just some examples. When we hold a lot of information and we can't find it, people get deported and people die. So let me give you two. So I think I might have talked about Vivian Salon. Last time I was on. Because she's kind of like, the inspiration for why we even started this company. So the Department of Immigration in Australia had Vivian Salon's record in their record keeping system. Vivian Salon is a very vulnerable woman. She was found with a head injury in a park in Lismore. She was mentally unwell and she was deported by the Department of Immigration to the Philippines because they couldn't find her citizenship records in their record keeping system. And she was left in the Philippines with no advocate. She was left in a home for the dying and destitute, being looked after by nuns, and she wasn't discovered and repatriated for literally years. And the day she was deported, her son had been left in childcare when she went missing and never picked up. And he was put into the foster system, where he remained even after she returned. And that's just because they did a search and they just couldn't find her, because the system wasn't designed to be discoverable, it wasn't designed to link information together. And that's one individual, right? And unfortunately, not just one. It's not just Vivian. The same thing happened nearly to Cornelia Rao, another australian citizen, a vulnerable woman locked in immigration detention because they couldn't relate her records between systems. So these are two catastrophic outcomes for these two women and their families. But worse, here in the UK, which is where I am based, you'll remember the Manchester arena terrorist bombing that happened here. Well, an inquiry found that there were 400 pieces of misinformation in the policing databases about the perpetrator of that crime. And if they had been discoverable, which they weren't, then that event would have been prevented. Not could have been, but would have been.
You know, this is what happens when we don't manage information properly, when we just keep it all and we can't find it and we can't use it. On the flip side, if we get rid of it too soon, like you mentioned before. Well, they did that here in the UK as well. So there's something in the UK called the Windrush generation, and that's the caribbean migrants who came to the UK post war. The first ship that came and brought them was called the Windrush. So all these migrants came and they didn't need to have papers. That was part of the policy. They just need, like, come here, be a nurse, be a bus driver, come and help grow the economy, because you're part of the commonwealth. Well, a few years ago, they introduced something called the hostile environment immigration policy here. And that changed the rules. And suddenly you can't rent a house, you can't get credit if you don't have proof, if you don't have papers. So even people who'd come and their grandparents had come back in the 50s suddenly now had to provide this information. Well, the only information that still should have existed to prove their right to remain in the country was their arrival cards, the little slips that came with them on the ships that got stamped, and they had been stored by the government here. Well, unfortunately, the government agency storing those in a basement needed to move premises and they didn't want to bring, so they justified destroying it. They said, oh, it's personal information, so it's technically risky even though it was locked in a basement, but still it's personal information and it's reached its minimum retention. So we're going to destroy all these arrival cards. And they did, ostensibly for regulatory reasons, but actually because they didn't want to move them to the new building. And what that resulted in when this hostile environment policy came into force was no evidence left. It was all gone. It was destroyed. And dozens of people were deported, elderly people, people that had lived here their whole lives, were deported because of that destruction of information, information that should have been kept if they'd considered, again, the other rules, not just the one retention rule and the one privacy rule, but all the other rules and obligations that come under other legislation. So these are the kind of things that can happen to humans, to people, vulnerable stakeholders in our community, if we get data wrong.
[00:24:18] Speaker B: Yeah, that's nuts. Especially going back to it wasn't discoverable, so it's like, oh, we can't find you. Sorry, you're going to get deported.
That just seems so rudimentary from where we are as a society. We can't find someone. That's the part that I don't understand.
Why don't we have a better system? Why don't we have well oiled machine?
This is people's lives. This is them in and out of the country.
[00:24:49] Speaker A: Yeah, it's really important.
And it's hard. I get that it's hard, but it isn't that hard anymore, not now that we have AI to do this. It used to be too hard. So when I used to audit, I would fail. I really probably failed everyone that I ever audited, because they couldn't do these fundamental things like find their information, manage their risk, manage their value and retention. Again, not for want of trying, but they just couldn't succeed. So what the problem I had as an auditor was, I could fail people. But what you're supposed to do as an auditor is then provide a recommendation, and there was nothing to recommend. I couldn't say, oh, if you just go buy this product, if you just go install this or build this custom thing, your problems will be solved. There was no way to solve it because there was no paradigm that would work. It's fine to have technology that solves the problem, but if that technology introduces a new problem, it won't succeed. That's what you find out when you're auditing. So there's always been technology that could technically classify and sentence everything in the environment. If every time a user saved a document or an email, they added five or six or ten or 20 metadata labels to that information, it's a great theory. Like, everything would then be fully compliant, but the impact is way too high. It doesn't succeed. It never has. So what you have to do, and what we did was say, well, we need a new way to do this. We need this approach where we have full coverage. So we're managing all the information in all the systems all the time. Cloud on, prem structured, unstructured. We're indexing everything and making it all discoverable. We're auditing everything so we know what happens to it, and we're applying all the kinds of regulation that you need to apply to it. We have to do that, but we have to do it without impact. It can't have any impact on general users. They need to just be out of the loop, just do their job and don't worry about compliance. It can't have impact on those systems because there are dozens or hundreds or thousands of systems. You can't have a connector for each one. You can't be customizing each one. What you also discover is that anything that has an impact on the data, as in wants to put it up in its own data lake and take it out of the environment, or wants to modify it or duplicate it, that doesn't work either. That's too risky. And anything that has an impact on that regulatory team, remembering that there's only two or three reckless managers, there's only five or six in house, legal, maybe ten cyber people in the organization. If you build a solution that has a big impact on them to run and govern and train and manage, it won't work either. So what you have to do is full coverage and no impact. And until we could achieve that, which we launched now, five years ago, and defined that new category of AI, it really just wasn't possible. So now that it is possible, the regulators are taking notice. They're not really taking it seriously anymore. When big corporates say, oh, it's just too hard, we just can't, it's just too hard, they say, well, we can because we use AI like we use Castlepoint, and it actually isn't that hard. So maybe you should be using AI as well. No more excuses. So the needle's really moving. But in all fairness, this capability just wasn't really available at the time of the ANU breach. And even though it's been available for a while, it takes time to get penetration and change the way people do things fundamentally. So maybe another five years, everyone will be managing their stuff and this problem will be obsolete. That's what I hope. Anyway. I've got my fingers crossed.
[00:28:33] Speaker B: Just going back to your time as an auditor for a moment. So you're auditing organizations and you're saying that, in your words, that they're all non compliant, like failing, basically.
So you don't think someone like, I don't know, was looking at all of your audit reports and you're saying it's failed and no one's thinking, well, hang on a second, why isn't anyone compliant? Why are they all failing? You don't think at that point people are asking those questions a bit more?
[00:29:01] Speaker A: They for sure have been like, if you look at the Australian National Audit office, they've been releasing negative audit reports since about 2007, about how federal government is managing their records. Every few years, they release another one and say, it still sucks. It's still no good, it's worse than ever, and nothing really happens. So now that it's becoming such a big cyber problem, and now that privacy law is becoming much more robust than it's ever been, there's now more pressure from more sides to get this right. When it was just the National Archives and the audit office saying, guys, please manage your records, it's important. Not really taken seriously, but once it's a cyber problem, once it's a privacy problem, there's a lot more pressure from a lot more commissioners. Right, coming down. So we're seeing definitely the attitudes and the focus changing. So we're definitely seeing an improvement. People are taking it more seriously. They know they need to succeed now, and they will once they start adopting new technology to make it possible.
[00:30:05] Speaker B: So do you think with the updates to the Privacy act that you've spoke about today, as well as the major data breaches last year, with again going back to people's records being out there that weren't even active customer for. I think I read ten years someone was saying in an article, do you think now, unfortunately, other people taking the hit has probably put a bit more of a focus on for the industry as well as the regulator coming in a bit more and the updates of the Privacy act, et cetera.
[00:30:32] Speaker A: Yeah, it definitely has. And not least because not only is there constantly new and fresh cyber guidance and regulation, there's this kind of global upswing in governance around privacy, as I mentioned. And really importantly, there's a new focus on artificial intelligence and the regulation and governance of artificial intelligence. AI is a big part of our problem. So if you think previously about those terabytes of word documents we have, it would take a human, even a really fast human, doing a pretty fast and loose job, it would still take you a few days to write 10, 20, 50,000 word report. Right?
With generative AI, you can create one of those in minutes.
We're seeing now, with the advent of generative AI, we're seeing even higher volume, higher variety and higher velocity of content and lower veracity at the same time.
So AI is a risk for us. For information management, AI makes more information. We're constantly collecting more and more data now, like it's exponential, the amount of information that we generate and we keep.
So regulation is being introduced. The EU AI act, the ethical AI act in the EU will probably be the first one to be formally written into law. But there are similar proposed acts in the UK, New Zealand, Canada, USA and Australia and other advanced economies that will regulate the use of artificial intelligence. And a big part of that regulation is contestability and explainability. So what we're going to see, I think, with these information governance AI solutions is that since we launched ours five years ago, there's been an industry trend of other vendors starting to do what we do, which is fantastic. It's what we need. Like, we need AI to solve this problem. But we're going to see most of those that have been gradually going up, we're going to see them drop off very steeply because this legislation is being introduced that says if your decision that comes from your AI, the decision about whether something should be kept or whether something should be destroyed, for example, if that decision is obfuscated, if that decision has come from an algorithm that you can't explain, that no one can then contest, that no one can argue with, you're not allowed to use it, you're not allowed to use it for anything that could impact the citizen. So the reason for that is things like robodet and things like the horizon scandal in the UK, which is where if you use an algorithm and the algorithm's wrong and people want to contest it and say, well, I think this is wrong and I don't owe this money, or I didn't perpetrate this fraud, it's not going to be good enough to say, well, algorithm says you did, we trained the machine learning and algorithm says you did, so too bad that's not allowed anymore. So if people want to contest a decision and say, well, why did you destroy my record? Like, I definitely still needed that. You can't just say, oh, the machine learning was trained and this is the outcome. What we've seen is we kind of define this new category and then we've seen a bit of an upswing in machine learning based classification tools for records retention. And we're going to see a drop now in the ones that don't use explainable AI. Because I'm an auditor, right? I built in explainable AI from the beginning because years and years ago in Australia, it was already a best practice standard. It's just now it's become law. So we're okay. But we're going to see a big change in how artificial intelligence is used, because if that AI is not explainable and therefore not contestable, it just won't be allowed to be used for regulatory purposes.
[00:34:20] Speaker B: So hypothetically, if it's like, okay, KB, you owe some money. And I'm like, no, I actually don't believe that. I do. I could contest it. And if there's nothing, the algorithm can't be justified, what happens then?
[00:34:31] Speaker A: Totally, well, they can't enforce it, they can't uphold it. So what happened back in the is pre AI, but it was still algorithmic. Right here in the UK, the post office implemented a new system, a fujitsu system that had an algorithm that started flagging postmasters, like the people who run the little mum and dad post offices around the country for theft. And saying, like the algorithm was saying, I've crunched the numbers and I'm pretty sure these people are stealing. And they prosecuted a whole bunch of people. People went to jail, people got divorced, people went bankrupt, people died by suicide. And the big problem was a, that the algorithm was wrong, but mostly that people couldn't contest it, the burden of proof was on them. So when they got issued with this notice and they'd say, but I haven't been stealing anything, like, explain this to me.
The post office running the system, this horizon system, couldn't really explain how the calculation was done. And they wouldn't explain how the calculation was done. They just said, well, burden of proof is on you, really, to prove that you haven't been stealing. And that was just deeply unfair and it's that kind of thing. And what happened also with robodet, where people were just issued those debt notices, and then when they contested, told, like, no computer says you've been overpaid, you need to pay it back, that causes significant harm. So ethical AI is explainable AI. AI that can be explained in plain language, how it came to its conclusions, and therefore can be understood and therefore can be contested. So that's going to become law that we use those. So machine learning, neural nets, generative large language models, none of those are what's called explainable AI. They're closed box, or what we used to call black box systems that you can't see inside of. And that's a real problem for any regulated outcome.
[00:36:23] Speaker B: So just to probably press a little bit more, Rachel. So do you think that people are going to. Well, like, what's happening and that can't be justified and sent to their algorithm, and then there's going to be lots of issues. Are we going to start to see this happening a lot more than historically? Is this something that now we've opened, like, another can of worms a little bit here?
[00:36:44] Speaker A: Yeah, we totally have.
The reason that people are really worried about ethical AI is because of the really frothy stuff, like, you can't use AI to decide whether I should get probation or whether I should get bail or whether I should be sentenced, you know what I mean? Without being able to explain the use of that AI. You shouldn't be able to use AI to decide whether I get credit or what my interest rate is without being able to explain that AI. So they'll be the first real global case studies is people that have been really unfairly discriminated against by an AI process. But what will be the second wave is the thousand tiny paper cuts of harm with information governance. So the australian government recently released a discussion paper asking for feedback on this proposed regulation. And our feedback, Hustlepoint's feedback, was, you need to be thinking about the harm that comes from mismanaging information, not just the harm that comes from deciding who goes to jail and who gets a loan, but the harm that comes from spilling or sharing information that you shouldn't have, the harm that comes from destroying stuff like those Windrush arrival cards before you should. That can be significant, and that needs to be incorporated into this thinking, because so far, the ethical AI legislation, all the policy mapping they've done, doesn't include any mapping to national Archives policy, and seemingly doesn't include any consultation with the archives either. So we've suggested, hey, if you really want this to work, you need to understand that most of what we deal with day to day is data, and we need to manage that data as well as we manage other kinds of risks. And therefore, we need records managers involved. We need the archives involved in thinking about how we do this safely.
[00:38:33] Speaker B: So, Rachel, is there any closing comments or final thoughts you'd like to leave our audience with today?
[00:38:38] Speaker A: Look, I always try to just really bring home the fact that what we do with information is important. It's not just about ticking a box. It's not just about us being able to find it later so we can more conveniently access it, or we can do our job better. We have to think about the person, the community, or the environment at the end of the line of that information. If we mishandle that information, what can happen? What will the impacts be? Not just on us and our job performance, not just on our audit findings, but on humans in the community. And like I said before, we definitely need people thinking about that harm from a cyber point of view. Like, oh, should I be protecting this? What could go wrong? But we need them thinking about that from a lifecycle point of view as well. If I delete this, then what? If I keep this forever, then what? So we need organizations, boards, executives thinking about the impact and the risk of data. It's not something you can just hoard, but it's also not something you can just thanos snap out of existence when your storage runs out, or when you think it might be a bit risky to keep it. You really have to trade this off very carefully. And if you've sorted through, and if you've made decisions based on evidence and consultatively, then your decision, whatever it is, will be defensible. Even if it's the wrong decision, it will be defensible because humans can make mistakes. But we try our best. If we make those decisions based on half baked algorithms, those decisions will not be defensible, and that will not be fair. So keep thinking about the risk, keep thinking about the value, and think about how you can do your best to make good decisions based on good data.
This is KVcast, the voice of cyber.
[00:40:33] Speaker B: Thanks for tuning in. For more industry leading news and thought provoking articles, visit KBI Media to get access today.
[00:40:41] Speaker A: This episode is brought to you by Mercsec your smarter route to security talent Mercksec's executive search has helped enter rise organizations, find the right people from around the world since 2012. Their on demand talent acquisition team helps startups and midsize businesses scale faster and more efficiently. Find out
[email protected] today.
