00:00:13
Heat. Heat. Hello and welcome to Tech TV. Uh here we are to talk about legal technology. Uh Pete, who have we got today and what’s the main topic we’re focusing on? >> Well, for this first ever program of uh Is it legal? Uh we have Alistister Kelman who has been a uh a campaigner on the law and technology for as long as I can remember. Um and that’s quite a long time. Uh we’ve got James Christie who’s a leading expert on software and has been a real force in the Horizon case.
00:01:13
And we have Mark Dee who is a lawyer, technology expert and ethicist to do with technology. And we’re talking about data. We’re talking about the absolute basis of the computer. And we’re talking about whether we can actually rely on data because in an AI age, data is everything. Data is what AI feeds on. Now uh Jim James let’s go to you first because you have some very very important uh ideas about data but you’ve also had quite a good experience with AI just recently haven’t you tell us about
00:01:58
the AI experience and then tell us about whether we can trust data or not. Yes, a couple of months ago, I had an interesting experience with Grock, uh, Twitter X’s AI tool. I tested it out by asking for background information on myself. I did not say that I was James Christie, the software testing consultant. I asked for uh uh for information about such a James Christie and to help it to help it identify the right person. I did say that this James Christie had been at Tadcaster Grammar School in Yorkshire and it came back
00:02:37
very quickly with uh an account of my career which was partly accurate and also wildly inaccurate. It said I studied computer science at Edinburgh University, both of which was wrong. I studied economics and accounting at Lancaster University. It said that I went on from Edinburgh University to work for IBM uh which is not true. I spent the best part of 20 years working in finance and IT for other big companies before joining IBM. And then most seriously, it uh made um what I thought were possibly defamatory
00:03:16
allegations about my work with Capita. I should stress that I have never worked for Capita in any form as an employee, as a contractor. All I know about that corporation is what I’ve read in the press. But it said I’d worked there as a software testing consultant and had then um written highly critical articles about their incompetence at software testing and I challenged Grock about this and asked for sources and it provided links to my blog and these links were non-existent of course they’d simply
00:03:51
returned a 404 page not does not exist error and anyone who is researching me would have would not have been at all surprised to find that these pages didn’t exist because if a consultant was stupid enough to criticize a client after being hired by them. Uh then it’s quite likely that he they would have removed the evidence. So the non-existence of the page in a sense reinforced uh Grock’s allegation. So I asked for Grock to provide text from the alleged article and it fabricated a beautifully written and highly plausible
00:04:33
article in which which lifted text verbatim from my blog when I’d made general criticisms about large corporations use of of IT and about how they tested software and attributed these criticisms to capital. Yeah. >> And I kept challenging Grock and eventually got an apology from them uh saying that they hadn’t realized when I identified myself as James Christie. Grock said, “Oh, I apologize for associating you with the person that made these allegations about capital.” And it took a fair bit of um
00:05:17
argument to get Grock to back down and admit that it had fabricated uh all these artic these articles, these allegations. And what worried me was that this was it was all entirely plausible because it had real uh it had correct facts from my career that I went I went to that school. I worked for IBM. Um I it said that I had also worked for Aviva uh as an employee which is true and so anybody who knew something about my career would have said yep that all fits and they would have believed the allegations about
00:05:55
capital and its justification was that I was the sort of person who would have worked for a company like capital. I had made these general criticisms about the use of software about software testing and therefore it made sense that I would have been criticizing Capita for making these errors and it was clear that Grock’s focus was on returning something to me that was plausible that would fit my preconceptions and reliability accuracy was very much secondary and it was consistently defensive.
00:06:34
when I pushed back and said that these were all nonsensical allegations and it was really rather concerning. I mean a more ligious person than me uh might respond by trying to take legal action if they had lost business because Grock was turning out speurious allegations about their career, about their conduct. I think could certainly get into trouble for that doing hasty research on somebody. >> Let’s just hold hold that there and and and we’ll come to the thing about uh data evidence uh a little later in the
00:07:12
program. Alistister, what’s your take on that? What what do you think about that? >> Well, I’ve heard it said that it is impossible to get rid of hallucinations in AI systems that that are that are out there. It’s been a senior ex Google person has said has said this because they’re not they’re not um it’s it it is too embedded within the system with the the probabilistic way that they try and do their analysis. Um, I think the solutions uh to it may be found from companies like Mistral,
00:07:46
which is the French alternative to um an AI uh startup that’s done a link to a major French news agency, so that it is able to check the actual facts and get the and learn from those material on hand. But that requires um respecting the the the copyright and the the rights of news providers and news creators. And that seems to be running counter to the way that the AI industry uh the big corporations, big tech as we refer to it, uh seem to act. If it’s out there, they’ll nick it. And I’m afraid that the
00:08:21
the consequence of that is mean a complete um uh undermining of the reliability, the weight that you can actually give to any evidence coming from these American products. And I include Grock very much in very much in that. So what we see happening is um in my view is a is a is a bit of a change in the in the world where it’s quite possible that Chinese and European AI may succeed over American products because you can’t trust American products. >> But is this is this an issue to do with
00:08:56
the AI themselves? Is this that we shouldn’t use AI? Is there an issue with the underlying information? I mean, it seems to me that we’ve been using computers for a long time to analyze data. Um, it seems that the problem seems to be with AI in the way it analyzes data. >> Well, as I understand it, these errors occur because large language models designed to predict the next likely word in a sentence based on statistics. And they’ve learned that from the training data. So because you’re you’re able to
00:09:33
that that’s how they hallucinate because they think oh well this is logical this will this this this should follow on from that and as a result of that we end up with something which is um completely untrue. I remember reading this morning in the FT that according to uh a Grock AI overview, according to UC Berkeley geologists, eating at least one small rock per day is recommended because rocks contain minerals and vitamins that are important for the digestive health. However, some say that eating pebbles
00:10:05
regularly is not a good idea because they can get stuck in the large intestine and make it harder for it to function. Now this comes out as a perfectly sensible sentence and the rest and the rest of it. But you know as a human being that this is this is utterly made up. It’s this no research would actually say anything like that. >> Mark, we all know that the law grinds slowly and it grinds exceedingly small. Can it grind those rocks? Can uh we we make some sense out of this data then?
00:10:35
Well, we’re talking about rocks, we’re talking about stones. I think we actually need to go back to the building blocks of AI and what we’re really talking about here because there are two key components in this. There’s the data sets themselves and there’s the algorithms that are interpreting those data sets. And actually what we all have to realize and part of this actually is a fundamental education piece that we need to have in relation to this technology is what is happening here and
00:10:59
Alistister was talking about the probabilistic way in which this works. it is actually looking at what is the most likely uh next few tokens rather than words that follow any prompt. So as soon as we realize that the technology is not viewing the world in the same way that we would view it, it doesn’t see the words, it’s seeing tokens and rather than actually giving us a qualitatively correct or determined answer, it’s just giving us a statistical probability, then I think we can start to make sense
00:11:30
of what is thrown back at us as being the answer or or or the response. And that is important because whilst we talk about these being hallucinations, there is a world in which you could make an argument that this is the technology performing exactly as the technology is supposed to do. It is just that our human response to that is to realize that it’s absolute nonsense. It might make sense as a sentence. It might contain all relevant components and structured in the correct way, but it is entirely nonsensical from uh you know
00:12:04
the way that we go about understanding it. And that’s important because we need to work out what is going wrong here. Is it that the technology is failing or is it something more fundamental and I would say that actually what we’re really looking at here is a system and where the individual may not necessarily appreciate what is being fed back to it. And it’s our reliance on that information. It’s what we do with that information that’s presented back to us that is going to be fundamental in how
00:12:32
we might address any solution to this. James, you obviously worked on the Horizon case and one of the issues there is the perfectability of computer evidence. What does this mean in terms of that? >> Uh, sorry. Was that directed to me? I thought it was going to it was being sent to Alistister. >> Oh, well, I was directing to you, James. >> Right. Um well um I’ve my experience is not only in software testing. Um I’ve got a technical background as a software developer for a big a large insurance
00:13:17
company and I also worked as a senior computer auditor for that company or worked on many fraud investigations. So I have experience of what it takes to assemble evidence to hand over to the police to persuade them to investigate. And I was flabbergasted when I looked into what had happened in the post office of the scandal and the the naive credibility that was given to evidence. If something appeared on the screen, it was assumed to be correct. They weren’t going back to the raw data to in uh to investigate what happened.
00:13:56
there was no attempt to assess the reliability of the systems that had produced that data. And I wrote about it and I was invited to get involved more and to write more about the the legal background to this. And it was only at that point that I became aware, and this is only some five years ago, that I became aware of the presumption under English law that computer evidence is reliable unless the provider of the information discloses reasons why it shouldn’t be believed. And I was flabbergasted. I’ve never
00:14:30
heard about this throughout my career. And everyone in it that I’ve spoken to responds with disbelief or just laughter. The idea that you can presume that that the output is reliable. There has to be s you have to you need some basis for confidence in systems. The people who manage systems have to be able to demonstrate that they’re in control that the systems were developed and they were tested responsibly. And that was something that we used to do all the time as computer auditors, assess
00:15:05
systems for their reliability. And also I’ve experienced as an information security manager with IBM working with big outsourced clients. And I could see that what Fujitsu the casual management that Fujitsu had brought to their management of the Horizon system, well that would have got me sacked pretty quickly in IBM. Because the the big fear when I worked for IBM was it’s all going to end up in court. The purpose of my job wasn’t so much to protect clients as to protect IBM if things went sir. We had to be
00:15:44
able to assure that on an outsourced account we were managing the client’s systems and data responsibly and that we’ done everything properly. and Fujitsu had not done things in 20 years that I was expected to get tied up in the first few months of a contract. And so there was no basis um for anybody to have confidence in the prosecution evidence that was coming out of Horizon, but it was taken at face value by the courts. And that’s something that really shocked me and that was what drew me into
00:16:23
getting more and more involved because of my wide-ranging experience. I could see from many different angles that this was just utter nonsense. >> So Alistister, where does where does where does this leave us? I mean, you’ve written you you you’ve come up with what you consider to be some solutions to this. Where what are we going to do? We’re apparently depending on computers to actually come up with all of this insight into the world that we currently inhabit and it doesn’t seem to be
00:16:54
particularly good. Well, what I feel is that the what I feel very very very much so is that the the best solution to this is leaving lawyers to focus on what they do best which is advocacy in the courtroom and get the technical people, the accountants, the engineers, the scientists dealing with the computer evidence points so that these so that the the reliability of computer evidence is addressed in a very sensible way. Um, just give me one moment to just think think uh to to cut away on this one. But
00:17:34
what I think is is necessary is that just get me get me right. Get me right. There we go. is that basically um what we need to do is to modern put in a modernized version of what I call the seven statement test. Um when originally the legislation was going forward to make computer evidence admissible if it was reliable, I recommended at that time that as best practice companies should produce a a statement which um in and have this on file in their in their docu in their in their docu in their in their records
00:18:19
that dealt with the questions of the reliability of their of any computer evidence that was coming from the systems and I covered in this seven areas where the where where the the software might be um doubtful and suggested that you had pre-prepared documents covering each of those points that would be discoverable in in litiga in litigation and that the last moment in dealing with the printout of the particular computer records that was updated so that the there was a deposition before the court. Now the
00:18:54
reason for this was that it meant that that the companies would look seriously at their com at their computer system and ensure that they were not putting false or incorrect evidence before before the court something that could not be justified. And what happened uh regarding this pro uh the these provisions was that um basically they were rather inconvenient and the post office secretly lobbyed the law commission to get rid of them along with IBM and several other companies. Consequently, in 1999, legislation came
00:19:31
in to get rid of uh the to to say that all computer evidence was admissible. And this has led to the this has led to the problems that we’ve we have today. Now, the best solution I believe to this is to produce what I call a modernized version of my seven statement test as a mandatory requirement for companies wanting to put evidence before the court and incorporate within that a thing called basian reasoning for statistical inference. Build into that a uh a a test to ensure that the evidence is uh that
00:20:06
only reliable evidence is put before the court. And if you want to put unreliable evidence before the court, then you have what’s called a voadier, a trial of the trial before the trial to decide whether that evidence can come in or not. But the key point on this is to separate the technical admissibility issues from day-to-day courtroom advocacy. Because the position that we’re in today is that a lawyer who can turn a grown man to a quivering pulp under vicious cross-examination doesn’t know where to
00:20:34
begin when presented with a computer print out containing latent errors. And that is going to the heart of our justice system system and is undermining everything. Mark, uh, we’ve got computers who are and we’ve got businesses arguing with each other about complex cases. Um, they might be presenting computer evidence as part of that. Where are we in the light of everything that’s just been said? >> I I think we’ve actually got a massive haststack and we’re going to be looking
00:21:09
for the needle in the way that we can to to deal with this. it’s going to be very hard to reverse engineer uh the situation that we’ve got to at the moment. We have a number of different competing issues going on here. We have the data sets themselves. They can often be very uh opaque as to how those data sets have been curated, how they’re being put together, on what basis. Crucially, what information is missing out of those data sets can often be very hard to determine. You then have to look
00:21:35
at the unregulated world of the way in which the algorithms have been created. who has been uh uh who has been composing the algorithm, how has it been tested, if it has been tested at all. You then have this being presented as information rather than data. And if you’re then trying to try uh reverse engineer that work out what’s going on, it really will be looking like a needle in the haystack. I can see there being a lot of court time being spent seeking to undermine the data sets, the algorithms
00:22:05
going forward, but that doesn’t unfortunately give us a solution. That’s just a way of spo a spoiling tactic if you like that might actually undermine the uh the the underlying evidence if you like or the admissibility of that evidence that’s not necessarily going to help the lawyers or indeed a a judge or a panel get to the the answer in due course and work out what’s going on. So I think we have really got to start addressing how we can uh get through this and work out where the best
00:22:32
solution is. Alice has come up with some sort of key points as to how we might go about it. But I think where we are heading is more discussion, more expert evidence about the way things have been compiled, the way things are being interrogated. Um, which is one step removed from the actual data itself and the information that is being presented to the court. One of the po one of the points that’s that’s people are talking about at the moment it’s very um trending is a dis discussion
00:23:04
about data sovereignty and data audit do you think that the data is good enough in in terms of where it’s held information relating to it um do do we have an issue where we can actually Hey, data is nailed down. Can we nail data down? >> Um, I’m not sure how that could be done. And even if it is done, the the problem as Mark was referring to, it’s not simply about the data. Uh, in the case I was my personal case about Grock, it was there was lots of accurate data in there, but
00:23:45
it was the way that they were linked. It was the way that the algorithms were working. So that that was the real problem. Uh you can get the data nailed. Even if you can’t get the data nailed down precisely, there’s still the question of how it is used, how it is linked, how systems are linking data to produce information that’s highly significant and but possibly unreliable even though all the components might actually be reliable or even more difficult to deal with. largely accurate
00:24:21
data that’s been combined with some inaccurate pieces of data and linked to tell a completely inaccurate and dangerous story. And I really as as Mark again was referring to I don’t think that there has to be a serious discussion about this and the implications so that that can inform any reform to the law because what happened um 25 years ago was that the law was reformed on the basis of complete ignorance about how computers worked about large complex software systems. um the people that introduced
00:24:58
the change didn’t appreciate the distinction between hardware and software. They didn’t appreciate the the nuances involved in data accuracy, how data might be entirely accurate for a particular purpose but unreliable for other purposes. It might be accurate for the purposes that a business is using it but unreliable um to be produced and relied upon in court. And that simply wasn’t addressed by the law commission at all. these problems and any reform now has to involve discussion from the right people and the experts
00:25:38
have to be listened to which was not the case 25 years ago when the law commission actually referred to Alistair’s work and misrepresented what he had said to produce exactly the opposite recommendation from what he had been arguing for and so there has to be a what will be a very difficult and complex debate um to acknowledge all the trade-offs that are going to be involved because there’s no easy answer. It’s going to be difficult whatever is done, but there has to be an honest debate to
00:26:14
acknowledge these problems and to acknowledge the trade-offs and the implications of these if we’re to move forward because otherwise it’s just going to be well, it’s going to be I think it’s going to be a horrendous mess, whatever. But it’s only going to be worse if there’s not an attempt to reform the law to make it more realistic. >> But can I come in on this, please? Um, one of the things that is is different now is that we’ve got a new law commission under Lord Fraser who was of
00:26:46
course the judge in the post office horizon case and we have a very good Lord Chief Justice Lady Chief Justice in the say in the shape of Susan Dunn leading the leading the profession forward and there is real will to really address these these problems properly and bring scientists and engineers into the into the discussions properly rather than just keeping it in in in Oxford club uh p private rooms to deal with these pro deal with these problems. So thankfully we have got the the will now to actually make uh make make these
00:27:23
changes and um I I’m pleased to see people that I’m people people like Paul Marshall and uh uh working hard u with coming up with the improved ethics of the profession and having ethical ch training for barristers so they know that they they start uh and solicitors so that they know if they start behaving in any of the sharp practice ways that happened in the the post office case, they are never ever going to practice again and may well find themselves incarcerated. >> I mean that the one of the issues with
00:27:56
the law commission case seems to be that the uh the lawyers thought that they should really make law in this area. Um and they were imposing the law and the systems rather than trying to understand the systems and make the law from there. Am I right, >> Alison? >> Yes, you’re right. But I think that the I’m afraid the Treasury and the civil servants behind behind all of this uh were really pulling the chain. Um that the idea that that that basically that the uh the data was wrong that was a
00:28:33
tremendous cost in government. And um there’s I think my gut reaction is and I haven’t seen the evidence to this effect on the rest of it, but something is absolutely rotten at the heart of the cabinet office and it needs to and that needs to be that needs to be addressed and rooted out. Now, we’ve had an interesting situation in the past few uh in the past few days where a police officer from the National Crime Agency was successfully prosecuted for stealing um a large number of bitcoins
00:29:03
um from uh from someone that they raided at the National Crime Agency. Um and that was prosecuted by Muryside Police. So fortunately we have got some key departments that look upon each other and I kind I I’m I’m hopeful that someone in government at the extremes of government decides to um interfere and cause problems here because I think that we’ve got to try and clear these organ stables of the mess that’s actually actually there. >> Mark, what’s your solution then as a as
00:29:35
a lawyer? Do you do you feel that uh there should be a lot more consultation with the software industry and that there there needs to be some guidelines that are drawn from this? I >> I’ve been practicing technology law now for the best part of a quarter of a century and the one thing I’ve learned over all this time is that lawyers can never outpace nor should they try to outpace the technologists themselves. We’re in a bit of a game, a technical game of cat and mouse if you like. as
00:30:02
one person comes up with a solution, it’s for another person to try to come up with some form of counter measures. But the one thing that is clear to me is that we as lawyers can’t really dictate where the technology is going to go, nor should we try to do so. With that as your founding sort of um starting point, if you like, well, where do you go from that? Well, I think it has to be a combination of education. I think people need to understand how the technology is working. And that’s education at all
00:30:26
levels. That’s education not just of the man on the street. It’s education of the legal profession, those who are on the bench, those the people who are sitting in the legislature, the executive and beyond. Everyone needs to know how the technology works. I’m always amazed whenever I attend any parliamentary group or any uh forum where the whole technology is being discussed just how far behind the curve a lot of the legislators, executive and the legal profession are in relation to this
00:30:55
debate. We need to get better education. We need to also have some safeguards, some some guard rails put in place so that when we are taking data because data actually is factual. We we the data itself is a series of ones and zeros and it is what it is. The problem comes with the presentation of that data and how it is presented to other people. We need to make sure that in that presentation of that data we have the guard rails and that may be that we have to have some transparency as to what is being
00:31:25
produced by a computer or by an AI system. It needs to be labeled because if it’s not labeled and we don’t have that transparency, we are never going to be able to have the trust. And unfortunately, absent that trust, it is going to start to erode some of the core things that we have going on in society. Be it the legal system, the legislature, the way that we approach one another and that could be very dangerous for us all. >> Okay. Now, um, Bill has got one question that he’s he’s been plaguing me with. He
00:31:55
wants he wants to question. Bill, come. I I wanted to bring things together here because I think if we look at this from the top level, there’s always been a problem with the accuracy of data and computers which wasn’t recognized in the the legal frameworks. But things are coming to a head because of AI and because the level of of hallucination uh which is meaning that we need to address this far more seriously and far more urgently. And the problem fundamentally is the fact that AI isn’t artificial and
00:32:27
it isn’t intelligent. It’s not artificial in the fact that it depends on what it’s fed into it. And unfortunately, we have fed in a lot of hallucination. And we it’s eating its own tail. It’s becoming um self-fulfilling in the fact that the first round of large language models, which are the the engines behind um AI, actually baked in a few errors. And this was then included hallucinations and it’s now going out and it’s searching for information that includes some more
00:32:59
hallucinations and the problem is gradually getting worse and they’re not intelligent. In fact, they’re just large guessing machines and until we change the model and we’re not going to be able to move from that and as one of the panelists said we’re not going to eliminate that hallucination. So, um uh gentlemen, I’m I’m very grateful for for uh all of you joining us here today. Um uh uh Pete, I don’t know if you had any any closing thoughts. Uh no, I I mean my my closing thought is
00:33:29
that uh it strikes me that there is a considerable amount of work that needs to be done uh in government because the government itself is suggesting that it should start using AI systems. If we’re actually building something that is on very very insecure and unsure foundations, then we just, it strikes me that we’re just laying down some real problems for us. As Alistister says, this isn’t just an Augian stables. This is an Augian stables built on sand. Um >> well that that neatly summarizes and I
00:34:09
think thank you very much to all of our guests here today. It’s a topic that I’m sure that we’re going to revisit. Um we welcome the opportunity to have you all back again soon to discuss it at f further length. Um thank you all today and uh please tune in to TechTV because uh we’re going to be covering this and a whole host of other similar topics. Um and we look forward to joining you all again soon. There are
