The software company was also responsible for collecting data and for witness statements about the data for prosecutions not just for code. There were bugs that the company knew about which affected both the collection of data presented and the collecting of data. The conversations about costs related to these bugs in some instances it seems. Additionally the witness statements were written possibly with bias in them.
Right now on YouTube there are Fujitsu staff being asked questions by the enquiry on this topic.
Worth also saying that some bugs were discovered in a product which was due to be replaced by a new system so it was considered low priority to fix. However it impacted cases of several years before it was discovered.
I think maybe that 2 Fujitsu employees are either investigation or being currently prosecuted by the met police, so there are criminal proceedings being under taken right now in relation (I guess) to these bugs in the older version of the code. There's not much info about this currently for obvious legal reasons.
My burning question about this: What was the 'state of mind' of the higher-ups at the post office when they started prosecutions and then continued prosecutions into the hundreds? It's clear that the higher-ups at the post office knew these issues were caused by software rather than the postmasters, they must have known as early as 1998 or 1999.
Maybe at first they must have thought that these were one-off issues and that they could just prosecute one or two postmasters, sweep the whole thing under the rug and forget about it. But that doesn't explain how they continued prosecutions into the hundreds years down the road. Like what did they think the endgame here was? Just prosecute indefinitely to cover up for the system?
> What was the 'state of mind' of the higher-ups at the post office
The Post Office was a money-hole; it hadn't made money since privatisation. It was effectively still a government agency.
UK government agencies have a long history with ICL, the UK "champion" mainframe maker, which was taken over by Fujitsu. It appears that this "champion" status was inherited by Fujitsu.
ICL couldn't compete with companies like IBM on merits; they had to have backdoor support from politicians and the civil service. I don't know how that worked; but if MI5, the armed services, the National Grid and so on were reliant on ICL, then it's understandable that government would want to encourage agencies to buy ICL.
So I suspect that the PO were pressurised by ministers and civil servants to buy Horizon from ICL/Fujitsu, and further pressurised to keep schtumm about the defects. Well, that's my guess: the higher-ups owed their jobs and reputations to ministers and civil servants, because they were effectively employed by a government agency, and it was losing money, and the government wanted to protect the reputation of ICL/Fujitsu.
I don't understand the state of mind of the government elected in 2011. The whole system was set up under the previous government, it would have been costly and difficult to come clean and do the right thing, but they could have blamed the entire thing on Labour and made political hay over it.
Instead Ed Davey doubled down on the lying and persecuting ordinary postmasters, over an issue that until then he had no personal accountability for. This is what democracy is for, to provide the opportunity for a reset, to change course, and a mechanism for accountability of those who made the mistakes. Instead he allowed himself to be co-opted by the system.
Most 'enterprisey' systems of the era were just as shoddy as Horizon. The problem was that the post office were in a position to aggressively prosecute users and decided to do so (and then the whole chain of cover ups and dissembling for years after as widely documented).
All in all its a lesson in just how easy it is to cover your arse as an executive in a large company, and get away with pretty much anything. Only the most tenacious work by many people over decades has any chance of holding you accountable.
It seems to me that the Horizon system is just a simple accounting/POS thing. I can't understand how it could be built so shoddily. It doesn't even have that many clients: there are less than 12k post offices in the UK.
Governments often have a ton of overly complex and poorly documented rules. Add in bad incentives and people put into roles they are poorly suited and it can go to hell pretty quickly.
Here is an anecdote from a past job.
1. Years ago (decades), someone made the contract rules. They did it in an ad hoc not very systematic way. They assumed everything would be judged by a human and be judged reasonably.
2. That person left and the department expanded.The rules had to become more formalized, but still room for plenty of exceptions. An exception could be a penciled scribble in the margins in a file or later on, a sticky note.
3. Time to digitize. Unfortunately, computers like standards and need every little rule must be programmed.
4. A business analyst with no real understanding of the system is hired to generate requirements for digitization. The current overseer of the rules, someone who doesn’t use a computer much (yes, these people still exist) kind of just nods every time he creates these tremendous flow charts. this is the first degradation of knowledge. Even then though, this business analyst is not technical or legally trained, so doesn't appreciate why defining things like "end of day" or "contract billing cycle" precisely is crucial. even if he could, many contracts had sticky note individual definitions.
5. The business analyst goes back and feeds the information into the ticket mill of Scrum. To be nice and agile, work is done in bite sized increments with no regard for context. This is the second knowledge degradation. All those flow charts are chopped into little boxes. Devs see ambiguities, but have no interest in pushing back.
6. Developers only have the tickets to work on and quotas to meet, so write the code as requested. Testing is another set of people, so code is just tossed over to QA. QA has no more context than dev, so if it matches the ticket, they approve it.
6a. Nobody budgeted for automated testing so things break over time for lack of tests. Product Owner wouldn't grasp the concept, business analyst had a budget and wanted to deliver more features, and devs didn't want to fight about it.
7. Final increment goes back to product owner who doesn't understand and just nods. A piece of software that doesn't work all that well gets approved. Repeat for years and constantly adding post-it note rules as code and you have a system that has lots of errors.
At some point one should just give up and do it by hand. There is so much overautomization nowadays that in practice are time sinks and deadlocks the org.
A sane system would be 'contract A type' and all other be done manually or something.
There was a quote from one of the insiders[1] which said there was a team of 8 developers, 2 knew what they were doing, a couple more were competent, and the other 4 were incompetent (on the Joel scale, in the idiot + gets things done quadrant). I can imagine that trying to write a reliable product in that environment was difficult to say the least.
It’s not ‘simple’ EPOS. Some of the things sold were local and held at the counter, e.g books of stamps, others were centralised virtual services (eg taxing your car). Even today the Post Office has a range of weird options not found elsewhere, e.g use of ‘postal orders’ that allow people to send money to recipients who don’t have bank accounts. All of these can be accessed at a post office counter.
'Shoddy', being a euphemism for 'beyond knowingly-being fucked-up, so much so that we'll cover it up oh fuck we're still alive this wasnt what we were told would happen'?
Slightly different take: it was a state contract fulfilled by the company best able to impress state procurement. Quality is rarely on the agenda for this sort of procurement, because if it goes wrong they can blame the management who were overseeing the implementation. Thus it was unlikely to be great from early on.
I'm currently working on a 'simple' integration between an accounting and crm software. I'm already pulling my hair because of various exceptions, limitations and whatnot. I think Horizon was anything but simple.
Yes, you're probably right. But this is the part that really bugs me :
"The inquiry heard that in 2008, a glitch in a system called CABSProcess, which automatically summarises a post office’s transactions at around 7pm daily, resulted in users working at the same time having balancing issues."
This, for an accounting system is just fucking inexcusable!
I worked on a huge accounting project for a financial institution processing 2M account bookings per day (at that time, it's a lot more now). A lot of effort was put into the audit log (every booking was written to an append only log) and into the reconciliation process. When it was 20 Rappen off there was digging until the inconsistency was found.
That transactions are not completely isolated when that CABSProcess kicked in is just completely and uttelry inexcusable.
Yeah. I work as a PM in a company that builds integrations between ERP and "heavier" accounting systems. It is absolutely not simple, to the point where we are building an ETL-style engine to drive things. It's not "copy record X to Y".
Certainly there were bugs that made it seem like there was more money coming in than actually did, but it seems like also there were capabilities in the platform that would also allow editing records to make it seem like there was actually less money coming in, which would enable skimming money out of the system almost (?) invisibly.
Some details are there, see the inquiry website. From what I gather the "covert" remote access could only add records rather than edit records. Like putting something on a message stack. However you could add a record to edit a balance I imagine. Where "records" are "messages". I don't think you could actually get money out that way only remotely. Also a record of the record would be made, you couldn't erase remote access activities on the stack.
I also don't think this capability was ever suggested anywhere as being used in conjunction with a postmaster to skim money out. Maybe its possible in theory?
If I'm reading the article correctly, the headline claim is about a bug that was discovered in the original Horizon system right as it was about to be replaced with a new system, Horizon Online, at which point the bug fix would be completely useless. Other things like bugs in audit reports that continued to matter and affected the prosecution of subpostmasters were officially priorities, they just did a terrible job of handling them.
Even now, just by observing the use of systems at the Post Office, you can tell it's still a bad system. You can see the staff constantly mashing a button on their non-standard keyboard.
Readit News
Right now on YouTube there are Fujitsu staff being asked questions by the enquiry on this topic.
Worth also saying that some bugs were discovered in a product which was due to be replaced by a new system so it was considered low priority to fix. However it impacted cases of several years before it was discovered.
I think maybe that 2 Fujitsu employees are either investigation or being currently prosecuted by the met police, so there are criminal proceedings being under taken right now in relation (I guess) to these bugs in the older version of the code. There's not much info about this currently for obvious legal reasons.
Maybe at first they must have thought that these were one-off issues and that they could just prosecute one or two postmasters, sweep the whole thing under the rug and forget about it. But that doesn't explain how they continued prosecutions into the hundreds years down the road. Like what did they think the endgame here was? Just prosecute indefinitely to cover up for the system?
The Post Office was a money-hole; it hadn't made money since privatisation. It was effectively still a government agency.
UK government agencies have a long history with ICL, the UK "champion" mainframe maker, which was taken over by Fujitsu. It appears that this "champion" status was inherited by Fujitsu.
ICL couldn't compete with companies like IBM on merits; they had to have backdoor support from politicians and the civil service. I don't know how that worked; but if MI5, the armed services, the National Grid and so on were reliant on ICL, then it's understandable that government would want to encourage agencies to buy ICL.
So I suspect that the PO were pressurised by ministers and civil servants to buy Horizon from ICL/Fujitsu, and further pressurised to keep schtumm about the defects. Well, that's my guess: the higher-ups owed their jobs and reputations to ministers and civil servants, because they were effectively employed by a government agency, and it was losing money, and the government wanted to protect the reputation of ICL/Fujitsu.
Instead Ed Davey doubled down on the lying and persecuting ordinary postmasters, over an issue that until then he had no personal accountability for. This is what democracy is for, to provide the opportunity for a reset, to change course, and a mechanism for accountability of those who made the mistakes. Instead he allowed himself to be co-opted by the system.
I'm not defending the post office, but I suspect you have to look at it in context.
My understanding is that during that period of time (1999–2002) the Post Office's accounts took two large hits:
So perhaps the higher-ups regarded it as form of revenue management, claw-back of losses from the postmasters.https://www.telegraph.co.uk/news/2024/01/10/post-office-exec...
All in all its a lesson in just how easy it is to cover your arse as an executive in a large company, and get away with pretty much anything. Only the most tenacious work by many people over decades has any chance of holding you accountable.
Here is an anecdote from a past job.
1. Years ago (decades), someone made the contract rules. They did it in an ad hoc not very systematic way. They assumed everything would be judged by a human and be judged reasonably.
2. That person left and the department expanded.The rules had to become more formalized, but still room for plenty of exceptions. An exception could be a penciled scribble in the margins in a file or later on, a sticky note.
3. Time to digitize. Unfortunately, computers like standards and need every little rule must be programmed.
4. A business analyst with no real understanding of the system is hired to generate requirements for digitization. The current overseer of the rules, someone who doesn’t use a computer much (yes, these people still exist) kind of just nods every time he creates these tremendous flow charts. this is the first degradation of knowledge. Even then though, this business analyst is not technical or legally trained, so doesn't appreciate why defining things like "end of day" or "contract billing cycle" precisely is crucial. even if he could, many contracts had sticky note individual definitions.
5. The business analyst goes back and feeds the information into the ticket mill of Scrum. To be nice and agile, work is done in bite sized increments with no regard for context. This is the second knowledge degradation. All those flow charts are chopped into little boxes. Devs see ambiguities, but have no interest in pushing back.
6. Developers only have the tickets to work on and quotas to meet, so write the code as requested. Testing is another set of people, so code is just tossed over to QA. QA has no more context than dev, so if it matches the ticket, they approve it.
6a. Nobody budgeted for automated testing so things break over time for lack of tests. Product Owner wouldn't grasp the concept, business analyst had a budget and wanted to deliver more features, and devs didn't want to fight about it.
7. Final increment goes back to product owner who doesn't understand and just nods. A piece of software that doesn't work all that well gets approved. Repeat for years and constantly adding post-it note rules as code and you have a system that has lots of errors.
A sane system would be 'contract A type' and all other be done manually or something.
Er....shit in. Shit out?
[1] https://www.theguardian.com/uk-news/2024/jan/09/how-the-post...
That said, it was certainly shoddy
'Shoddy', being a euphemism for 'beyond knowingly-being fucked-up, so much so that we'll cover it up oh fuck we're still alive this wasnt what we were told would happen'?
I dunno, or maybe just me surmising.
Yes, you're probably right. But this is the part that really bugs me :
"The inquiry heard that in 2008, a glitch in a system called CABSProcess, which automatically summarises a post office’s transactions at around 7pm daily, resulted in users working at the same time having balancing issues."
This, for an accounting system is just fucking inexcusable!
I worked on a huge accounting project for a financial institution processing 2M account bookings per day (at that time, it's a lot more now). A lot of effort was put into the audit log (every booking was written to an append only log) and into the reconciliation process. When it was 20 Rappen off there was digging until the inconsistency was found.
That transactions are not completely isolated when that CABSProcess kicked in is just completely and uttelry inexcusable.
<<I wish I was joking>>>
Certainly there were bugs that made it seem like there was more money coming in than actually did, but it seems like also there were capabilities in the platform that would also allow editing records to make it seem like there was actually less money coming in, which would enable skimming money out of the system almost (?) invisibly.
I also don't think this capability was ever suggested anywhere as being used in conjunction with a postmaster to skim money out. Maybe its possible in theory?
Deleted Comment