I emailed them. I live I'm SF. This is the sake, solved, problem we have in the machine tool world (milling machines, lathes, etc). Similar to old keyboards, you can just buy and use a floppy emulator.
They emailed me back, they said that the floppy thing makes a good headline but is really just the tip of the iceberg. It's really the whole system that's like this at every layer, it needs replacing they say.
This was my thought. Floppy emulators are a dime a dozen thanks to the retrocomputing community. Emulators for the custom boards full of discrete logic, opaque ROM chips, PLAs, and so on from long dead companies are going to be a much bigger challenge.
If they had good schematics for all of the parts it might be possible to keep the the system running for a long time with a couple of smart EEs who are comfortable with the scope and soldering iron, but eventually they're going to run out of some obscure part and be up a creek.
Or maybe they could replace entire boards with home designed versions condense all of the old logic down to one chip and a handful of support components and start in-place upgrading without a total system revamp. Still an expensive process, and one that requires some hard to find engineers on staff, but theoretically spreads out the upgrade process over many years. It also loses out on functional improvement opportunities while your system is made up of a hodgepodge of old and new hardware.
SelTrac was designed to use commercial off the shelf (COTS) parts so there shouldn't be much in terms of needing custom chips, etc. The problem is that Alcatel (now Thales) basically doesn't support any of this (and hasn't for a long time).
The computers in question are, I believe, just thin clients.
RE "....Still an expensive process, ....."
Actually probably NOT , if all the $$ saved over the last 25 years , and also considering the many millions to get a completely new replacement system....
A once a generation refresh seems okay in my book. The Breda trains that came in 1996 were being phased out about twenty years into their tenure, most generally seem to think that is about reasonable a time to do something like that. I don’t see software related regimes as inherently that different.
Since I experienced 5.25 inch floppy disk era...and even the occasional bernoulli disk...we could simply say: the system had its run, replacement is reasonable. A lot of stuff had changed since then, and not just in storage media.
That ancient code is the crystalization of the knowledge of how to run, at the most basic level, all of the MTA. Emulating hardware is a good way to extend it's life while the decade long task of replacing it is underway.
Until you have directly used "legacy" machine that are mission critical, and understand how even a tiny error could cause the failure of a business and all the jobs that go with it, you can't avoid underestimating the true scale and scope of the problem.
Here in the Washington DC area, our metro still has two series of rolling stock that date back to 1982 and 1987 (2k and 3k series). At the earliest they won't start retiring them in 2024 and 2025 at the earliest...
The real scope of the project is a mind-boggling $600 million to track the light rail trains inside and outside the tunnels and to control the trains inside the tunnels. Divided by the 250 light rail vehicles (apparently does not include historic streetcars), it works out to $2.6 million per car. Plus $36M for consulting (2023-11-07 board presentation from the documents that I linked earlier https://www.sfmta.com/sites/default/files/reports-and-docume...).
The PC and the floppy disk should be running as a virtual machine. ( The Vm software and host OS and the hardware would all be current supported versions of course )
That doesn't result in bigger budgets. If the goal of a manager is to increase their budget and hence scope of responsibility, any solution which doesn't result in a bigger budget isn't really a solution. Same reason it's going to cost a hundred million, yet a single indie game dev can write a train scheduling simulator by themselves.
the state-of-good-repair grift will continue until the morale improves! SelTrac is fine. and if it ever actually breaks (dubious) they can fall back to block signaling which comfortably handles more trains than muni is capable of running. the MBTA does just fine with it on the green line, and when the MBTA is making you look incompetent you've got to question where it all went wrong.
this is the agency that brought you the central fucking subway, a $300M sewer project masquerading as some red paint on van ness, that picked the already-mostly-grade-separated M Ocean over the N Judah to subway-ify and that is incapable of flicking the traffic preemption switch on the rest of the T to the on position without a decades long pedestrian detection LIDAR project for the unique in the world needs of san francisco.
It sounds like they let this problem fester until it has reached this existential end. They clamor for new technology, which they get, then they use the most risk adverse management strategy and never upgrade or change it, until it reaches this problem state.
They either need significant third party help deploying and managing this system, or they should go back a few generations of technology and use the simplest possible system that meets their needs. Pen and paper should be considered if it can be made more efficient.
I guess the people who work on public transit aren't interested in having the best public transit system available. They're only interested in keeping it running for as long as possible with zero changes or responsibility.
This is why I'm strongly doubtful on public transportation in the US. Our bureaucracy can't handle it.
> I guess the people who work on public transit aren't interested in having the best public transit system available. They're only interested in keeping it running for as long as possible with zero changes or responsibility.
Let me guess, you also think public servants are lazy, overpaid and unskilled? Comments like this belie how many people form opinions of public agencies without having any real experience with government.
The problem is the public and the politicians - transit employees largely do the best they can with the resources and constraints they are given. In order to succeed transit needs municipalities and states to: pay more taxes; accept that transit won't generate a profit, but does generate non-monetary public value; and gather their resolve to deal with NIMBYs.
The problem is "lower my taxes" single issue voters. People vote for whomever lowers taxes and then complain about potholes. It's always the same people. GOVERNMENT IS INCOMPETENT BLAH BLAH. Yeah, cause you refuse to fund it and everything goes to shit, then is 10x more expensive to fix it compared to properly funding maintenance. You end up paying more in the end because these people can't see past the couple hundred dollars they saved on their property taxes and wonder why everything is turning to shit.
It reads like you are conflating newer with better.
It was more than a few years ago, but I remember the posts on Slashdot where tech people were wringing their hands over the fact that election offices were not adopting touchscreen ballots fast enough. Some posters were also promoting things like internet voting. How did that work out?
Sometimes it actually is better to move slowly instead of adopting something new just because it is new. With old equipment and older technology, we know the failure modes, we have the procedures in place for addressing them. Whether it's floppy disks, paper ballots, (or human beings,) older does not necessarily mean worse.
or they're not immune to the same urge all SW devs have to rewrite - it's a lot easier to start from scratch than (a) figure out how an existing system works, (b) figure out how to upgrade or modernize
My company provides new compute modules for another major global city's infrastructure that still relies on Intel CPU's from the early 80's. That is, we are building them brand new boards populated with chips that are over 40 years old.
They haven't shown any interest in updating the system. It works, they can get service, and get "new" replacements for things that go bad.
What they might not know though is that there is basically just one engineer we have (and probably the only one on Earth) who knows how to work on these things. He's getting old, and obviously none of the younger engineers really have an interest in learning ancient forgotten systems.
LOL:) this story seems to resurface every couple of years. My favorite part of the article:
"Jeffrey Tumlin: "It's a question of risk. The system is currently working just fine but we know that with each increasing year risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure."
This seems to imply they have been using the exact same disk for the past 20 years (absurd), they have absolutely no idea what is written on the disk and how it can be safely backed up or restored. This would be a problem regardless of the medium used.
Although I hold the line at using paper tape there is nothing wrong with using floppies other than it seems antiquated. It certainly is reliable and cheap. Maybe the only thing that needs replacing is the people running the Muni.
Every sizable manufacturer of floppy disks has exited the market. There are no more (major) suppliers for the tech. They’ve likely been depending on a dwindling supply of functional disks; if at some point they find themselves without enough working disks to operate the system, they will indeed be screwed.
There’s also the chance that they take a disk that’s on the verge of failure, plug it into the system, and some corrupted commands get loaded into the system. That could easily result in a “catastrophic failure”.
Floppy disks are not reliable or cheap. They physically degrade over time, and at this point are nowhere near cheap for “new” disks.
You make a good point. However, even with the disks backed up to a better storage medium, detecting data loss on the floppy and getting a replacement written and deployed to the correct location might take a while. During that time the system may face unacceptable transit delays.
One of my older jobs relied on ancient hardware, and the life of the company was measured by parts in the warehouse and vendors who still cared. From their perspective they'd have a modern system as long as your company delivers.
> hey haven't shown any interest in updating the system.
So expensive to update that there's a calculated end-of-life to the system. They'd love to know about your engineer situation. That'd trigger plans put in place a while ago.
Your company could do a last big batch for the city and send the old guy out with a nice bonus.
Nah getting you fee engineer interested isn't that hard, there's always a fun challenge in mastering something. You just have to give them time to figure it out.
At my workplace, we still ship devices with AMD 8086 processors to the military. And AMD still makes us 8086 processors on special order.
We got a few guys under 40 that work on that project once in a blue moon when a change is required.
Where are you going to find a young engineer who wants to build a dead-end career? I'm sure that if you pay them enough, someone will show up, but most people would rather build skills that will continue to be useful rather than skills that will be useless once an inevitable, sorely needed refresh happens.
> This system was designed to last 20 to 25 years. SFMTA's director Jeffrey Tumlin said upgrading the system will take another decade and cost hundreds of millions of dollars.
This is the problem, not that they're using a floppy. This isn't web dev where you get to rewrite everything every 6 mos. Systems have to have decades long life cycles BUT THEY EVENTUALLY NEED TO BE REPLACED and that's not happening quickly enough here.
Edit: It was last updated in 1998, so it's due now not a decade from now.
I'd like to see a breakdown of that "hundreds of millions of dollars" and which companies provided the quotes, plus who owns the companies and who are their husbands/wives/uncles/nephews/nieces/etc.
I looked into this last year, a significant part of the cost is in new hardware, both on the vehicles, but also sensors spread through the system. They all need commercial grade RF systems that allow for 2 way signaling communication and integration into the vehicles control systems.
It’s probably not overt corruption. It’s probably because in this kind of large system, there is a strong demand for validation and assurance at all levels, which is usually achieved by hiring experts* to write reviews, recommendations, and reports that hold up in court.
* They must be experts, otherwise how could they get away with charging so much, right?
The cost is distribute across all the PowerPoint, Word, and Excel spreadsheets created to give the illusion that people are doing actual work. Go read a couple of the SFMTA PDF's linked to by a commenter. They are totally enlightening justifications like - reduced congestion blah - better monitoring blah. No meat whatsoever. No analysis backing anything up - just pretty pictures.
I rode Muni for 10 years almost every working day. Do you want to know what reduced congestion the most. Buying new muni cars because the old muni cars door mechanism had been repaired so much they did not open or close %50 of the time and people had to use other doors to get on and off a train on a daily basis.
Still, when you have a 20-25 year replacement cycle being 10 years behind is pretty significant. And that's assuming their estimate of needing another decade is correct. Which seems doubtful considering they don't have a contractor, they don't have the money, and the previous system probably took a decade to develop considering their choice of 5.25" floppies (3.5" floppies overtook 5.25" floppies in sales in 1988).
Right, if they think it will take 10 years, 15-20 is probably more realistic, which means you need to start the upgrade process 5-10 years after buying your system: seems very expensive.
What's really happening is going to be:
Emerging new use-cases and rising costs from specialized vendors for replacement parts that match the existing system are driving a desire for the agency to replace the control system. The floppy system bit can 100% be replaced with a solution that isn't a floppy, but that wouldn't help them with the rest of what they want.
>Turns out that in 1998, SFMTA had the latest cutting edge technology when they installed their automatic train control system.
> "We were the first agency in the U.S. to adopt this particular technology but it was from an era that computers didn't have a hard drive so you have to load the software from floppy disks on to the computer,"
In 1998, most personal computers already had hard drives [0]. From Wikipedia "The IBM PC/XT in 1983 included an internal 10 MB HDD, and soon thereafter, internal HDDs proliferated on personal computers."
The 3.5" floppy is from the mid 80's, again from Wiki [1] "In the early 1980s, many manufacturers introduced smaller floppy drives and media in various formats. A consortium of 21 companies eventually settled on a 3½-inch design..."
Why would a hard drive be better? If the floppy fails just grab another off the shelf and try it (surely they have more than one copy). Downtime measured in seconds.
The other good thing about the floppy is it can't hold very much code. So the system has a tight upper bound on how bloated and complex it can get. Simpler systems are more maintainable.
These things seem like great assets for maintaining critical infrastructure.
EDIT: Another great thing is such a system will be stateless. No disks, no filesystems, no databases. Sign me up.
I'm not sure moving it back a whole decade saves them?
My family's first PC in 1987 had a hard disk. The Wikipedia quote they provide lines up with that, and provides a more authoritative point of when it was introduced.
And yeah, the "5 inch floppy" quote paired with a photo of a diskette. God only knows what actual hardware the system uses. But point being … the journalist doesn't seem to have found out.
Why do I have to do this research instead of the "journalist"?
Ratchet the snark back. The journalist was referring to the train control system not home computers in someone's basement. And, yes, twenty five years ago SelTrac was cutting edge. Moving block systems were basically unheard of back then.
> And, yes, twenty five years ago SelTrac was cutting edge. Moving block systems were basically unheard of back then.
Meanwhile in Germany, we have had moving blocks from the late 80s, based on a technology developed from the mid-60's and production-ready by the 70s [1]. Incredibly, the LZB technology never had an actual accident happen in all the time, only three "bare misses" (one of which was pretty spectacular in that it caused a train to pass over a switch rated for 80 km/h with around 185 km/h without derailing).
The only snark here is yours. What are you talking about with "computers in someone's basement?"
He pointed out glaring factual errors in the story, which should not have made it through any kind of editorial review. For example: 25 years ago pretty much every computer had a hard drive. And the disk depicted in the article is obviously a 3.5", not a "five-inch floppy."
Given that the agency seems not to be able to cope with change at a reasonable pace, I wonder if 1998 was just when the control system project finished, having been planned and started several years prior.
I had the displeasure to operate a sort of industrial/automotive rack computer running very simple Simulink data logging that booted only from a floppy, and I think it was bought around 2008. For a greenfield project.
Probably cost like hell and was decade or two behind in technology, but I didn't buy it and definitely wouldn't have. Salespeople get people to do stupid stuff.
I'd guess these are used a lot in industrial settings where the code and task is actually very simple. Would run no problem with a $1 microcontroller and a hundred lines of C and with a lot less hassle. Likely vendors have gotten a reputation and keep on selling them with FUD.
Well... I feel slightly better about Boston constantly pushing back being able to use our phones as tickets after seeing the timeline for the transition and how just that they are still doing this.
What is it about public transit in the US that it is so... bad? Inadequate funding seems to be the easy one, but the MBTA (Boston) doesn't even handle the funds it has well. Yeah it needs more funding but there is also just a core issue to how it's run.
It is sad to see the state of public transit in this country, particularly in dense urban areas where we should be discouraging Car use as much as possible.
I am very curious what other countries are doing that we are not.
I am an elected official (12 years and elected six times in Illinois) unfortunately seeing this process work over 12 years is discouraging. For example President Obama allocated 8 billion dollars for high sped passenger rail service improvements. By 2017 almost 2 billion of this money had been spent on railroad improvements / railroad crossings and line inspections.
Fast forward to 2024 no high speed rails have actually been built. For example Chicago Illinois was to get a line from Iowa City to Quad Cities to Chicago (straight line). This project was fully funded at one point but Iowa and Illinois had (still have) a disagreement on bridge repairs connecting the states.
Now Governor Pritzer has authorized a new high speed rail commission to try and get this project going again.
In a nutshell .. Rail project between Iowa City and Chicago was fully funded as of 2016… all that actually happened is some rail road crossing were improved.. lines were inspected and no actual new lines constructed. Instead 1 out of the 2 billion was spend on “engineering” costs and compliance paperwork … which now has expired and need to be redone if the project is to be completed.
The amount of money spent on compliance paperwork and “engineering” is staggering. Many six figure salaries depend on slightly altering existing engineered projects to meet compliance requirements for projects that are never built. From waste treatment upgrades, water treatment, road improvement, traffic studies, on and on. The amount of money spent on services that are not finished or lead to an actual project is absolutely staggering … entire industries depend on this inefficiency and lobby effectively to keep things “obtuse”.
From someone who's seen the other side... every time the engineer has to quabble over email with the permitting departments over matters such as "you sent it to the wrong bureaucrat so I'm ignoring instead of forwarding it" or "the permit admin wrote the wrong 1 of 50 Comcast subsidiaries on the paperwork so now I have to get a skip-manager's signoff" it's billed at $400/hr.
I feel like Boston has had similar, we have contracts with companies to complete a thing by a specific time, they don't but we are stuck paying them even more to complete a project so it ends up being over budget, late, and possibly not done well as is evident by our green line extension that opened last year that was shut down to redo it because the tracks are the wrong width.
It just feels like the money is going to the wrong places like you said.
I would probably argue that we still need more funding even if we fixed how we used the money, but we need to fix how we use it first.
I think about this all the time every time I see an lcd screen in my local train stations. If people who designed them actually used transit, maybe they'd have the signs you can only see as you walk out of the station say something about the bus lines you'd transfer to, instead of when the train behind you that just departed from is set to arrive. At some stations the screens they installed don't display anything useful at all, not when the next train will come, just the date and time as if everyone doesn't have that in their pocket, and this needs to be displayed every 25 feet on the platform.
A lot of transit could be fixed by just taking a regular routine user, empowering them to become a dictator for a week and point out all the friction points they hit actually using the system. But then that would make the entire bureaucratic system that is the transit agency look like idiots who don't understand their own jobs, so it will unfortunately never be done.
For sure. I think a lot of this is just about how heavy the planning cycles are. A lot of stuff like that is developed through big, waterfall planning efforts and large outsourcing contracts. Most of the choices were made before they had their first user, and once it's all installed any revision would be a) expensive, and b) a black eye for the people in charge.
And in some ways I don't blame people for doing this, because you need really supportive stakeholders to work in an iterative fashion. Otherwise you get a ton of nitpicking that amounts to a lot of "why didn't you guess 100 perfectly everything up front" and "how can we start if you can't give me a full plan and a firm price?" In a blame-hungry environment, waterfall is the safest choice for the people doing the project.
"Which way do I go?" orientation signage in the Boston subway system has been less than wonderful. A lot of "the obvious thing needed here is ..." not happening. Places you could stand to help confused and grateful tourists. So tempting to do a self-adhesive-vinyl signage project. And then the DNC did.
When the Democratic National Convention came to Boston (2004), the subway got a lot of nice "which way do I go? what are these stairs? if not these, then where?" temporary orientation signage. It lingered afterward, with some becoming permanent signage, and others puzzlingly not. Such an event involves massive interdepartmental communication, tight timelines, altered incentives and constraints, and additional resources. Some of the altered communication channels are said to have persisted. I might be interesting to look at events where things are "shaken up", to better understand the steady-state tangle.
Japan's transit is private and semi-competitive. (there are 10+ train companies in Tokyo), (4+ in Kyoto), I haven't counted Osaka's
It works because the companies own land and facilities around every station. Grocery stores, office buildings, shopping centers (stores at many stations), apartments, etc. This creates a virtuous cycle where the more riders the more people use their other services and visa-versa.
As for semi-competitive, at least in Tokyo, Osaka, and Kyoto there are enough lines that you often have a choice. For example Tokyo (area) to Yokohama there's JR, Keikyu, Tokyu, (3 different companies). They have slightly different routes so if you're closer to one you might take one or the other but they do advertise trying to get you ride their's over the other's. To Hanada there's Keikyu and the Tokyo Monorail (it's own company). To Narita there's the Narita Express (JR) and Keisei (a different company) as well as local lines from both. Same in Kyoto. You can go to Kyoto to Nara via JR or via Keihan. You can go Kyoto to Osaka via JR or Hanshin.
So, if one company offers easy pay methods and another doesn't it quickly gets the reputation and "a crappy old line" (who wants to live there, open an office there, etc...)
I'd love to know what incentives make this stuff work in countries with pure "public" transportation. There the obvious "it's the right thing to do" but this American (me) sees that incentives work and I can't see what incentives keep politicians, which are human just like everyone else, from syphoning funds from public transportation to other things, not able to get a large enough budget to upgrade since they need all the other politicians in, not get influenced by industry trying to sell cars, etc..
- political point-scoring - blame-oriented cultures discourage experimentation and incremental improvement
- political polarization - one-party areas can more easily slide into cronyism, and fighting between parties makes it hard to compromise even on things like fixing infrastructure
- classism - in a lot of places, transit is for the poors
- racism - many don't want transit bringing Those People around
- manager culture, not engineer culture - as we see with Boeing, standard MBA thinking doesn't work well for long-term safety and reliability; the focus on short term metrics, mostly financial ones, leads to underinvestment and decay of infrastructure
I’m a liberal democrat, but it’s clear that one party politics is not tenable. There has to be an option to the status quo that you can vote for to incentivize competency in government. But with the state of national politics it would be almost impossible for a republican to win in most of urban America. So that leaves just primary challengers from within the Democratic Party which is very difficult. So urban political machines have become absolutely rotten with complacency and now the richest zip codes on the planet have trains running on floppy disks.
I struggle to understand how unions/government are often seen as the problem. Some of the best public transit I've experienced have been in places with strong unions and more government-owned/managed public transit e.g. much of Europe. Some of the worst in places where unions are weak and "public" transit is largely out-sourced to private entities operating on behalf of public transit agencies e.g. the US, UK, Australia, New Zealand, etc.
I live in New Zealand public transit in NZ used to be incredibly bad during the heights of private operation in the 1990's but has improved heaps over the last 10-20 years as local/central govt have progressively taken back control over more aspects of public transit. However public agencies are still not allowed to directly operate bus or train services which still sees private companies failing to provide good service with no consquences and public agencies unable to take over to provide the service improvements that are needed. It is interesting to compare New Zealand in the 1950's had one of the highest levels of public transit usage per capita under public ownership of buses and railways. This dropped down to embrassingly low levels after out-sourcing to private entities to a low in the 1990's.
It has only been under government leadership that we have seen a revival to the current levels (e.g. doubling to 100+ million trips per annum in our biggest city from well below 50 million in the mid-1990's) due to local government coordinating public transit systems (and contracting private entities to operate to the timetables set by public authorities) that meets the needs of the users and not just private entities making a profit. Unions have been vocal too for improving systems not only for transit employees but also recognising that what's best for transit users also benefit transit employees.
YMMV and possibly depends on the political environment in each country.
> What is it about public transit in the US that it is so... bad?
Massive money and entrenched behind preserving a car-based life.
Car companies, auto workers unions, railroads can siphon off funds from promised improvements to cover deferred maintenance and give themselves bonuses for being so clever… Hell, AAA which one might expect to spend its income on serving its members instead diverts money to proactively lobbying against improvements in public transit in an ongoing display of cynical self-preservation
Nonsense. There is working transit in the US and a huge appetite for more. But it takes decades to do this stuff. "China did massive infrastructure project in two years" -- China is also a communist dictatorship (for some value of those words), and we can't do that here. We could streamline things a bit more, of course, but it's never going to be cheap and easy in the US.
If you want to help, vote. Or even run for something on a pro-transit, YIMBY platform.
When BART was built in the 60s/70s it was innovative. The construction techniques for the Transbay Tube were unique at the time. The system was billed as a transit system built from the ground up for the space age. It was equipped with, at the time, a state of the art Automation Train Control system.
Upgrading those systems is expensive and difficult, so it's not surprising old prices of equipment hung around until the end of their useful life. Especially since BART is going through a massive budget crisis at the moment (and this isn't the only one in its history).
I've shared this experience/opinion with folks nearly verbatim :) Especially after having seen what I took to be a very well-integrated transit system in Chicago, the fragmented and confusing Bay Area experience was quite different from what I expected.
SF Voters rejected Proposition A in 2022 [1], which would have included funding to upgrade Muni's control systems (among many other projects). We'll eventually have to find the money somewhere else when the system fails.
The entire system has reached the end of it's design life and the floppy is just the shocking bite to get people to click through. Really this is the regular replacement of an old system with a newer control system that will make the trains run better and more reliably.
Why will a new system make the trains run better? It could also make it run worse, if they don’t cover all the current cases.
I’ve ridden Muni in SF for years, drivers control most of the decision making.
There’s a handful of single track tunnels/sensors/necessary software-based coordination, but as another commenter pointed out, the doors cause more issues than signal problems.
Readit News
They emailed me back, they said that the floppy thing makes a good headline but is really just the tip of the iceberg. It's really the whole system that's like this at every layer, it needs replacing they say.
If they had good schematics for all of the parts it might be possible to keep the the system running for a long time with a couple of smart EEs who are comfortable with the scope and soldering iron, but eventually they're going to run out of some obscure part and be up a creek.
Or maybe they could replace entire boards with home designed versions condense all of the old logic down to one chip and a handful of support components and start in-place upgrading without a total system revamp. Still an expensive process, and one that requires some hard to find engineers on staff, but theoretically spreads out the upgrade process over many years. It also loses out on functional improvement opportunities while your system is made up of a hodgepodge of old and new hardware.
The computers in question are, I believe, just thin clients.
Since I experienced 5.25 inch floppy disk era...and even the occasional bernoulli disk...we could simply say: the system had its run, replacement is reasonable. A lot of stuff had changed since then, and not just in storage media.
Until you have directly used "legacy" machine that are mission critical, and understand how even a tiny error could cause the failure of a business and all the jobs that go with it, you can't avoid underestimating the true scale and scope of the problem.
Dead Comment
It may feel emotionally like this at every layer, but the layers that are not floppy discs are completely different from floppy discs.
Other than electrolytics caps (easy replacement items), old electronics is reliable.
Moving parts need service; that's life.
> needs replacement
Not believable without detailed justification.
In the present article, someone is literally quoted as everything working just fine.
Good luck getting the city to see it that way. If you'd like help, let me know.
this is the agency that brought you the central fucking subway, a $300M sewer project masquerading as some red paint on van ness, that picked the already-mostly-grade-separated M Ocean over the N Judah to subway-ify and that is incapable of flicking the traffic preemption switch on the rest of the T to the on position without a decades long pedestrian detection LIDAR project for the unique in the world needs of san francisco.
Deleted Comment
(they probably are…)
https://www.businessinsider.com/nyc-mta-subway-delay-2017-6
It sounds like they let this problem fester until it has reached this existential end. They clamor for new technology, which they get, then they use the most risk adverse management strategy and never upgrade or change it, until it reaches this problem state.
They either need significant third party help deploying and managing this system, or they should go back a few generations of technology and use the simplest possible system that meets their needs. Pen and paper should be considered if it can be made more efficient.
I guess the people who work on public transit aren't interested in having the best public transit system available. They're only interested in keeping it running for as long as possible with zero changes or responsibility.
This is why I'm strongly doubtful on public transportation in the US. Our bureaucracy can't handle it.
Let me guess, you also think public servants are lazy, overpaid and unskilled? Comments like this belie how many people form opinions of public agencies without having any real experience with government.
The problem is the public and the politicians - transit employees largely do the best they can with the resources and constraints they are given. In order to succeed transit needs municipalities and states to: pay more taxes; accept that transit won't generate a profit, but does generate non-monetary public value; and gather their resolve to deal with NIMBYs.
It was more than a few years ago, but I remember the posts on Slashdot where tech people were wringing their hands over the fact that election offices were not adopting touchscreen ballots fast enough. Some posters were also promoting things like internet voting. How did that work out?
Sometimes it actually is better to move slowly instead of adopting something new just because it is new. With old equipment and older technology, we know the failure modes, we have the procedures in place for addressing them. Whether it's floppy disks, paper ballots, (or human beings,) older does not necessarily mean worse.
They haven't shown any interest in updating the system. It works, they can get service, and get "new" replacements for things that go bad.
What they might not know though is that there is basically just one engineer we have (and probably the only one on Earth) who knows how to work on these things. He's getting old, and obviously none of the younger engineers really have an interest in learning ancient forgotten systems.
"Jeffrey Tumlin: "It's a question of risk. The system is currently working just fine but we know that with each increasing year risk of data degradation on the floppy disks increases and that at some point there will be a catastrophic failure."
This seems to imply they have been using the exact same disk for the past 20 years (absurd), they have absolutely no idea what is written on the disk and how it can be safely backed up or restored. This would be a problem regardless of the medium used.
Although I hold the line at using paper tape there is nothing wrong with using floppies other than it seems antiquated. It certainly is reliable and cheap. Maybe the only thing that needs replacing is the people running the Muni.
There’s also the chance that they take a disk that’s on the verge of failure, plug it into the system, and some corrupted commands get loaded into the system. That could easily result in a “catastrophic failure”.
Floppy disks are not reliable or cheap. They physically degrade over time, and at this point are nowhere near cheap for “new” disks.
> hey haven't shown any interest in updating the system.
So expensive to update that there's a calculated end-of-life to the system. They'd love to know about your engineer situation. That'd trigger plans put in place a while ago.
Your company could do a last big batch for the city and send the old guy out with a nice bonus.
At my workplace, we still ship devices with AMD 8086 processors to the military. And AMD still makes us 8086 processors on special order.
We got a few guys under 40 that work on that project once in a blue moon when a change is required.
Dead Comment
This is the problem, not that they're using a floppy. This isn't web dev where you get to rewrite everything every 6 mos. Systems have to have decades long life cycles BUT THEY EVENTUALLY NEED TO BE REPLACED and that's not happening quickly enough here.
Edit: It was last updated in 1998, so it's due now not a decade from now.
* They must be experts, otherwise how could they get away with charging so much, right?
I rode Muni for 10 years almost every working day. Do you want to know what reduced congestion the most. Buying new muni cars because the old muni cars door mechanism had been repaired so much they did not open or close %50 of the time and people had to use other doors to get on and off a train on a daily basis.
Dead Comment
What's really happening is going to be: Emerging new use-cases and rising costs from specialized vendors for replacement parts that match the existing system are driving a desire for the agency to replace the control system. The floppy system bit can 100% be replaced with a solution that isn't a floppy, but that wouldn't help them with the rest of what they want.
"because life cycles" is a lazy description.
> "We were the first agency in the U.S. to adopt this particular technology but it was from an era that computers didn't have a hard drive so you have to load the software from floppy disks on to the computer,"
In 1998, most personal computers already had hard drives [0]. From Wikipedia "The IBM PC/XT in 1983 included an internal 10 MB HDD, and soon thereafter, internal HDDs proliferated on personal computers."
The 3.5" floppy is from the mid 80's, again from Wiki [1] "In the early 1980s, many manufacturers introduced smaller floppy drives and media in various formats. A consortium of 21 companies eventually settled on a 3½-inch design..."
[0] https://en.wikipedia.org/wiki/Hard_disk_drive
[1] https://en.wikipedia.org/wiki/Floppy_disk
Why do I have to do this research instead of the "journalist"?
The other good thing about the floppy is it can't hold very much code. So the system has a tight upper bound on how bloated and complex it can get. Simpler systems are more maintainable.
These things seem like great assets for maintaining critical infrastructure.
EDIT: Another great thing is such a system will be stateless. No disks, no filesystems, no databases. Sign me up.
(These days, you could also replace the floppy with a USB drive: they make adapters/emulators.)
Edit: Even the news article hints at 5 inch floppy disks...which makes 1998 make absolutely no sense to me at least.
https://abc7news.com/san-francisco-train-system-has-been-run....
My family's first PC in 1987 had a hard disk. The Wikipedia quote they provide lines up with that, and provides a more authoritative point of when it was introduced.
And yeah, the "5 inch floppy" quote paired with a photo of a diskette. God only knows what actual hardware the system uses. But point being … the journalist doesn't seem to have found out.
Meanwhile in Germany, we have had moving blocks from the late 80s, based on a technology developed from the mid-60's and production-ready by the 70s [1]. Incredibly, the LZB technology never had an actual accident happen in all the time, only three "bare misses" (one of which was pretty spectacular in that it caused a train to pass over a switch rated for 80 km/h with around 185 km/h without derailing).
[1] https://de.wikipedia.org/wiki/Linienf%C3%B6rmige_Zugbeeinflu...
He pointed out glaring factual errors in the story, which should not have made it through any kind of editorial review. For example: 25 years ago pretty much every computer had a hard drive. And the disk depicted in the article is obviously a 3.5", not a "five-inch floppy."
Probably this. Probably they got the plans worked up in 1990, but didn't finish everything until 1998.
Probably cost like hell and was decade or two behind in technology, but I didn't buy it and definitely wouldn't have. Salespeople get people to do stupid stuff.
I'd guess these are used a lot in industrial settings where the code and task is actually very simple. Would run no problem with a $1 microcontroller and a hundred lines of C and with a lot less hassle. Likely vendors have gotten a reputation and keep on selling them with FUD.
What is it about public transit in the US that it is so... bad? Inadequate funding seems to be the easy one, but the MBTA (Boston) doesn't even handle the funds it has well. Yeah it needs more funding but there is also just a core issue to how it's run.
It is sad to see the state of public transit in this country, particularly in dense urban areas where we should be discouraging Car use as much as possible.
I am very curious what other countries are doing that we are not.
Now Governor Pritzer has authorized a new high speed rail commission to try and get this project going again.
In a nutshell .. Rail project between Iowa City and Chicago was fully funded as of 2016… all that actually happened is some rail road crossing were improved.. lines were inspected and no actual new lines constructed. Instead 1 out of the 2 billion was spend on “engineering” costs and compliance paperwork … which now has expired and need to be redone if the project is to be completed.
The amount of money spent on compliance paperwork and “engineering” is staggering. Many six figure salaries depend on slightly altering existing engineered projects to meet compliance requirements for projects that are never built. From waste treatment upgrades, water treatment, road improvement, traffic studies, on and on. The amount of money spent on services that are not finished or lead to an actual project is absolutely staggering … entire industries depend on this inefficiency and lobby effectively to keep things “obtuse”.
It just feels like the money is going to the wrong places like you said.
I would probably argue that we still need more funding even if we fixed how we used the money, but we need to fix how we use it first.
A lot of transit could be fixed by just taking a regular routine user, empowering them to become a dictator for a week and point out all the friction points they hit actually using the system. But then that would make the entire bureaucratic system that is the transit agency look like idiots who don't understand their own jobs, so it will unfortunately never be done.
And in some ways I don't blame people for doing this, because you need really supportive stakeholders to work in an iterative fashion. Otherwise you get a ton of nitpicking that amounts to a lot of "why didn't you guess 100 perfectly everything up front" and "how can we start if you can't give me a full plan and a firm price?" In a blame-hungry environment, waterfall is the safest choice for the people doing the project.
When the Democratic National Convention came to Boston (2004), the subway got a lot of nice "which way do I go? what are these stairs? if not these, then where?" temporary orientation signage. It lingered afterward, with some becoming permanent signage, and others puzzlingly not. Such an event involves massive interdepartmental communication, tight timelines, altered incentives and constraints, and additional resources. Some of the altered communication channels are said to have persisted. I might be interesting to look at events where things are "shaken up", to better understand the steady-state tangle.
It is now a fairly regular occurrence to have a train show up that the screen had no reference too existing a minute or two before.
But yeah it feels like they were not designed or setup by people that actually use the trains and were setup by a comity thinking they know best.
Maybe that is the big difference, in other countries the people using it are also the ones managing it?
It works because the companies own land and facilities around every station. Grocery stores, office buildings, shopping centers (stores at many stations), apartments, etc. This creates a virtuous cycle where the more riders the more people use their other services and visa-versa.
As for semi-competitive, at least in Tokyo, Osaka, and Kyoto there are enough lines that you often have a choice. For example Tokyo (area) to Yokohama there's JR, Keikyu, Tokyu, (3 different companies). They have slightly different routes so if you're closer to one you might take one or the other but they do advertise trying to get you ride their's over the other's. To Hanada there's Keikyu and the Tokyo Monorail (it's own company). To Narita there's the Narita Express (JR) and Keisei (a different company) as well as local lines from both. Same in Kyoto. You can go to Kyoto to Nara via JR or via Keihan. You can go Kyoto to Osaka via JR or Hanshin.
So, if one company offers easy pay methods and another doesn't it quickly gets the reputation and "a crappy old line" (who wants to live there, open an office there, etc...)
- chronic underfunding - we all know how problems build up when maintenance is deferred
- waterfall planning - Mary Poppendieck has a nice talk on how much trouble this causes: https://www.infoq.com/presentations/tyranny-of-plan/
- political point-scoring - blame-oriented cultures discourage experimentation and incremental improvement
- political polarization - one-party areas can more easily slide into cronyism, and fighting between parties makes it hard to compromise even on things like fixing infrastructure
- classism - in a lot of places, transit is for the poors
- racism - many don't want transit bringing Those People around
- manager culture, not engineer culture - as we see with Boeing, standard MBA thinking doesn't work well for long-term safety and reliability; the focus on short term metrics, mostly financial ones, leads to underinvestment and decay of infrastructure
One option is ranked choice voting.
Deleted Comment
other cities (in the east, with the best public transit) don't let unions grab the entire city by the balls
government employees in other cities aren't as hellbent on extracting their pound of flesh from the taxpayer
I live in New Zealand public transit in NZ used to be incredibly bad during the heights of private operation in the 1990's but has improved heaps over the last 10-20 years as local/central govt have progressively taken back control over more aspects of public transit. However public agencies are still not allowed to directly operate bus or train services which still sees private companies failing to provide good service with no consquences and public agencies unable to take over to provide the service improvements that are needed. It is interesting to compare New Zealand in the 1950's had one of the highest levels of public transit usage per capita under public ownership of buses and railways. This dropped down to embrassingly low levels after out-sourcing to private entities to a low in the 1990's.
It has only been under government leadership that we have seen a revival to the current levels (e.g. doubling to 100+ million trips per annum in our biggest city from well below 50 million in the mid-1990's) due to local government coordinating public transit systems (and contracting private entities to operate to the timetables set by public authorities) that meets the needs of the users and not just private entities making a profit. Unions have been vocal too for improving systems not only for transit employees but also recognising that what's best for transit users also benefit transit employees.
YMMV and possibly depends on the political environment in each country.
Massive money and entrenched behind preserving a car-based life.
Car companies, auto workers unions, railroads can siphon off funds from promised improvements to cover deferred maintenance and give themselves bonuses for being so clever… Hell, AAA which one might expect to spend its income on serving its members instead diverts money to proactively lobbying against improvements in public transit in an ongoing display of cynical self-preservation
It’s basically hopeless.
If you want to help, vote. Or even run for something on a pro-transit, YIMBY platform.
Dead Comment
I was expecting it to be some kind of utopia, with futuristic technology on every corner.
In reality, it is roughly the same as anywhere else in America. A bit of a let down. The innovation does not take place in the infrastructure.
Beautiful place, though!
Upgrading those systems is expensive and difficult, so it's not surprising old prices of equipment hung around until the end of their useful life. Especially since BART is going through a massive budget crisis at the moment (and this isn't the only one in its history).
*: Many people don't. Self-driving cars are better than human-driven cars, but affordable and reliable trains are even better.
[1] https://www.sfchronicle.com/sf/article/S-F-voters-narrowly-r...
Sounds like they are using the floppy as an excuse to push for an upgrade that has nothing to do with the floppy drives.
I’ve ridden Muni in SF for years, drivers control most of the decision making.
There’s a handful of single track tunnels/sensors/necessary software-based coordination, but as another commenter pointed out, the doors cause more issues than signal problems.