"Every time I read about the Waterfall Model, I think to myself: I'm sure there must be companies out there that have tried this approach, but I have a hard time believing some software book would seriously propose it as the best way to write software"
I've been on software development projects (the final product would be source delivered to the customer) where the waterfall model was explicitly specified, with the whole design- and implementation phase based on this, with milestones where each step of the waterfall would be delivered and checked by the customer. This was particularly prevalent back in the eighties and beginning of the nineties. It's as real as can be, it did exist.
Obviously developers couldn't follow that 100%, so there would be some amount of back-designing an earlier stage, but if it wasn't 100% it was very close in practice too.
Agreed! There's waterfall & agile and dogmatic & pragmatic. I've never worked on a dogmatic waterfall project but have worked on several dogmatic agile projects (that made me strongly wish I wasn't).
In the pragmatic waterfall projects I worked on, during the Design phase there was still coding (by juniors, on submodules with obvious predictable designs as well as some refactoring (yes!)) and testing going on. It's just that in the Design phase it was understood any coding might need to change. In the Code/Build phase, design changes were still possible as issues were discovered (with maybe just a little more resistance). In short, we were very pragmatic about it. Much more so than with the Agile religion.
The Design phase focused on designing but not exclusively so. The Build phase focused on building but not exclusively so. Humans were building things long before Agile came along. The pyramids were probably built with a pragmatic waterfall method, not an agile method (pragmatic or dogmatic).
Hardware development is almost exclusively Pragmatic Waterfall. It's pretty much a requirement. Lots of reasons. Too many to list here (it would probably not be useful).
As a software dev for hardware companies, most of my life, I splashed in a lot of waterfalls.
One reason that waterfall method I mentioned actually managed to produce a working system was that the customer (together with all the companies working on the project) put extensive effort into the requirements phase. Working on the requirements and removing requirement conflicts (a very common thing), understanding of requirements, feasibility of requirements etc. etc., and, at the final stage, a week-long meeting with everyone (customer, companies) to work out the last kinks. Only after that did the next phase start: Creating software requirements from the system requirements, and from there, an architecture and so on. And, as this was something involving many companies in many countries and even continents, having everyone on the same page was essential. And it did work. Of course in the years following the project there would be bugs etc, but not fatal ones (which are typically caused by errors in the requirements- and architecture phases).
(We had several projects of the type above, and those which had more problems were those were the requirements phase wasn't as extensive)
EditAdd: Just to be clear - I don't think the pure waterfall model is the best way to work. However I did learn a lot from it, particularly how important requirements are. And the forth-and-back-and-forth-and-back on this which can happen in Agile projects isn't really the solution either.
Same here. Just 10 years ago I had the displeasure of working on such a project. Months and months of upfront specification work. The UX was defined in full separately from the UI. All the DB models built in one go. Every Gantt cascade one could imagine. Horizontal slice by horizontal slice. The system was to be fully integrated at the very end, followed by a few weeks of QA.
If anyone doubts that waterfall was the dominant approach to development and project management, just go back to the tools of the times, like Microsoft Project. Gantt charts were the unquestioned way you managed a project.
We use extreme waterfall. All the disadvantages of agile and waterfall in one process. Nothing gets to production due to meetings so the customers are happy we aren’t fucking anything up.
Of course there's a range of transitional states between waterfall and [whatever miracle cure is in vogue today]. Like all historical revisionists, today's techies like to impugn the old and laud the new. ('How could they have been so clueless?')
Fact is, long-lived products, esp. from BigCorps, and esp. those in heavily regulated spaces like medicine and transportation always have and always will spend a lot more time and effort on design and process than will ephemeral products and services from startups. My software career started ca. 1990 just as waterfall was evolving into the 'Spiral' iterative software model, which since has been rebadged and remorphed several times into today's 'agile' and 'test-driven' gift from god. However in the real world, makers of every S/W product inevitably must adapt their actual development process to suit the lifecycle needs of their product and customer, in ways that scrum can't serve. Often the best way to reduce technical debt is to increase investment design and testing. And taken to NASA-like limits of fault-tolerance and reliability, that can take you all the way back to waterfall.
So like a waterfall, but the day to day is connected by little bits of agile? You end up at the bottom of the waterfall, but by taking a bunch of much smaller falls instead?
Have you considered calling it Rapids development? You will sell a million books or whatever, just on the name.
This. Some products are platforms with thousands of functionalities all interlinked in a uniform data model, with similar but specific behavior expected for all modules of the platform.For instance, a Core Banking System, or OMS in Telcos. And since we all get paychecks, it seems to work.
Building such an product in agile was often tried and failed in the past decade. You need common data structures and routines that a few thousand developers can all expect to find ready further down the line. Like the Oregon Trail game, you launch all workstreams in different moments and expect them to converge at the finish line.
I think “100% waterfall” was ditched in the early 90s, for all the known reasons, with smaller cycles of releases becoming the norm. But still the hard thinking, laying down the key mechanisms of a platform was heavily thought out early on.
I think all criticism is a bit unfair, as agile is also an infinite source of cock ups, mediocrity and dead ends just as waterfall. There is a space for each approach, and bottom line, the right people will make all the difference. But that’s also why it’s important to learn and respect both approaches…
The whole strawman/dichotomy you see with Agile and Waterfall boils down to needing both flexibility and vision for large projects, and each fails at one of these.
How much do you think it was an artifact of the time? Agile relies on it being fast/cheap/easy to ship and deploy incremental changes. I have a hard time believing that to be the case in the 80s. When planning and designing on paper is cheap in comparison, it makes sense to me that is the development method people used.
> How much do you think it was an artifact of the time?
I was a software engineer in the 90s at a small company (15 people) doing simulation development for the UK Ministry of Defence. We had two distinct software development processes, which were waterfall and a form of rapid development known as the Dynamic Software Development Method (DSDM). (I threw away the manuals years ago so can’t give more info).
The company directors came from an aerospace background so had a very strong software quality focus. Many of our contracts specified waterfall and our project artefacts reflected that – user requirements docs, system requirements, architectural design, detailed design, integration test plans, system tests, acceptance tests, etc etc. Coding happened somewhere in the middle. It did work, sort of, in situations where the customers knew what they wanted, and we mutually understood it. Which was quite often the case, for our regular customers on projects that were extending existing systems.
We trialled DSDM for projects where the customer said ‘prototype’ and I liked it. It lacked the onerous agile / scrum boilerplate and IIRC was based around loosely specified timeboxes and relied on having ‘intelligent customers’ who were prepared to be flexible in terms of what they got at the end.
The need for DSDM was motivated by a project that failed badly (not with the MOD) because the customers said ‘here are our requirements for this prototype system, don’t come back until you have finished and tested it’. Needless to say the result wasn’t what they expected, even though all of the requirements had been, on paper, met.
But for any development where specific technical outcomes were necessary (e.g. a maths module), waterfall would usually be used.
I think it was more about the corporate culture which dictated that people had a plan and stick to it.
It's still the case in many trades. You don't build a house or a bridge the Agile way for instance.
IMHO Agile got a lot more prevalent because the planning phase became more and more onerous to have any reliability, and management was tired of paying upfront for it. I steady they could get a more regular billing (in particular when dealing with consulting companies) with reports of stuff allegedly shipped, instead of six months of "we're planning very hard, just trust us". Whether the end result matched what they needed/wanted to build being another story.
I know many companies that had an official Waterfall cycle, while actually working on something more granularly shipped with many phases of prototyping. And they didn't move to Agile either, just stopped calling their process anything and went on doing whatever felt right.
I think it was an artifact of applying physical construction project concepts to building software.
If you're building a house, or something bigger, you don't just start nailing lumber together. You have a blueprint. Pretty detailed, and it uses industry-standard terms and markup. Any builder can take it and use it to construct the building. It has the dimensions and specifications for the foundation and all the rooms. It has considered where the rooms are, which walls are load bearing, where the staircases and HVAC and plumbing and electrical runs are, where the windows and doors are, pretty much everything. An engineer has signed off on the overall design as being safe and meeting all the applicable codes. You can order a list of supplies based on this plan and you'll be within a few percent of what you actually use.
Some small changes were allowed. You could change finish details like what style of cabinets or trim or door knobs would be used. But you could not really approach the builder once the framing was complete and say actually I wanted the living room over there and the bedrooms over here and also I want to add a third storey. You also didn't build one bedroom, and then based on that experience extrapolate the design for the next one, and then apply that to building the kitchen, and so on. It was all done up front.
People, especially people who were used to managing physical projects, thought that with the same level of planning, and the same approach to management, the same results could be achieved in software projects.
At that time the waterfall model method was one solution to the problem of software quality. Other methods hadn't yet been developed (or at least not become well known yet), and waterfall was in use by the industry (not everywhere though - re sibling comment). The customer I mentioned was huge and lots of companies in lots of European countries, and overseas too had development contracts which demanded this process to be used. And the results were actually good, even though the process was heavy. That didn't mean that it couldn't be improved on, and that did happen later, with more agile-like (though not "Agile") methods modifying the approach.
I find agile passive aggressive because the consensus is built under duress. I prefer a well reasoned approach. I can understand agile process in between waterfall milestones.
I can even support an agile process to create waterfall process.
I think waterfall and waterfall-ish development is more prevalent in government contracting. You aren't building for yourself, you're building for a single external customer, and you have to build both the functionality and the interfaces that they want.
And I note that government contracting was probably a bigger fraction of all software engineering in the 1980s than it is today. (Still more in the 1960s.)
So, yes, it was real. It is mercifully less prevalent than it was.
In those days, having in-house competence to do the coding was very rare. With each step being much more connected to a specific area of competence. Hollywoods image of programmers was that they must not be seen anywhere near a customer.
This led to silos, where design was done by one group of people in the company, then sent to a third party contractor for a fixed scope and price.
This naturally creates a waterfall and when everybody just sees their own silo from inside, nobody questioned the big picture, nor that their own design would be flawed from the beginning. Until the product came back, not at all working as desired.
The biggest improvement today is that teams have much higher autonomy within their fully vertically integrated responsibilities. With wider areas of competence covered within a single team, or even a single person. As such, iterating and focusing on what is most important become more natural.
> with milestones where each step of the waterfall would be delivered and checked by the customer
I think this is one of the things that don't explicitly follow the waterfall diagram as presented and criticized though. You're delivering smaller units of work here and I presume incorporating their feed back at least to the extent of if it is functionally meeting the specified requirements which is kind of agile-y.
The main difference between the two at that point seems to come down mostly to the willingness to revisit the requirements which even when my employer was more waterfall we would try to incorporate changes if the customer came back and said what we were doing wasn't going to work at all or they needed changes.
Nope, in waterfall, "deliverables" might mean things like "UI design" (as a slide deck) or "database schemas" (on paper, without data) or "this microservice" (in isolation, and which you cannot run because dependencies and consumers are not implemented yet)
If you are producing anything a customer might actually before the very end of the project, it's not waterfall.
Indeed. Every real-world practice of development can be placed somewhere on the sliding-scale between Waterfall and Agile.
One of the reasons that waterfall was a reasonable way to proceed was that, in the past, tighter coupling to hardware and longer hardware lead times meant the need to design and specify in advance. This is still somewhat the case where these conditions are true, albeit to a lesser degree.
Having worked waterfall and now agile, I miss waterfall. Agile feels too fast and loose. We work faster but we also redo a lot of things. It's an illusion of progress.
it works the same if you apply it to other disciplines. The saying, "a plan is useless, but planning is invaluable". Agile just skips the second part by trying to throw out the first part.
I think a real criticism of Agile would be that it works great assuming a) domain expertise and b) good understanding of how accurate / precise a solution has to be to work.
Agile seems to work best when the domain itself is technology, rather than, say, when the domain is HVAC control or real-time ECMs.
What you describe is from micro-managent of the development process. I have never met anyone qualified to micro-manage a software project, and never will.
I too worked on waterfall dev projects. It was a nightmare and did not ship. The Agile Manifesto was a breath of fresh air, just because it captured what a lot of people were thinking in a way that was coherent enough to present to managers as a viable alternative.
On scrum projects I've been on, its essentially the waterfall model. Business comes up with a product and date, product and engineer give high level estimates. Program manager creates a project with all the stakeholders, product managers then write requirements, engineers then write designs. Finally this gets translated into jira epics/stories so engineers can do the implementation. The implementation runs in sprints so engineers can feel like they are doing agile. Finally when its all done, there's manual end to end testing with product and engineering.
> I've been on software development projects (the final product would be source delivered to the customer) where the waterfall model was explicitly specified
On the other hand, in my 50 years or so in this business, working with every size of company, I have never actually seen "waterfall" being done in practice. I'm sure it may have happened -- as you report -- but I don't think it was ever common.
> On the other hand, in my 50 years or so in this business, working with every size of company, I have never actually seen "waterfall" being done in practice. I'm sure it may have happened -- as you report -- but I don't think it was ever common
In my ~20 years of mostly startups and small businesses I have seen waterfall creep in at almost every company. You start by being small-a agile because there is no process. Then something happens – a few big fires ship, or someone misses a requirement, or an exec gets grumpy about feeling a loss of control, or internal users start treating it as just-a-job and want official training for every little thing, or you get engineers who just want to code – and your small-a agile process turns into waterfall.
Ah but we’re a startup, we can’t do waterfall! So you do a series of tiny waterfalls and call them Agile. Everything takes eons of planning, the management class feels super busy, the engineers become disempowered and widget factory like, and all the plans are wrong by the time you implement.
You start getting lots of “we’ll figure out the details then engineers just need to implement, it’ll be easier that way”. It is not easier that way.
Neither have I. Most of my software career has been spent in medical device development: a tightly regulated industry, and waterfall is just a concept.
Sure, you do requirements elicitation, review and documentation, and then architectural design, followed by detailed design & coding, etc. But it's never a linear process end-to-end. There's always feedback. There's always something you didn't account for showing up and requiring a change to an earlier part of the process.
Waterfall can, and does work, but only on fairly small projects that can be completely defined correctly the first time around. And like everything else, it's easier if you've done that kind of project before.
If you consider this not as a binary choice, but "waterfall" on one end meaning that the whole design process has basically no feedback, and "agile manifesto agile" (not scrum or any other calcification of the process, but the true chaotic "just depend on your engineers to be smart" process), it's probably true there has never been a "true", all-the-way-to-one-end Waterfall design that went for multiple years, because such a thing is essentially impossible.
It is absolutely the case though that many projects were driven with management methodologies way, way too far to the Waterfall side, and that managers would attack developers for essentially failing the process if anybody ever had to incorporate feedback or backtrack or do anything other than 100% rigidly and correctly follow the previous plan.
I don't consider it a "strawman" so much as a "simplification". Nobody could ever truly do waterfall for extended periods of time but there were plenty of people who acted and managed on the idea that the ideal is very, very close to it and to the extent deviations were necessary they are all automatically failures. To the extent that one is trying to discuss management methodologies across an entire industry, if not arguably across multiple industries simultaneously, waterfall is not an unreasonable approximation.
And personally I think in the occasional interminable "oh wouldn't it be wonderful if programmers were real engineers like architects" threads, where people fail to understand those processes used by other engineering disciplines are contingent based on the nature of their work rather than abstract Platonic ideals all should strive for, and those other disciplines would love to work with continuous integration servers and automated testing and strongly-typed components, there are a lot of developers that believe even today that if we just tried hard enough, waterfall could not only work but be the optimal design methodology, and it is we who are failing waterfall rather than the other way around.
I have certainly seen projects that started by planning everything out. High level designs, milestones, integration dates. Giant GANNT charts. But by the second quarter everyone was pretty much just redrawing the whole map, getting what they could running as early as possible and throwing things off the bus.
Yeah, I think the Waterfall model is what naive non-experts want from a project. I say what I want, you build it, it's delivered on time and on budget, and we're done.
But I think it's basically a fairy tale. "And they lived happily ever after" for executives. And I can prove it. There are basically no successful products that started with an executive idea, were created in a waterfall fashion, and then stayed at that initial version forever. Waterfall is built on the fantasy of managerial omniscience, where the people in charge know everything important on day 1. But the reality is that the beginning of a project is when we know the least. Waterfall in practice is a bet nobody will learn anything as time goes on. And on waterfall projects, that often turns out to be true, if only because course corrections would cause executives to lose face.
I joined a company that had adopted "agile" (in reality scrum + 2 week sprints) after a dramatic waterfall failure. They had attempted to rewrite a good, working, profitable product and had spent 2 years on the designs, only to throw it away because they just couldn't get it completed.
The requirements and design phases can/did often work very well. It's what comes after that derails the process.
When you spend more time trying to update your design to match the implementation (via change requests) than implementing you start to wonder if there's maybe a better way..
2+ decades ago, US Air Force acquisition classes taught exactly this method. I don't recall whether they used the word "waterfall", but they showed the same diagrams and treated software development like massive hardware projects, such as building an F-16: first you write a long requirements document, then generate a voluminous design document, then break the design into a work breakdown structure, farm out component implementation to developers, then finally integrate, test, and ship. It borrowed heavily from the field of Systems Engineering.
In my experience it never worked. We were able to achieve success only by secretly ignoring the rules and process and instead developing software in what detractors called a "cowboy" process. When "agile" came out, it changed our lives. We adopted a few new good ideas, but mostly kept working as we had been. But now we could point to this quasi-official "agile process", with tailoring as they recommend. As long as we were following a process, and not being "cowboys", folks seemed satisfied.
These days the Air Force has caught on to "agile". They even have 400-page manuals on the process to follow to be "agile"! We still cut some corners here and there and do what we think makes sense, but there's far less process push-back than there was in the past.
Acquisitions didn't use Waterfall that I ever saw, until you got to the software specific courses. Then they did, and it was one of several software development lifecycles you were allowed to choose between for projects.
Many program offices and project managers selected it because it was pretty, not because it was sensible.
> These days the Air Force has caught on to "agile". They even have 400-page manuals on the process to follow to be "agile"! We still cut some corners here and there and do what we think makes sense, but there's far less process push-back than there was in the past.
Circa 2010 (+/- a couple years) USAF also picked up Lean in a big way. But their version of Lean was almost exactly the opposite, in practice, of what was done and described by industry as Lean. A key element of Lean being that you actually let the people doing the work provide feedback and direct improvement efforts. USAF's version had the managers (who never did the work) walk around and observe, then change processes to be "leaner". That is, pure scientific management in the Taylorism style.
USAF is a great place for good ideas to get their names reused for old ideas.
Here’s a 2019 presentation about Lockheed Martin’s F-16 team adopting SAFe (Scaled Agile Framework). It was a successful first step because the dev team was able to deliver Program Increments to QA in just nine months, down from 3-4 years.
I'd argue that designing an aircraft still fails badly with waterfall and it is a major reason that aviation is massively behind where it should be right now. Space-x shows that you can build rockets, a lot better, if you ditch that waterfall process.
Agreed, waterfall fails right off the bat at "gather requirements."
If you ask the Air Force how fast a new proposed plane has to fly, they'll say "as fast as possible." Or maybe they'll say "fast enough to accomplish the mission and win wars." But neither of those would be accepted as requirements.
So in reality the Air Force will answer that question with another question: "well how fast can you make it fly?" Which of course depends on a lot of tradeoffs of performance, munitions, and cost. So in reality lots of design and technology tradeoffs go back and forth during the requirements phase, and at some point someone makes a rather arbitrary requirement like "The plane must be able to fly at Mach 2.5." That gets set in stone and drives the program forever after.
So the very notion of "requirements" is complete fiction. Perhaps the plane could have a top speed of Mach 2.2 and still achieve the mission, or go Mach 2.8 and fail to achieve the mission, depending on other aspects of the design.
I worked at a place where, while it was never called "waterfall", it was absolutely the way things worked. I think it naturally evolves this way:
1) a software project, when finished, turns out not to fulfill all of the things people wanted
2) "we should plan more carefully next time, so that all the requirements are identified at the beginning"
3) the next software project starts to resemble waterfall, but still misses some requirements
4) "we should spend more time on the requirements phase next time, so we don't miss anything"
5) etc.
It is the natural method that large organizations end up with, in the absence of any philosophy saying otherwise. It's not the natural method that startups or small organizations end up with, probably, because it involves a ton of meetings first, before anything is done, and that is something that happens more in large organizations.
Waterfall is what you get when it is always the safer option to say, "wait, let's spend more time planning out and getting buy-in". There may not be many places that call it "waterfall", but there are a lot of places that do it. Some of them even call it "agile".
The way you can tell you're in one of these dysfunctional companies in my experiences is at the moment you, as a software developer, have to attend TWO meetings in one day. Companies that have too many meetings tend to have middle managers with only one goal for all these meetings: "Covering their ass." They treat these meetings as a way to collect evidence to be used later to prove that they weren't at fault when the project fails to meet it's objectives.
I'd say more than 5 meetings a week for an IC rather than two in a day. (I have 3 meetings a week, but two of them are coincidentally on the same day). Managers will naturally have more meetings.
When people talk about Waterfall Model they forget the context of the era it originated. I've worked on a waterfall project early in my career. Please keep in mind that back in the 90s most software companies didn't have CI/CD, there was no cloud, releases were hand-delivered to the customer and usually rolled out about every 6 months because technically agile wasn't possible yet for most software shops. Waterfall was a valid method back then due to the technical (and cultural) limitations.
Haven't lived back then I'm not sure today's CD is better. Too often web breaks features people rely on. Worse they just do it randomly with no notice and no ability to have a transition time while you update. I miss the stability of knowing things will work like they did for a long time and when they break it is at a time I choose to apply the upgrade. (it didn't always work out this way, random bugs were a thing and still are)
Quality control was also a lot more important - you couldn't ship something with many bugs as customers would be forced to look at competitors if they hit your bug.
Meanwhile, the Royce paper that takes down the simplistic series of steps arranged in a waterfall, is a sort of formalization of an approach better visualized in late 80s as "the spiral model":
The concept of both, well before the 90s, is you don't know what you don't know, so its faster and more successful and costs less to make, learn, and document from prototypes before you know, to establish what you're really making.
I feel like this post has some nuance or clever wordplay that I'm missing. So if that's the case, apologies for adding noise.
But in the real world, the waterfall model was absolutely not a "strawman", but instead was literally how almost all software was built, up to even the turn of the century and beyond. Software projects were smaller in general, compiling was a major undertaken, you collaborated on projects by sending files around or, best case, working on a shared network directory. The pace of change of tooling and "libraries" was much slower and more predictable.
As we got source control, intellisense, massive knowledge bases, and then the rapid iteration of code sharing in the form of libraries and so on, things started changing because the model was too slow. But it was absolutely real and the norm.
Not exactly. From first principles, it could only possibly work if the project was very well-known/trodden ground already. So of course mistakes were either fixed when discovered or the project failed.
(Yes, some govt projects pretended to follow a strict process. What’s new?)
But even Brooks’ original MMM (1975) encouraged iteration as a way to validate assumptions.
By the second time, you actually have a reasonable chance of knowing what the requirements should have been the first time. They might even be valid for the rewrite, if things aren't moving too fast.
That's how I did waterfall. We would go through all the requirements, group things that were related, and assign them to a developer who owned them. Repeat until there are no more unsatisfied requirements.
I dive into this in my blog, "What is your alternative to Agile?" - https://www.ebiester.com/agile/2023/04/22/what-agile-alterna... - but the Structured Systems Analysis and Design Method as well as PRINCE2 were common in the industry, but both were examples of methodologies that took waterfall and codified it. However, the last waterfall project I was on was in 2013. We had formal gates out of each section of the process. (Even there, we tried to make it as incremental as possible, but it was a Fortune 500 company that had an overarching process.)
You also have to remember that there are a lot of software projects that do have a "done" state, after which the project goes into maintenance until there is more funding for improvements. Consider an internal project to turn a department excel sheet into an application. You can quickly reverse engineer the current behavior, go to the stakeholders and get a set of the desired behaviors above and beyond what the spreadsheet does, negotiate a scope, write a small design document, and just go do it. You then demo it at the end and change what is necessary before they start using it. You have a small set of bug fixes and QOL improvements that follow, then the project is handed off to a team managing 500 similar projects overseas and is responsible for security and bug fixes.
This doesn't make sense in product companies for good reason. However, on small projects, waterfall can work.
The Royce paper is actually still worth reading. It's well written and if you can step over the 50+ years of history since then also still somewhat informative. And of course the word waterfall doesn't actually appear in it. That word was slapped on it by others afterwards. And of course Royce vaguely suggests that going through the steps more than once might be helpful. What we now call iterating.
The key concept for what we now call agile (which mostly boils down to variations of scrum these days) actually has roots dating back about half a century. People were suggestion spiral development at some point (converging on some final thing) as early as the eighties. There was the whole 4+1 model which got caught up in the whole UML movement and it's associated process.
The key innovation that extreme programming (one of original agile methodologies) brought was that increasing the iteration frequency and giving up on the notion that you can do a lot of requirements and design up front. Things change. Every iteration. So having a lot of requirements and design documentation to update every cycle is counter productive. Dropping that was considered extreme. But then people realized that that worked just fine and that these designs were kind of low value artifacts to begin with. Not really worth the hassle.
With Spiral, the iterations explicitly involve the customer. This is important.
With Waterfall, there is an upfront spec, but implementing the whole thing without consultation with the customer can be a bad experience.
With Agile, maybe there's a lot of internal correction. But the key thing is to involve the customer in integration testing. Unless both sides are very experienced, surprises due to miscommunication are the rule.
It's mainly iteration length that has shortened. Royce was saying do it twice (including requirements engineering, which presumably would involve some kind of feedback). Spiral development increased that to doing it multiple times. Rational unified suggested a quarterly pace. Most agile methodologies work with sprints of a few weeks.
And lately, continuous deployment and Kanban like methods remove iterations completely and release continuously. Ship when ready and develop asynchronously. You can still have planning cycles with this of course but they are decoupled from release cycles.
The Linux kernel is a good example where you either make the merge window or wait for the next one. Large companies like Meta are also know to work with scheduled releases that are independent from planning cycles (e.g. weekly) of teams.
Readit News
I've been on software development projects (the final product would be source delivered to the customer) where the waterfall model was explicitly specified, with the whole design- and implementation phase based on this, with milestones where each step of the waterfall would be delivered and checked by the customer. This was particularly prevalent back in the eighties and beginning of the nineties. It's as real as can be, it did exist. Obviously developers couldn't follow that 100%, so there would be some amount of back-designing an earlier stage, but if it wasn't 100% it was very close in practice too.
In the pragmatic waterfall projects I worked on, during the Design phase there was still coding (by juniors, on submodules with obvious predictable designs as well as some refactoring (yes!)) and testing going on. It's just that in the Design phase it was understood any coding might need to change. In the Code/Build phase, design changes were still possible as issues were discovered (with maybe just a little more resistance). In short, we were very pragmatic about it. Much more so than with the Agile religion.
The Design phase focused on designing but not exclusively so. The Build phase focused on building but not exclusively so. Humans were building things long before Agile came along. The pyramids were probably built with a pragmatic waterfall method, not an agile method (pragmatic or dogmatic).
As a software dev for hardware companies, most of my life, I splashed in a lot of waterfalls.
(We had several projects of the type above, and those which had more problems were those were the requirements phase wasn't as extensive)
EditAdd: Just to be clear - I don't think the pure waterfall model is the best way to work. However I did learn a lot from it, particularly how important requirements are. And the forth-and-back-and-forth-and-back on this which can happen in Agile projects isn't really the solution either.
Naturally nothing worked and everything was late.
https://upload.wikimedia.org/wikipedia/en/6/65/Screengrab_-_...
Waterfall is great for high level governance (such as specifying high level milestones). Agile is great for executing on the deliverables.
It is jus my experience. You're miles may vary.
Fact is, long-lived products, esp. from BigCorps, and esp. those in heavily regulated spaces like medicine and transportation always have and always will spend a lot more time and effort on design and process than will ephemeral products and services from startups. My software career started ca. 1990 just as waterfall was evolving into the 'Spiral' iterative software model, which since has been rebadged and remorphed several times into today's 'agile' and 'test-driven' gift from god. However in the real world, makers of every S/W product inevitably must adapt their actual development process to suit the lifecycle needs of their product and customer, in ways that scrum can't serve. Often the best way to reduce technical debt is to increase investment design and testing. And taken to NASA-like limits of fault-tolerance and reliability, that can take you all the way back to waterfall.
Have you considered calling it Rapids development? You will sell a million books or whatever, just on the name.
Building such an product in agile was often tried and failed in the past decade. You need common data structures and routines that a few thousand developers can all expect to find ready further down the line. Like the Oregon Trail game, you launch all workstreams in different moments and expect them to converge at the finish line.
I think “100% waterfall” was ditched in the early 90s, for all the known reasons, with smaller cycles of releases becoming the norm. But still the hard thinking, laying down the key mechanisms of a platform was heavily thought out early on.
I think all criticism is a bit unfair, as agile is also an infinite source of cock ups, mediocrity and dead ends just as waterfall. There is a space for each approach, and bottom line, the right people will make all the difference. But that’s also why it’s important to learn and respect both approaches…
I was a software engineer in the 90s at a small company (15 people) doing simulation development for the UK Ministry of Defence. We had two distinct software development processes, which were waterfall and a form of rapid development known as the Dynamic Software Development Method (DSDM). (I threw away the manuals years ago so can’t give more info).
The company directors came from an aerospace background so had a very strong software quality focus. Many of our contracts specified waterfall and our project artefacts reflected that – user requirements docs, system requirements, architectural design, detailed design, integration test plans, system tests, acceptance tests, etc etc. Coding happened somewhere in the middle. It did work, sort of, in situations where the customers knew what they wanted, and we mutually understood it. Which was quite often the case, for our regular customers on projects that were extending existing systems.
We trialled DSDM for projects where the customer said ‘prototype’ and I liked it. It lacked the onerous agile / scrum boilerplate and IIRC was based around loosely specified timeboxes and relied on having ‘intelligent customers’ who were prepared to be flexible in terms of what they got at the end.
The need for DSDM was motivated by a project that failed badly (not with the MOD) because the customers said ‘here are our requirements for this prototype system, don’t come back until you have finished and tested it’. Needless to say the result wasn’t what they expected, even though all of the requirements had been, on paper, met.
But for any development where specific technical outcomes were necessary (e.g. a maths module), waterfall would usually be used.
It's still the case in many trades. You don't build a house or a bridge the Agile way for instance.
IMHO Agile got a lot more prevalent because the planning phase became more and more onerous to have any reliability, and management was tired of paying upfront for it. I steady they could get a more regular billing (in particular when dealing with consulting companies) with reports of stuff allegedly shipped, instead of six months of "we're planning very hard, just trust us". Whether the end result matched what they needed/wanted to build being another story.
I know many companies that had an official Waterfall cycle, while actually working on something more granularly shipped with many phases of prototyping. And they didn't move to Agile either, just stopped calling their process anything and went on doing whatever felt right.
If you're building a house, or something bigger, you don't just start nailing lumber together. You have a blueprint. Pretty detailed, and it uses industry-standard terms and markup. Any builder can take it and use it to construct the building. It has the dimensions and specifications for the foundation and all the rooms. It has considered where the rooms are, which walls are load bearing, where the staircases and HVAC and plumbing and electrical runs are, where the windows and doors are, pretty much everything. An engineer has signed off on the overall design as being safe and meeting all the applicable codes. You can order a list of supplies based on this plan and you'll be within a few percent of what you actually use.
Some small changes were allowed. You could change finish details like what style of cabinets or trim or door knobs would be used. But you could not really approach the builder once the framing was complete and say actually I wanted the living room over there and the bedrooms over here and also I want to add a third storey. You also didn't build one bedroom, and then based on that experience extrapolate the design for the next one, and then apply that to building the kitchen, and so on. It was all done up front.
People, especially people who were used to managing physical projects, thought that with the same level of planning, and the same approach to management, the same results could be achieved in software projects.
I can even support an agile process to create waterfall process.
And I note that government contracting was probably a bigger fraction of all software engineering in the 1980s than it is today. (Still more in the 1960s.)
So, yes, it was real. It is mercifully less prevalent than it was.
This led to silos, where design was done by one group of people in the company, then sent to a third party contractor for a fixed scope and price.
This naturally creates a waterfall and when everybody just sees their own silo from inside, nobody questioned the big picture, nor that their own design would be flawed from the beginning. Until the product came back, not at all working as desired.
The biggest improvement today is that teams have much higher autonomy within their fully vertically integrated responsibilities. With wider areas of competence covered within a single team, or even a single person. As such, iterating and focusing on what is most important become more natural.
I think this is one of the things that don't explicitly follow the waterfall diagram as presented and criticized though. You're delivering smaller units of work here and I presume incorporating their feed back at least to the extent of if it is functionally meeting the specified requirements which is kind of agile-y.
The main difference between the two at that point seems to come down mostly to the willingness to revisit the requirements which even when my employer was more waterfall we would try to incorporate changes if the customer came back and said what we were doing wasn't going to work at all or they needed changes.
If you are producing anything a customer might actually before the very end of the project, it's not waterfall.
One of the reasons that waterfall was a reasonable way to proceed was that, in the past, tighter coupling to hardware and longer hardware lead times meant the need to design and specify in advance. This is still somewhat the case where these conditions are true, albeit to a lesser degree.
I think a real criticism of Agile would be that it works great assuming a) domain expertise and b) good understanding of how accurate / precise a solution has to be to work.
Agile seems to work best when the domain itself is technology, rather than, say, when the domain is HVAC control or real-time ECMs.
On the other hand, in my 50 years or so in this business, working with every size of company, I have never actually seen "waterfall" being done in practice. I'm sure it may have happened -- as you report -- but I don't think it was ever common.
In my ~20 years of mostly startups and small businesses I have seen waterfall creep in at almost every company. You start by being small-a agile because there is no process. Then something happens – a few big fires ship, or someone misses a requirement, or an exec gets grumpy about feeling a loss of control, or internal users start treating it as just-a-job and want official training for every little thing, or you get engineers who just want to code – and your small-a agile process turns into waterfall.
Ah but we’re a startup, we can’t do waterfall! So you do a series of tiny waterfalls and call them Agile. Everything takes eons of planning, the management class feels super busy, the engineers become disempowered and widget factory like, and all the plans are wrong by the time you implement.
You start getting lots of “we’ll figure out the details then engineers just need to implement, it’ll be easier that way”. It is not easier that way.
Sure, you do requirements elicitation, review and documentation, and then architectural design, followed by detailed design & coding, etc. But it's never a linear process end-to-end. There's always feedback. There's always something you didn't account for showing up and requiring a change to an earlier part of the process.
Waterfall can, and does work, but only on fairly small projects that can be completely defined correctly the first time around. And like everything else, it's easier if you've done that kind of project before.
It is absolutely the case though that many projects were driven with management methodologies way, way too far to the Waterfall side, and that managers would attack developers for essentially failing the process if anybody ever had to incorporate feedback or backtrack or do anything other than 100% rigidly and correctly follow the previous plan.
I don't consider it a "strawman" so much as a "simplification". Nobody could ever truly do waterfall for extended periods of time but there were plenty of people who acted and managed on the idea that the ideal is very, very close to it and to the extent deviations were necessary they are all automatically failures. To the extent that one is trying to discuss management methodologies across an entire industry, if not arguably across multiple industries simultaneously, waterfall is not an unreasonable approximation.
And personally I think in the occasional interminable "oh wouldn't it be wonderful if programmers were real engineers like architects" threads, where people fail to understand those processes used by other engineering disciplines are contingent based on the nature of their work rather than abstract Platonic ideals all should strive for, and those other disciplines would love to work with continuous integration servers and automated testing and strongly-typed components, there are a lot of developers that believe even today that if we just tried hard enough, waterfall could not only work but be the optimal design methodology, and it is we who are failing waterfall rather than the other way around.
But I think it's basically a fairy tale. "And they lived happily ever after" for executives. And I can prove it. There are basically no successful products that started with an executive idea, were created in a waterfall fashion, and then stayed at that initial version forever. Waterfall is built on the fantasy of managerial omniscience, where the people in charge know everything important on day 1. But the reality is that the beginning of a project is when we know the least. Waterfall in practice is a bet nobody will learn anything as time goes on. And on waterfall projects, that often turns out to be true, if only because course corrections would cause executives to lose face.
Still does, sadly.
There's often a clear goal in these cases, and there's no 'minimum valuable product' you can quickly iterate on.
When you spend more time trying to update your design to match the implementation (via change requests) than implementing you start to wonder if there's maybe a better way..
In my experience it never worked. We were able to achieve success only by secretly ignoring the rules and process and instead developing software in what detractors called a "cowboy" process. When "agile" came out, it changed our lives. We adopted a few new good ideas, but mostly kept working as we had been. But now we could point to this quasi-official "agile process", with tailoring as they recommend. As long as we were following a process, and not being "cowboys", folks seemed satisfied.
These days the Air Force has caught on to "agile". They even have 400-page manuals on the process to follow to be "agile"! We still cut some corners here and there and do what we think makes sense, but there's far less process push-back than there was in the past.
Many program offices and project managers selected it because it was pretty, not because it was sensible.
> These days the Air Force has caught on to "agile". They even have 400-page manuals on the process to follow to be "agile"! We still cut some corners here and there and do what we think makes sense, but there's far less process push-back than there was in the past.
Circa 2010 (+/- a couple years) USAF also picked up Lean in a big way. But their version of Lean was almost exactly the opposite, in practice, of what was done and described by industry as Lean. A key element of Lean being that you actually let the people doing the work provide feedback and direct improvement efforts. USAF's version had the managers (who never did the work) walk around and observe, then change processes to be "leaner". That is, pure scientific management in the Taylorism style.
USAF is a great place for good ideas to get their names reused for old ideas.
https://youtu.be/nvfYLZ52zX0
> Acquisitions didn't use Waterfall that I ever saw
I mean, the term Waterfall. It definitely had the structure of Waterfall.
If you ask the Air Force how fast a new proposed plane has to fly, they'll say "as fast as possible." Or maybe they'll say "fast enough to accomplish the mission and win wars." But neither of those would be accepted as requirements.
So in reality the Air Force will answer that question with another question: "well how fast can you make it fly?" Which of course depends on a lot of tradeoffs of performance, munitions, and cost. So in reality lots of design and technology tradeoffs go back and forth during the requirements phase, and at some point someone makes a rather arbitrary requirement like "The plane must be able to fly at Mach 2.5." That gets set in stone and drives the program forever after.
So the very notion of "requirements" is complete fiction. Perhaps the plane could have a top speed of Mach 2.2 and still achieve the mission, or go Mach 2.8 and fail to achieve the mission, depending on other aspects of the design.
1) a software project, when finished, turns out not to fulfill all of the things people wanted
2) "we should plan more carefully next time, so that all the requirements are identified at the beginning"
3) the next software project starts to resemble waterfall, but still misses some requirements
4) "we should spend more time on the requirements phase next time, so we don't miss anything"
5) etc.
It is the natural method that large organizations end up with, in the absence of any philosophy saying otherwise. It's not the natural method that startups or small organizations end up with, probably, because it involves a ton of meetings first, before anything is done, and that is something that happens more in large organizations.
Waterfall is what you get when it is always the safer option to say, "wait, let's spend more time planning out and getting buy-in". There may not be many places that call it "waterfall", but there are a lot of places that do it. Some of them even call it "agile".
Quality control was also a lot more important - you couldn't ship something with many bugs as customers would be forced to look at competitors if they hit your bug.
If you want to see a valid method "back in the 90s", see DSDM:
https://en.wikipedia.org/wiki/Dynamic_systems_development_me...
Meanwhile, the Royce paper that takes down the simplistic series of steps arranged in a waterfall, is a sort of formalization of an approach better visualized in late 80s as "the spiral model":
https://en.wikipedia.org/wiki/Spiral_model
The concept of both, well before the 90s, is you don't know what you don't know, so its faster and more successful and costs less to make, learn, and document from prototypes before you know, to establish what you're really making.
But in the real world, the waterfall model was absolutely not a "strawman", but instead was literally how almost all software was built, up to even the turn of the century and beyond. Software projects were smaller in general, compiling was a major undertaken, you collaborated on projects by sending files around or, best case, working on a shared network directory. The pace of change of tooling and "libraries" was much slower and more predictable.
As we got source control, intellisense, massive knowledge bases, and then the rapid iteration of code sharing in the form of libraries and so on, things started changing because the model was too slow. But it was absolutely real and the norm.
(Yes, some govt projects pretended to follow a strict process. What’s new?)
But even Brooks’ original MMM (1975) encouraged iteration as a way to validate assumptions.
https://pragtob.wordpress.com/2012/03/02/why-waterfall-was-a...
It is hard to foresee how the world of development would look like today, had companies used the waterfall process twice on each project.
[0] https://dl.acm.org/doi/10.5555/41765.41801
You also have to remember that there are a lot of software projects that do have a "done" state, after which the project goes into maintenance until there is more funding for improvements. Consider an internal project to turn a department excel sheet into an application. You can quickly reverse engineer the current behavior, go to the stakeholders and get a set of the desired behaviors above and beyond what the spreadsheet does, negotiate a scope, write a small design document, and just go do it. You then demo it at the end and change what is necessary before they start using it. You have a small set of bug fixes and QOL improvements that follow, then the project is handed off to a team managing 500 similar projects overseas and is responsible for security and bug fixes.
This doesn't make sense in product companies for good reason. However, on small projects, waterfall can work.
The key concept for what we now call agile (which mostly boils down to variations of scrum these days) actually has roots dating back about half a century. People were suggestion spiral development at some point (converging on some final thing) as early as the eighties. There was the whole 4+1 model which got caught up in the whole UML movement and it's associated process.
The key innovation that extreme programming (one of original agile methodologies) brought was that increasing the iteration frequency and giving up on the notion that you can do a lot of requirements and design up front. Things change. Every iteration. So having a lot of requirements and design documentation to update every cycle is counter productive. Dropping that was considered extreme. But then people realized that that worked just fine and that these designs were kind of low value artifacts to begin with. Not really worth the hassle.
With Waterfall, there is an upfront spec, but implementing the whole thing without consultation with the customer can be a bad experience.
With Agile, maybe there's a lot of internal correction. But the key thing is to involve the customer in integration testing. Unless both sides are very experienced, surprises due to miscommunication are the rule.
And lately, continuous deployment and Kanban like methods remove iterations completely and release continuously. Ship when ready and develop asynchronously. You can still have planning cycles with this of course but they are decoupled from release cycles.
The Linux kernel is a good example where you either make the merge window or wait for the next one. Large companies like Meta are also know to work with scheduled releases that are independent from planning cycles (e.g. weekly) of teams.