The fundamental problem with Agile is the fundamental problem with all product development:
The customer doesn't know what they want
Agile assumes as a first principal that including the customer throughout the development process will align the building team and the customer to come to the same conclusion.
This is almost never actually true. Having built and managed a lot of products I can state fairly confidently that the amount of time the customer has, and clarity about the problem they want solved is orders of magnitude more ambiguous than is required to actually build and deliver something to their satisfaction.
In my opinion successful products, and I mean ones with significant flywheels with customers and viral growth and stickiness are with few exceptions - accidents. It was someone who had an interesting thought and a lot of people for whom the product was good enough for their desire that they often didn't even know they had
You can't write down how to do that, it's like asking a nobel prize winner how they came up with their discovery. It's not repeatable. Some people have better intuitions than others and these are the people who are repeatably successful with products. But it doesn't generalize.
That said, the fundamental problem isn't wrong in theory, it's just naiive in practice.
The agile manifesto was the problem. It's a vague as hell set of proscriptions that everybody could project their own ideas on to that got turned into a pseudo-religion.
I once had it used against me to justify not writing tests ("processes and tools!"). We had meetings about bugs instead. Seriously. Complain all you like about perversion, that was a perfectly valid interpretation coz those hallowed commandments are, frankly, ill-defined bollocks.
The principles are better but still suffer the same problem. If the most efficient way to convey information in a dev team is face to face why don't you come over here and I'll whisper my pull request in your ear. Jesus.
There's no point complaining about the perversion of a thing that was never very specific about what it wanted to be in the first place.
Once people (especially managers) get specific about what agile means to them it turns out it means very different things to different people. This is entirely the fault of the originators.
One benefit of Scrum (the catholic church to agile's christian sect) was that it was proscriptive and was specific. Unfortunately it's also a bit shit and its practitioner-priests LOVE to tell you that if it isnt working, well, You Probably Just Weren't Doing It Correctly.
I think the way the Manifesto and early community around it lent itself to this inversion is this:
(1) The Manifesto is written in non-actionable terms.
(2) Many of the same people that initially attached themselves to the manifesto were already pitching canned processes, that may have been the outcome of agile process where they originated but did not put the critical feedback mechanisms more prominently than the low-level processes. So, those became the actionable versions of “Agile”.
It would be so much better if instead of the “X over Y” language, the Manifesto has spent a few words talking about how the less important things were subordinated to the more important ones.
The weird thing is that continuous, bottom-up, empirically-based process improvement wasn't an unknown concept at the time the Manifesto was written. It wouldn't have been hard for it to reference more concrete, actionable, concepts and practices.
There’s a fascinating talk by Dave Thomas, one of the authors of the agile manifesto, speaking of the simple meaning of the manifesto and the insanity of the Agile Industrial Complex bureaucratic cult created around it.
The Agile Manifesto never came out and said so, but it only ever made any sense if you give up the idea of a fixed delivery date. Of course, the idea of a fixed delivery date for a software project never made sense in the first place, but you’ll have to pry that nonsense from their cold dead hands. Big-A “Agile” is an attempt to keep what they (think they) want… so it ends up being useless.
This reeks of “no true scotsman” to me. They claim is the manifesto, but the author even makes the claim that his process allowed engineers “a month of time without management intervention (a sprint length of 4 weeks). The author being one of the original signers and creator of scrum.
If that’s not following a plan over responding to change, i don’t know what is.
The problem is most agile followers take
"Responding to change over following a plan" to mean
Don't plan.
So you have 100+ person organizations running in a direction quickly for 2 week sprints to find out what the new direction they are going to run once they get there.
Problem is that most developers think they're in product manufacturing rather than product design. If only the requirements were clear, we could build it right in one go.
Agile tries to turn software development into a product design process rather than a product manufacturing process. Together with the client, iteratively you try to find the right solution and design for the customer problem.
By being easy to adopt, software lends itself to an iterative design process, where you adopt it multiple times based on the iterative feedback of the customer. But this means that at the end of the design process, you can skip the manufacturing part, as you've already built the software during design.
Alternatively you could argue that this is too expensive, inefficient and slow, and you need some design before you begin to manufacture software. Modeling tools could help here, e.g, low code tooling like CAD tools in fysical product design.
> Problem is that most developers think they're in product manufacturing rather than product design. If only the requirements were clear, we could build it right in one go.
Oh, man. I think you hit the head of the nail. One succinct phrase that describes the entire problem.
I don't think agile was meant to take everything client says without critical thinking. Quite opposite actually, it encourages dialogue. Often developers know better how to solve certain problems.
Early in my career somebody said something like "make sure you distinguish between <<what they (customers) want>> and <<what they need>>.". This thought stuck with me for years and in my opinion has been reposible for success I witnessed.
It's important to unwrap "what they want" so you, as developer, can apply more first principle thinking to arrive at "what they need".
BAs have tendency to project their own solutions, however when unwrapped so you see the problem that needs solving, you may arrive at solution that is simpler (like orders of magnitue is not that rare in my experience), reflects reality better, takes advantage of somehow hidden from BAs PoV system internals etc.
Agile enouraged dialogue is the platform where those ideas can surface.
I thought agile hedged against that by getting product in front of customers early - they may not know what they want but if you show them 100 things they don't want you'll probably be close
That is the theory, except on the customer side you actually need the people that will use the product.
Usually what happens is that you get a customer team, that is supposed to voice how the organization wants the software to be.
So the same mismatch is bound to happen anyway.
And when they do involve the people from the field, depending on the company culture, they might even be quite positive on the demos (cause being negative isn't good) and then completely find it unusable on final delivery.
Agile misses that many engineering teams lack people skills to actually navigate and avoid these scenarios.
We kind of successfully to that in my shop, the early stage product usually being graphic mock-ups (wireframes may not be enough when the client really lacks product vision) to iterate fast.
I disagree mildly all succesfull products are pure accidents.
Especially in B2B and technical realm it is feasible to actually plan in every sense of the word "plan" a new product - business, tech, etc. Of course the plan must be malleable on the facts on ground. And probably is incorrect on a few places. But never the less, a plan that will get a business from place a (no product, but strong understanding of customer needs and business plus strong technical savvy) to place b (final product, customers, revenue etc).
I've seen this many times, both in startups with a new product and new customers (automotive software) and established companies that just expand their existing portfolio (cad software).
I'd say the "not an accident" is feasible if these facts are in place:
- enough A-team players (tech,business,sales)
- healthy culture (that facilitates long term growth and can focus on the product without too much politics)
- understanding of customer business
- identified an OBVIOUS need for the customer
- strong engineering org (can be just a few coders that get things done or larger)
So I agree, you can't train for these requirements like you can train airplane pilots.
But I do claim you can have enough understanding that is the current org in possession of these qualities. And if it is, it's a very reasonable risky investment to try to create a product for the need.
The vast majority of software is not for flywheel type situations.
For internal customers or in B2B settings getting the actual users and outcome owners involved early is vital - but they are often poorly incentivized to get into the action as there are layers of business mangers etc in place that are supposed to stand in (but lack the detailed knowledge).
"Customers don't know what they want": this is something Agile specifically tries to address by releasing often and adapting to change. This is a quick feedback loop: the customer can quickly see a product and the development can quickly adapt.
What was the 'standard' way to develop systems when the Agile Manifesto was written: the same as for building a bridge.
You gather and freeze requirements. You write a massive plan and plenty of specs. You code accordingly. You deliver.
Time to working software is long and a lot of the effort that goes into specs is wasted, especially for design specs.
Your characterization is not true. The reality was quite the opposite.
Here's the introduction of Steve McConnell's "Rapid Development" from 1994 - well before the Agile Manifesto! - Chapter 1, page 1:
> The product manager told me he wanted to build a product right for a change. He wanted to pay attention to quality, prevent feature creep, control the schedule, and have a predictable ship date.
> When the time came to actually do the project, it became clear tha getting the product to market quickly was the only real priority. Usability? We don't have time. Performance? It can wait. Maintainability? Next project. Testing? Our users want the product now. Just get it out the door.
> This particular product manager wasn't the manager on just one product. He could have been almost any product manager I've worked for. This pattern is repeated day after day, state by state, all across the country. ...
This is as far from building a bridge as you can get. The next page of the book starts:
> [1.1] What is Rapid Development?
> To some people, rapid development consists of the application of a single pet tool or method. To the hacker, rapid development is coding for 36 hours at a stretch. To the information engineer, it's RAD - a combination of CASE tools, intensive user involvement, and tight timeboxes. To the vertical-market programmer, it's rapid prototyping using the latest version of Microsoft Visual Basic or Delphi. To the manager desperate to shorten a schedule, it's whatever practice was highlighted in the most recent issue of Business Week.
Agile drew from earlier methods ("intensive user involvement", "rapid prototyping") which were pretty common even before Agile.
Isn't Agile more or less gradient descent (well, approximating the gradient for an undifferentiable function), applied to software development?
The objective function is some function of the quality of the product, as measured by the customer, and the work put in. The gradient evaluation is then the process of determining which feature gives the most bang for the buck, implementing it, and repeating.
But gradient descent has problems with local minima. So if that's what Agile is, no surprise it doesn't do surprising projects - even if the customer does know what they want. If the programmers are too inexperienced and it's not just a CRUD, then you'll end up with something like Jeffries' TDD Sudoku solver.
> Isn’t Agile more or less gradient descent (well, approximating the gradient for an undifferentiable function), applied to software development?
Lean is.
Agile is “It would be nice to do what works for your team in your particular challenges, not some pre-packaged one-size-fits-all processes sold by consultants. Now, here’s a bunch of pre-packaged one-size-fits-all processes, and an army of consultants selling them.”
You’re right that customers don’t know what they want and that can make it a bad idea to include them in the dev process directly. But is that what Agile is about? I thought it was about iterative development. As a long time product dev I know that you want to follow your own vision when building your product. You let your vision be influenced by customer feedback for sure, but you should always second guess the feedback, never take it at face value, never just build something because a customer asks for it - that would be Henry Ford trying to breed faster horses.
It's probably better to phrase it being a development method that focuses on a tight feedback loop with all stakeholders.
You push decisions as far out as possible and gather feedback from individuals every step of the way.
This is what allows the greatest amount of flexibility.
The biggest problem with agile is that it assumes all problems can be put off to the last minute.
I'm frequently finding myself in situations where it is 100x better to hammock program it out for a few weeks before pushing any code. Those systems last ten years. Those systems are able to be modified simply and quickly when business needs change. Those systems do build up legacy code, but it's at a slower rate than other systems. Those systems have fewer hacks over time and fewer bugs as a result.
This isn't a dig at agile, it's a dig at using agile as a design process, for which it is utterly inept. Agile is great at pushing products, terrible at pushing thoughtful system design. (On average, sometimes working a problem works too but more often than not you don't get the chance to redo middle sprint 5.)
> The fundamental problem with Agile is the fundamental problem with all product development: The customer doesn't know what they want
I don't disagree, though I firmly believe Agile does a much better job of dealing with the problem you state than any other software development methodology that has been attempted before.
Is it perfect? of course not. As a methodology, is it better for building software than methodologies that have been used to build enormous physical structures for generations? Undoubtedly yes.
As far as I'm aware, there currently isn't anything better for building complex software with changing requirements (i.e. the real world).
> Agile assumes as a first principal that including the customer throughout the development process will align the building team and the customer to come to the same conclusion.
The only thing that is less likely to have the building team and customer come to the same conclusion is to add many more layers in between.
If that were true the customer never drop you for the competition. The customer will drop you in a heartbeat.
The customer cannot express what they want in a language you can articulate which is something amazingly different. As a product manager you need to account for that.
I'm not sure if Agile ever claimed to be a good process for coming up with unicorn startups though.
It's a way to meeting actual business software needs, much more suited to custom development for one customer than product development, which necessarily requires some additional insight into what the market wants.
Develop product engineers to be better. Have empathy and communicate with the target customer more clearly and understand the landscape better.
That's pretty much all you can do. How you organize it only matters insofar as it makes your team effective - sprints, waterfall, XP whatever... is kind of a toss up and context dependent.
Agile is a generic umbrella term that involves a vast array of complex, subtle knowledge and skills. It's like saying "Engineer", or "Chef". Each has a shared skillset, to be sure. But each category's members can't just work the same way at all jobs, and there's no book on how to be an Engineer or Chef everywhere.
The Agile Manifesto is a failure at trying to make Agile happen because it can't tell you how to make it happen, because it varies wildly. No manager can read a book on how to "Agile-ify" their org, they have to apply their brains and figure out how their specific version of "Agile" will work. But the skill-set required to do this is not a Managerial skill, it is a lower-level-worker skill. But it's also a very advanced lower-level-worker skill.
And that's why things like "Agile", "DevOps", etc will fail. People at the higher end have no clue how to make it happen, and people at the lower end who have an idea how to make it happen don't have the power to make the organizational changes to do so. You need a way for the lower-end people to tell the higher-end people what to do, and have the higher-end people listen to them, and make the changes happen. This is very hard in a traditional organizational hierarchy, because higher-end people have big egos and bigger concerns over things like politics.
The product and engineering managers I've had in the past have generally been humble, acting as a touchstone of coordination for the folks on the ground rather than a controlling force. And the engineering managers have all just been former engineers who decided they preferred doing this kind of facilitation work instead of writing code.
I do enterprise consulting for a couple of years, the kind of organisations that buy Oracle or MSDN licenses without thinking twice about it.
The best I have seen were mini-silos that managed to de-couple themselves from the org chart.
However I also have seen that it hardly lasts more than a couple of years, because as soon as someone noticed the success of the business unit they were re-integrated so that they could teach the reasons for their success to others.
You can imagine how well did they usually fare afterwards.
> And that's why things like "Agile", "DevOps", etc will fail.
I completely disagree. Most team leads and technical managers where I've worked get promoted up from within a highly technical position. I wouldn't have any respect for my team lead or my PM if they didn't know what they were talking about.
If my PM is going to try to tell me that I should work on this feature over this other feature or I should implement it in this way over this other way do you really think I'm going to listen to them if it's clear they have no idea what the implementation details are let alone the tools? Of course not.
I'm making DevOps happen and I do that by knowing the tools and practices. I'm being given the power I need to make it happen. I earn the respect every day I come in and write the code or identify the weaknesses we need to shore up to move faster.
The old Microsoft (the new one is much better at this), you couldn't just go around sharing department knowledge, if you on lets say Visual Basic team wanted the deep knowledge for a Windows feature as means to product improvement, it was easier to have a neutral contractor somehow get hold of the information across departments, than asking directly as Microsoft employee.
I'm not an expert on this subject, but I feel like "agile" could be summarized as "a process for actively incorporating feedback early and often".
That doesn't mean you need points or tshirt sizes or constant status meetings. You just need channels of communication with the right people and a chance to adapt to what they say.
And it’s something that good devs will already be trying to accomplish anyway. There’s also the part about breaking stuff down into small parts so that small cycles of meaningful dev can be accomplished. But that’s mainly to be able to release small and often - another thing you should already be doing to incrementally update your app avoiding big releases. It also stands in the face of waterfall development, though I see lots of teams doing test at the end still, making it waterfally.
I also see too many agile teams that have iterations that never finish, get feedback in prod, create half complete features that are weird or don’t gel - all in the name of agile.
I think the kids that get it were already doing it as it’s common sense. The rest end up sticking an agile badge on their practices when it’s all but.
You need <anything> because of the hostile nature between the creator and the patron of the creator.
The patron wants more, the creator doesn't want to create more, so you need to force a detente so the patron will continue to pay the creator, and the creator will continue to produce.
You get t-shirt sizes and points when the patron feels the creators aren't doing what they said they'd do, so in a way it's a communication problem, yes, but the cause of the communication problem is that there is almost never an alignment between the two groups, and so you need some kind of shared language where expectations can translate, and accountability can exist at all, usually in some flawed form.
That’s not a bad starting point. There’s more than one kind of feedback though! No matter how regularly you reflect on the process, if you’re ploughing through an unchanging backlog it’s still going to be a mediocre experience. Worse if the strategy keeps changing and you have no influence over it. Those unfortunately describe the experience of many - sometimes well-meant but an imbalanced and process-centric Agile with ironically little regard for the people doing the work.
This is how I see agile too. With it’s links to lean, it’s about minimising waste by minimising work in progress. Since the biggest waste is building something no-one wants, getting a small increment into the hands of a user/customer as soon as possible is key.
The problem, in my experience, is many development contracts are still structured in such a way that this becomes too difficult. Management fearing early exposure to the customer will lead to scope creep and cost growth.
I think the big divide here is between tech companies and non tech companies.
Tech companies can look at the agile manifesto and use it as a heuristic guide, because engineers are already kind of on the same page about it. You don't need heavyweight process etc.
Non tech-companies need the window dressing of tech companies to retain their best engineers, but ultimately agile is kinda telling them to turn everything upside down and also threatening to make them superfluous. Nobody is really up for that.
So agile in non-tech companies is Kabuki theater and engenders cynicism, and agile in tech companies is basically superfluous "water is wet" advice no one even bothers to comment about.
You'll notice a lot of these blogs about how agile has failed are coming from consultants, who are basically brought into traditional companies that are struggling with some part of this process, not tech companies.
This very much resonates with my experience. I've seen this very process take place when working at a small tech startup that ended up being bought by a non-tech giant keen on undergoing a "digitalization" initiative. Our prior processes were lightweight, goal-oriented and effective. No one bothered calling them "agile", they were just the natural way to deal with an ever changing landscape of customer and system requirements.
After the buy-out we were told to undergo a thorough transformation into this brand new unified top-down software development process the company had some consultants design for them. Complete with a baffling array of buzzword driven "agile" development practices and project/squad/team/chapter manager/lead/head roles to be filled. The more you kept inquiring what exactly those roles should entail, the more conflicting and vaguer the answers got until you realized that no one had the faintest idea how any of this was supposed to actually mesh together in practice. The license packages for the expensive project planning software we were to use where long paid however.
I totally agree, and I think another big reason Agile fails to cut through in companies is because anyone who wants to be Agile in a non tech company is fighting a crushing bureaucracy. So rather than innovating and climbing higher on innovation they leave due to the massive inertia that is encountered trying to change company culture.
I was at OOPLSA 1998 where it was going viral. Participated in the early C2 wiki. Participated in some of the early "workshops" on this upstart rebel movement.
Raised in a christian tradition, and somewhat a student of history of christianity, I was fascinated by parallels that unfolded in months` time what took place over centuries in christianity.
5 years after the movement began, I found that it felt as watered down, abstracted, and diverse as christianity spent 2000 years becoming. I could bond with a fellow programmer as an "agilist" and at a very vague level, there was some abstract similarity (e.g as christianity might be to "be kind", agile might be to "be lightweight"). Everyone took the parts they wanted and formulated what fit for them. Which was/is both a good thing and a bad thing.
We are unfortunately saddled with "religion" as the archetype of this behaviour.
Perhaps we could call it, "social virtue mythology".
What is a social virtue mythology? It answers the following questions:
* Why is there evil in the world? (ie., why arent things perfect)
* How do you overcome evil? (ie., become perfect)
* Who is good? Who is evil?
* In what or who should I place my trust?
* etc.
Today we can see many such competing virtue mythologies....
Why is there evil in the world? New Atheists: Religion; Feminism: Patriarchy; Socialists: Capitalism; Idenity-Woke: Essentialism.
Personally I regard this as wholely "mythological" because the questions these social-virtue systems are designed to answer "take place" in a utopian/dystopian reality.
Ie., many people eventually discover that there is no need of this type of mythology: reality is itself flawed. We aren't "owed" a utopia, and thus there is no question to answer.
Here, in agile then we see these patterns even in this thread. Some commenters are still trying to answer the question, "why isnt software design/construction perfect?" as-if there can be a virtue/vice answer, ie., "because we have failed to...".
Rather, no. These problems are irresolvable. The reality of software is intrinsically broken: there is no customer to give you the requirements, there is no habit which elicits them, there is no mechanism to build reliable software, and so on.
Many are extremely loath to confront this reality, especially as adolescents; ie., that they are powerless to build their utopia. The really-existing world has irresolvable contradictions that admit no virtuous resolution.
I've seen Agile at a handful of shops, a couple dozen projects. I've seen it implemented well just once...
The "scrum master" was a dedicated role, filled by a technically able person, who sometimes helped a little bit with the coding. This individual had read several books and taken a course on Agile. They were sincerely passionate about Agile and wanted to implement it effectively.
The "project manager", a DIFFERENT PERSON was part of the "business", and interfaced primarily with the scrum master, but they were with us for several hours of both planning and retro.
Our sprints were 2-3 weeks. At the end of the sprint, we spent AN ENTIRE DAY on retro. Before the next sprint started, we spent AN ENTIRE DAY on planning. We used a physical board which was matched by the tracker. They were kept in sync by individuals and validated by the DEDICATED DOCUMENTATION PERSON.
We broke down the tasks relentlessly, until there were almost no 3-point tasks, maybe one or two 5-point tasks, and everything else was 1-pointers. I think this one is the most bang for your buck, lowest hanging fruit, easiest to implement winner. Someone has to enforce it vigilantly until everyone is used to it.
Those three things are, IMNSHO, the keys to getting halfway decent Agile going.
Edit: Was one of the least soul-crushing and nicest experiences in professional software development. The planning meetings were a serious effort, but also a pleasure because I liked everyone on the team, and the large time windows made it not seem rushed, so we had time to joke around and shoot the shit.
How did this approach deal with difficult technical/algorithmic problems where someone needs to investigate something before further decisions can be made?
Example: You want to improve performance. Somebody will have to fire up a profiler and measure things, and then figure out which hotspots are worth optimizing.
I totally get breaking it down to "set up profiling tool", "do profiling runs for workflow X/Y/Z", and "identify low-hanging fruit". Presumably these tasks are not going to take the entire sprint.
How do you plan the next step in task format? Do you not do any further work on the topic until the next sprint, where you then plan out concrete "optimize function X" tasks?
If not, i.e. whoever does profiling also gets to optimize code during the same sprint, how do you account for that in planning? Three dummy tasks "optimize a hotspot" without any actual idea what it will involve? Maybe it requires rewriting a component to use different data structures? Maybe it's just a single JVM config tweak? "Could be a 1 or it could take my whole life?"
Maybe it's just less common in certain domains, but this kind of "we need to investigate this issue and then decide what changes to make based on the investigation (and good engineering judgement)" - which is bread and butter in my part of the org - does not seem to fit into the "tasks with points" concept at all. Well, unless you want to always spread the process of "investigate issue", "identify problem and solution(s)" and "implement solution" over a whole month. That seems soul-crushing to me.
To be clear, I'm very interested in seeing how to make this mesh with Agile. But it feels like trying to write a binary search in a language that has no indirection or conditionals.
> This sounds soul-crushing and I hope never to work in such an environment.
It isn't.
Soul-crushing is doing "Agile" where the "scrum master" tells you how long each ticket will take, where the client is invisible and never involved, and where retro is 1 hour at the end of the sprint and focused on blame.
The time spent in those retrospectives and planning sessions are about the technical aspects - how talks should be broken down, how overall design of interacting components should look like. It's 100% actual work.
I'd take these over another bs "status sync" any day.
I'd certainly like to be in an environment that has the dedicated people to those roles that so often are either combined or eschewed entirely, but 16+ hours of meetings per sprint sounds incredibly draining.
It wasn't particularly draining, because the large windows allowed a more relaxed pace, ensuring that nothing was missed, and a time for joking around.
And it was not "16+ hours of meetings", but two days of taking a break from coding to do more social work, also having a 2-hour lunch together in the middle :)
The best version of Agile I've ever been part of didn't start great, started like what you describe. And everyone hated the day long retro and planning. So we iterated over it until we got people to come to those meetings with their homework done and tasks reasonably thought out and split instead of having to waste everyone's time for it. And in the end it worked quite well. Retro and planning down to ~1 hour.
I will fight, in any team I'm a part from now on, to avoid the interminable scrum rituals. They are soul crushing for most good engineers I know. I never ever want to do them again.
It’s a very nice article. I’ve not seen agile done as originally visioned. Usually because the VP Engineering (or equivalent) and down wants to do it but the rest of the business isn’t interested.
Also the pattern I’ve seen is over time businesses want more and more fine grained control over the engineering process, as if treating the team like replaceable workers doing well defined tasks (like Ray Croc I guess) leads to more efficiency.
This doesn’t work when the burger and chips has zero cost of replication and your staff are not frying and serving chips, they are inventing new cuisines every day.
Agile is not about "mean efficiency", this is a serious and widespread misunderstanding.
Agile is a risk management method.
You won't build things faster with agile in general. It's aim is to decrease the chances of spending 80 manyears on a project and then realising at the end that you've built the wrong thing.
Well, a fluidity thing. But yes that’s related to risk. Being able to change direction at the drop of a hat. Also working with poor requirements which is a fairly common working condition in a lot of shops.
I really enjoyed this read. I’ve been grinding for so long in the huge corporation version of agile that I don’t even know what is what anymore.
It was refreshing to hear their take on project managers, as it matches how I’ve been feeling recently. They schedule 7 checkpoint meetings per week and I’ve given up on attending, I just can’t do it anymore.
I hope that we can move past agile into a new world that is free of the baggage and process of yesteryear. Even if the core principles are the same, let’s call it something different so we can start fresh and impress upper management.
This is how revolutionary Marxism functions. Nobody knows what the hell Marx wrote in those books, or what any of it means, but it cannot be argued in communist regimes that there is a system in place, and that the vanguard knows whats best. Replace Marxism with Agile, communist regimes with corporation, and vanguard with management, and you have the recipe for soulless employment.
People (read: Twitter users)[0] have discussed the parallels between Maoist/Marxist theory and both Agile and extreme programming—albeit less fever-dreamy
And here lies the difference: agile includes retrospectives. I
It's the only thing you need for agile. And if you don't have that, you don't have agile.
Readit News
The customer doesn't know what they want
Agile assumes as a first principal that including the customer throughout the development process will align the building team and the customer to come to the same conclusion.
This is almost never actually true. Having built and managed a lot of products I can state fairly confidently that the amount of time the customer has, and clarity about the problem they want solved is orders of magnitude more ambiguous than is required to actually build and deliver something to their satisfaction.
In my opinion successful products, and I mean ones with significant flywheels with customers and viral growth and stickiness are with few exceptions - accidents. It was someone who had an interesting thought and a lot of people for whom the product was good enough for their desire that they often didn't even know they had
You can't write down how to do that, it's like asking a nobel prize winner how they came up with their discovery. It's not repeatable. Some people have better intuitions than others and these are the people who are repeatably successful with products. But it doesn't generalize.
That said, the fundamental problem isn't wrong in theory, it's just naiive in practice.
This is not what the article is about. If you look at the Agile Manifesto, it says e.g.
- Individuals and interactions over processes and tools - Responding to change over following a plan
What you get in quite a few big companies following "Agile" with the air quotes is the opposite:
- Processes and tools over individuals and interactions - Following (and making) a plan over responding to change
Because that's what makes middle management happy.
So this perversion of "Agile", which is actually the opposite of Agile, is the FUNDAMENTAL problem.
I once had it used against me to justify not writing tests ("processes and tools!"). We had meetings about bugs instead. Seriously. Complain all you like about perversion, that was a perfectly valid interpretation coz those hallowed commandments are, frankly, ill-defined bollocks.
The principles are better but still suffer the same problem. If the most efficient way to convey information in a dev team is face to face why don't you come over here and I'll whisper my pull request in your ear. Jesus.
There's no point complaining about the perversion of a thing that was never very specific about what it wanted to be in the first place.
Once people (especially managers) get specific about what agile means to them it turns out it means very different things to different people. This is entirely the fault of the originators.
One benefit of Scrum (the catholic church to agile's christian sect) was that it was proscriptive and was specific. Unfortunately it's also a bit shit and its practitioner-priests LOVE to tell you that if it isnt working, well, You Probably Just Weren't Doing It Correctly.
(1) The Manifesto is written in non-actionable terms.
(2) Many of the same people that initially attached themselves to the manifesto were already pitching canned processes, that may have been the outcome of agile process where they originated but did not put the critical feedback mechanisms more prominently than the low-level processes. So, those became the actionable versions of “Agile”.
It would be so much better if instead of the “X over Y” language, the Manifesto has spent a few words talking about how the less important things were subordinated to the more important ones.
The weird thing is that continuous, bottom-up, empirically-based process improvement wasn't an unknown concept at the time the Manifesto was written. It wouldn't have been hard for it to reference more concrete, actionable, concepts and practices.
Agile is Dead - https://m.youtube.com/watch?v=a-BOSpxYJ9M
Here’s a blog post by him: https://pragdave.me/blog/2014/03/04/time-to-kill-agile.html
If that’s not following a plan over responding to change, i don’t know what is.
Agile tries to turn software development into a product design process rather than a product manufacturing process. Together with the client, iteratively you try to find the right solution and design for the customer problem.
By being easy to adopt, software lends itself to an iterative design process, where you adopt it multiple times based on the iterative feedback of the customer. But this means that at the end of the design process, you can skip the manufacturing part, as you've already built the software during design.
Alternatively you could argue that this is too expensive, inefficient and slow, and you need some design before you begin to manufacture software. Modeling tools could help here, e.g, low code tooling like CAD tools in fysical product design.
Oh, man. I think you hit the head of the nail. One succinct phrase that describes the entire problem.
Early in my career somebody said something like "make sure you distinguish between <<what they (customers) want>> and <<what they need>>.". This thought stuck with me for years and in my opinion has been reposible for success I witnessed.
It's important to unwrap "what they want" so you, as developer, can apply more first principle thinking to arrive at "what they need".
BAs have tendency to project their own solutions, however when unwrapped so you see the problem that needs solving, you may arrive at solution that is simpler (like orders of magnitue is not that rare in my experience), reflects reality better, takes advantage of somehow hidden from BAs PoV system internals etc.
Agile enouraged dialogue is the platform where those ideas can surface.
Usually what happens is that you get a customer team, that is supposed to voice how the organization wants the software to be.
So the same mismatch is bound to happen anyway.
And when they do involve the people from the field, depending on the company culture, they might even be quite positive on the demos (cause being negative isn't good) and then completely find it unusable on final delivery.
Agile misses that many engineering teams lack people skills to actually navigate and avoid these scenarios.
EDIT: several typos
"I thought this was your job" is something I've heard a few times from clients when we have sent over wire frames or beta applications.
Especially in B2B and technical realm it is feasible to actually plan in every sense of the word "plan" a new product - business, tech, etc. Of course the plan must be malleable on the facts on ground. And probably is incorrect on a few places. But never the less, a plan that will get a business from place a (no product, but strong understanding of customer needs and business plus strong technical savvy) to place b (final product, customers, revenue etc).
I've seen this many times, both in startups with a new product and new customers (automotive software) and established companies that just expand their existing portfolio (cad software).
I'd say the "not an accident" is feasible if these facts are in place:
- enough A-team players (tech,business,sales)
- healthy culture (that facilitates long term growth and can focus on the product without too much politics)
- understanding of customer business
- identified an OBVIOUS need for the customer
- strong engineering org (can be just a few coders that get things done or larger)
So I agree, you can't train for these requirements like you can train airplane pilots.
But I do claim you can have enough understanding that is the current org in possession of these qualities. And if it is, it's a very reasonable risky investment to try to create a product for the need.
For internal customers or in B2B settings getting the actual users and outcome owners involved early is vital - but they are often poorly incentivized to get into the action as there are layers of business mangers etc in place that are supposed to stand in (but lack the detailed knowledge).
What was the 'standard' way to develop systems when the Agile Manifesto was written: the same as for building a bridge.
You gather and freeze requirements. You write a massive plan and plenty of specs. You code accordingly. You deliver.
Time to working software is long and a lot of the effort that goes into specs is wasted, especially for design specs.
Here's the introduction of Steve McConnell's "Rapid Development" from 1994 - well before the Agile Manifesto! - Chapter 1, page 1:
> The product manager told me he wanted to build a product right for a change. He wanted to pay attention to quality, prevent feature creep, control the schedule, and have a predictable ship date.
> When the time came to actually do the project, it became clear tha getting the product to market quickly was the only real priority. Usability? We don't have time. Performance? It can wait. Maintainability? Next project. Testing? Our users want the product now. Just get it out the door.
> This particular product manager wasn't the manager on just one product. He could have been almost any product manager I've worked for. This pattern is repeated day after day, state by state, all across the country. ...
This is as far from building a bridge as you can get. The next page of the book starts:
> [1.1] What is Rapid Development?
> To some people, rapid development consists of the application of a single pet tool or method. To the hacker, rapid development is coding for 36 hours at a stretch. To the information engineer, it's RAD - a combination of CASE tools, intensive user involvement, and tight timeboxes. To the vertical-market programmer, it's rapid prototyping using the latest version of Microsoft Visual Basic or Delphi. To the manager desperate to shorten a schedule, it's whatever practice was highlighted in the most recent issue of Business Week.
Agile drew from earlier methods ("intensive user involvement", "rapid prototyping") which were pretty common even before Agile.
The objective function is some function of the quality of the product, as measured by the customer, and the work put in. The gradient evaluation is then the process of determining which feature gives the most bang for the buck, implementing it, and repeating.
But gradient descent has problems with local minima. So if that's what Agile is, no surprise it doesn't do surprising projects - even if the customer does know what they want. If the programmers are too inexperienced and it's not just a CRUD, then you'll end up with something like Jeffries' TDD Sudoku solver.
Lean is.
Agile is “It would be nice to do what works for your team in your particular challenges, not some pre-packaged one-size-fits-all processes sold by consultants. Now, here’s a bunch of pre-packaged one-size-fits-all processes, and an army of consultants selling them.”
You push decisions as far out as possible and gather feedback from individuals every step of the way.
This is what allows the greatest amount of flexibility.
The biggest problem with agile is that it assumes all problems can be put off to the last minute.
I'm frequently finding myself in situations where it is 100x better to hammock program it out for a few weeks before pushing any code. Those systems last ten years. Those systems are able to be modified simply and quickly when business needs change. Those systems do build up legacy code, but it's at a slower rate than other systems. Those systems have fewer hacks over time and fewer bugs as a result.
This isn't a dig at agile, it's a dig at using agile as a design process, for which it is utterly inept. Agile is great at pushing products, terrible at pushing thoughtful system design. (On average, sometimes working a problem works too but more often than not you don't get the chance to redo middle sprint 5.)
I don't disagree, though I firmly believe Agile does a much better job of dealing with the problem you state than any other software development methodology that has been attempted before.
Is it perfect? of course not. As a methodology, is it better for building software than methodologies that have been used to build enormous physical structures for generations? Undoubtedly yes.
As far as I'm aware, there currently isn't anything better for building complex software with changing requirements (i.e. the real world).
The only thing that is less likely to have the building team and customer come to the same conclusion is to add many more layers in between.
I’ve seen where that ends. It’s not a good place.
If that were true the customer never drop you for the competition. The customer will drop you in a heartbeat.
The customer cannot express what they want in a language you can articulate which is something amazingly different. As a product manager you need to account for that.
It's a way to meeting actual business software needs, much more suited to custom development for one customer than product development, which necessarily requires some additional insight into what the market wants.
That's pretty much all you can do. How you organize it only matters insofar as it makes your team effective - sprints, waterfall, XP whatever... is kind of a toss up and context dependent.
The Agile Manifesto is a failure at trying to make Agile happen because it can't tell you how to make it happen, because it varies wildly. No manager can read a book on how to "Agile-ify" their org, they have to apply their brains and figure out how their specific version of "Agile" will work. But the skill-set required to do this is not a Managerial skill, it is a lower-level-worker skill. But it's also a very advanced lower-level-worker skill.
And that's why things like "Agile", "DevOps", etc will fail. People at the higher end have no clue how to make it happen, and people at the lower end who have an idea how to make it happen don't have the power to make the organizational changes to do so. You need a way for the lower-end people to tell the higher-end people what to do, and have the higher-end people listen to them, and make the changes happen. This is very hard in a traditional organizational hierarchy, because higher-end people have big egos and bigger concerns over things like politics.
Maybe this isn't the norm?
The best I have seen were mini-silos that managed to de-couple themselves from the org chart.
However I also have seen that it hardly lasts more than a couple of years, because as soon as someone noticed the success of the business unit they were re-integrated so that they could teach the reasons for their success to others.
You can imagine how well did they usually fare afterwards.
Wich makes it all a glorious Catch 22.
These companies, that rely completely on software, are unable to make the required leadership changes. It’s cultural.
I completely disagree. Most team leads and technical managers where I've worked get promoted up from within a highly technical position. I wouldn't have any respect for my team lead or my PM if they didn't know what they were talking about.
If my PM is going to try to tell me that I should work on this feature over this other feature or I should implement it in this way over this other way do you really think I'm going to listen to them if it's clear they have no idea what the implementation details are let alone the tools? Of course not.
I'm making DevOps happen and I do that by knowing the tools and practices. I'm being given the power I need to make it happen. I earn the respect every day I come in and write the code or identify the weaknesses we need to shore up to move faster.
In the end if you are difficult you become easy to replace.
How much of each dollar you make for the company with your hard work do you take home?
That doesn't mean you need points or tshirt sizes or constant status meetings. You just need channels of communication with the right people and a chance to adapt to what they say.
And it’s something that good devs will already be trying to accomplish anyway. There’s also the part about breaking stuff down into small parts so that small cycles of meaningful dev can be accomplished. But that’s mainly to be able to release small and often - another thing you should already be doing to incrementally update your app avoiding big releases. It also stands in the face of waterfall development, though I see lots of teams doing test at the end still, making it waterfally.
I also see too many agile teams that have iterations that never finish, get feedback in prod, create half complete features that are weird or don’t gel - all in the name of agile.
I think the kids that get it were already doing it as it’s common sense. The rest end up sticking an agile badge on their practices when it’s all but.
The patron wants more, the creator doesn't want to create more, so you need to force a detente so the patron will continue to pay the creator, and the creator will continue to produce.
You get t-shirt sizes and points when the patron feels the creators aren't doing what they said they'd do, so in a way it's a communication problem, yes, but the cause of the communication problem is that there is almost never an alignment between the two groups, and so you need some kind of shared language where expectations can translate, and accountability can exist at all, usually in some flawed form.
The problem, in my experience, is many development contracts are still structured in such a way that this becomes too difficult. Management fearing early exposure to the customer will lead to scope creep and cost growth.
Tech companies can look at the agile manifesto and use it as a heuristic guide, because engineers are already kind of on the same page about it. You don't need heavyweight process etc.
Non tech-companies need the window dressing of tech companies to retain their best engineers, but ultimately agile is kinda telling them to turn everything upside down and also threatening to make them superfluous. Nobody is really up for that.
So agile in non-tech companies is Kabuki theater and engenders cynicism, and agile in tech companies is basically superfluous "water is wet" advice no one even bothers to comment about.
You'll notice a lot of these blogs about how agile has failed are coming from consultants, who are basically brought into traditional companies that are struggling with some part of this process, not tech companies.
After the buy-out we were told to undergo a thorough transformation into this brand new unified top-down software development process the company had some consultants design for them. Complete with a baffling array of buzzword driven "agile" development practices and project/squad/team/chapter manager/lead/head roles to be filled. The more you kept inquiring what exactly those roles should entail, the more conflicting and vaguer the answers got until you realized that no one had the faintest idea how any of this was supposed to actually mesh together in practice. The license packages for the expensive project planning software we were to use where long paid however.
Raised in a christian tradition, and somewhat a student of history of christianity, I was fascinated by parallels that unfolded in months` time what took place over centuries in christianity.
5 years after the movement began, I found that it felt as watered down, abstracted, and diverse as christianity spent 2000 years becoming. I could bond with a fellow programmer as an "agilist" and at a very vague level, there was some abstract similarity (e.g as christianity might be to "be kind", agile might be to "be lightweight"). Everyone took the parts they wanted and formulated what fit for them. Which was/is both a good thing and a bad thing.
Perhaps we could call it, "social virtue mythology".
What is a social virtue mythology? It answers the following questions:
* Why is there evil in the world? (ie., why arent things perfect)
* How do you overcome evil? (ie., become perfect)
* Who is good? Who is evil?
* In what or who should I place my trust?
* etc.
Today we can see many such competing virtue mythologies....
Why is there evil in the world? New Atheists: Religion; Feminism: Patriarchy; Socialists: Capitalism; Idenity-Woke: Essentialism.
Personally I regard this as wholely "mythological" because the questions these social-virtue systems are designed to answer "take place" in a utopian/dystopian reality.
Ie., many people eventually discover that there is no need of this type of mythology: reality is itself flawed. We aren't "owed" a utopia, and thus there is no question to answer.
Here, in agile then we see these patterns even in this thread. Some commenters are still trying to answer the question, "why isnt software design/construction perfect?" as-if there can be a virtue/vice answer, ie., "because we have failed to...".
Rather, no. These problems are irresolvable. The reality of software is intrinsically broken: there is no customer to give you the requirements, there is no habit which elicits them, there is no mechanism to build reliable software, and so on.
Many are extremely loath to confront this reality, especially as adolescents; ie., that they are powerless to build their utopia. The really-existing world has irresolvable contradictions that admit no virtuous resolution.
The "scrum master" was a dedicated role, filled by a technically able person, who sometimes helped a little bit with the coding. This individual had read several books and taken a course on Agile. They were sincerely passionate about Agile and wanted to implement it effectively.
The "project manager", a DIFFERENT PERSON was part of the "business", and interfaced primarily with the scrum master, but they were with us for several hours of both planning and retro.
Our sprints were 2-3 weeks. At the end of the sprint, we spent AN ENTIRE DAY on retro. Before the next sprint started, we spent AN ENTIRE DAY on planning. We used a physical board which was matched by the tracker. They were kept in sync by individuals and validated by the DEDICATED DOCUMENTATION PERSON.
We broke down the tasks relentlessly, until there were almost no 3-point tasks, maybe one or two 5-point tasks, and everything else was 1-pointers. I think this one is the most bang for your buck, lowest hanging fruit, easiest to implement winner. Someone has to enforce it vigilantly until everyone is used to it.
Those three things are, IMNSHO, the keys to getting halfway decent Agile going.
Edit: Was one of the least soul-crushing and nicest experiences in professional software development. The planning meetings were a serious effort, but also a pleasure because I liked everyone on the team, and the large time windows made it not seem rushed, so we had time to joke around and shoot the shit.
Example: You want to improve performance. Somebody will have to fire up a profiler and measure things, and then figure out which hotspots are worth optimizing.
I totally get breaking it down to "set up profiling tool", "do profiling runs for workflow X/Y/Z", and "identify low-hanging fruit". Presumably these tasks are not going to take the entire sprint.
How do you plan the next step in task format? Do you not do any further work on the topic until the next sprint, where you then plan out concrete "optimize function X" tasks?
If not, i.e. whoever does profiling also gets to optimize code during the same sprint, how do you account for that in planning? Three dummy tasks "optimize a hotspot" without any actual idea what it will involve? Maybe it requires rewriting a component to use different data structures? Maybe it's just a single JVM config tweak? "Could be a 1 or it could take my whole life?"
Maybe it's just less common in certain domains, but this kind of "we need to investigate this issue and then decide what changes to make based on the investigation (and good engineering judgement)" - which is bread and butter in my part of the org - does not seem to fit into the "tasks with points" concept at all. Well, unless you want to always spread the process of "investigate issue", "identify problem and solution(s)" and "implement solution" over a whole month. That seems soul-crushing to me.
To be clear, I'm very interested in seeing how to make this mesh with Agile. But it feels like trying to write a binary search in a language that has no indirection or conditionals.
Given the "about one hour of work, give or take" target point value we used, there weren't many tasks which had more points than that.
It isn't.
Soul-crushing is doing "Agile" where the "scrum master" tells you how long each ticket will take, where the client is invisible and never involved, and where retro is 1 hour at the end of the sprint and focused on blame.
The time spent in those retrospectives and planning sessions are about the technical aspects - how talks should be broken down, how overall design of interacting components should look like. It's 100% actual work.
I'd take these over another bs "status sync" any day.
I'm getting ill just thinking about this. Two whole days wasted every sprint!
Dead Comment
And it was not "16+ hours of meetings", but two days of taking a break from coding to do more social work, also having a 2-hour lunch together in the middle :)
I will fight, in any team I'm a part from now on, to avoid the interminable scrum rituals. They are soul crushing for most good engineers I know. I never ever want to do them again.
Dead Comment
Also the pattern I’ve seen is over time businesses want more and more fine grained control over the engineering process, as if treating the team like replaceable workers doing well defined tasks (like Ray Croc I guess) leads to more efficiency.
This doesn’t work when the burger and chips has zero cost of replication and your staff are not frying and serving chips, they are inventing new cuisines every day.
Agile is a risk management method.
You won't build things faster with agile in general. It's aim is to decrease the chances of spending 80 manyears on a project and then realising at the end that you've built the wrong thing.
It was refreshing to hear their take on project managers, as it matches how I’ve been feeling recently. They schedule 7 checkpoint meetings per week and I’ve given up on attending, I just can’t do it anymore.
I hope that we can move past agile into a new world that is free of the baggage and process of yesteryear. Even if the core principles are the same, let’s call it something different so we can start fresh and impress upper management.
In fact, all agile needs is good retrospectives, and all the rest you can decide there.
It’s like shaping a system of people. I think of it as 50% of the development endeavor at larger shops.
Retros are used to identify ”technical organization debt” starting to build up, in a way.
[0] A couple of said theory tweets, from memory
https://twitter.com/mmabeuf/status/1352450003231506432
https://twitter.com/mmabeuf/status/1323758677099270147
And here lies the difference: agile includes retrospectives. I It's the only thing you need for agile. And if you don't have that, you don't have agile.