I've found over the years that Martin Fowler blogs about things the C-Suite is already doing at their companies, and they use this as validation for their choices. This in turn has a virtuous cycle of keeping him popular.
Almost everything he hawks causes misery for software developers, like his whole multi-year spin for Microservices, he's turned refactoring into a fetish (honestly there are people at my dayjob who have been "refactoring" the entire 3 years i've been here) and now hes blogged multiple times about platform teams - which - having been on one - don't work. I see him as a very damaging character.
Well intentioned, i'm sure, but I don't think he has good ideas.
Microservices was at its core a good idea. I think Fowler was just overawed with how well it worked for him in his organizational context and was unaware of how critical that context was to making it work.
Most of my nightmares with microservices comes from where there has been > 1 per team. That came from people who read his "microservices are great!" blog post and just thought that they should build as many as possible. I used to hate him because of that - because it was a logical response to his blog post and people would use it to justify their technical decisions - with argument-from-authority.
In another company I worked at each team maintained ONE microservice and it worked really well.
Because it was a corporate environment with all the usual power politics, control freakery and lack of effective inter-team communication, even though it was more work overall and definitely a technically inferior solution, it did a neat end run around those systemic corporate issues by giving teams relative freedom of development, freedom of deployment and a deliberately simple, loosely coupled, standardized JSON interface to microservices run by other teams whom they didn't talk to much.
So, I get it, but I lost a lot of respect for him over his inability to parameterize his suggestions and the fact that he didn't seem to initially really understand why microservices worked so well for him.
I disagree, microservices at their core were not a good idea, they were missing the forest from the trees.
Micro services are an attempted solution for having highly bespoke infrastructure that required difficult, long, manual, fragile deployments that were high risk, and thus didn’t happen often and delayed business value. The problem was that they saw that issue and thought it was monolithic applications, not just having bad infrastructure. 99% of the benefits of devops with almost none of the drawbacks are done by having all of the standard dev ops stuff like Cicd, containers, software defined infra, etc and just doing it on the normal monolith.
Eventually you do need to break things up, but the micro services single functionality, single team per service type of micro services goes way to far and harms more than it helps.
Working for 3 years and only refactoring, that's a whole new level of comedy. If you blame Fowler for this madness, then I see how you think his work is damaging.
But its a bit like a health influencer telling people to stay hydrated and blame him for his fans damaging themselves by trying to drink 10 liters a day.
> he's turned refactoring into a fetish (honestly there are people at my dayjob who have been "refactoring" the entire 3 years i've been here)
That's kinda dumb, but the book is super good, I learned a whole lot from it, and apply the principles basically all the time.
He's been banging the drum on testing for longer, and that makes up for a lot of craziness (I'm with you on microservcies, except where you have 1k+ developers where they might make sense).
- Platform sounds like "everything that doesn't fit into a vertical slice of business" - that is a _hell_ of a lot of work for any team.
- Business hates and underfunds these teams because they are often blamed as being a blocker because of the previous statement. They are a pure cost center too. Career suicide joining them.
- Unless you have a very motivated team lead you will end up doing the work of the most aggressive and loud team leads from other teams. Anyone good at playing politics will dictate your roadmap.
- Given the first statement, how do you hire someone for these teams? If they are responsible for platform evolution, upgrades, CI + CD, making frameworks, writing scripts, improving developer experience PLUS you are the backstop for the full backlog of every other team so need a lot of domain expertise. You would need to pay someone a lot to do this right? Well no - because of point 2.
The idea of a platform team is one of those weird things that sounds obvious and clearly needed - until you marry the idea to that of any org that has layers of management + politics and a misunderstanding of the long tail effects of platform work.
An interesting thing that happened at our company when we ended up having a platform team was that the "Can do" attitude of our previously homogenous group of mobile devs stopped over night - "That's a platform role" became a common refrain.
My 2c's are that platform teams are a silo for expertise on how anything is run at a company and can become a bottleneck for other teams when getting things done. In an ideal scenario the product team can consume resources via an API, but in my experience resources are only assigned via a ticketing system and the associated communication barriers this creates leads to friction.
I led a large (50 people+) platform org - Organizations struggle with modern backend systems and cloud, and find themselves in a cyclical loop of hiring, reorganization and reprioritization. I just wrote about it here a few days ago: https://klo.dev/evolution-of-cloud-infrastructure-and-platfo...
If you have upper management reading a blog and doing a re-org to implement, that is the problem causing misery: management copy/pasting org structures/ideas and thinking it will work, much like technologists cramming some new tech into solutions where it doesn't belong.
I think the issue is a lot of people read his stuff and then blindly think it all applies to them and microservice themselves into a corner without actually thinking through if it actually applies to their situation. Same goes for various blogs from FAANG companies about scaling. Hardly anyone has scaling issues that they have, yet technologists like the bright and shiny new thing so they end up adopting over complicated solutions to their simple problems.
The team types he talks about in this post I see working well in some companies and situations but he's not saying "thou shall have a platform team and it solves all your woes". The first paragraph states the primary team is a "stream aligned team" which I take to mean a product team that is responsible for app(s) from top to bottom - UX, UI, backend, scaling, support, etc.
This is a general issue across all fields - blindly following other companies or people is never a good idea because you don't know all the internal and external variables that the person/organization was working with. In fact in business schools they often teach you to scrutinize established patterns and norms and apply various frameworks in conjunction with statistical analysis to draw conclusions.
What your employees and customers observe and perceive is the reality, not what Google, Martin Fowler or Bill Gates prescribe.
I don't know if I'd go that far; perhaps platform code could be done well, but you make it seem like it can't work.
I'm not sure I completely understand "platform" in this context. I've worked on teams that tried to write a common platform, then delegate to more specialized teams for specific implementations.
I have often hated working under these conditions. These problems are often true for software suppliers whether internal and external. There is a disconnect, and weirdly platform is not motivated to provide good quality code with whatever analysis / testing is needed. It can be even worse if the supplier is internal, and it's known that the company doesn't want to use someone external (where's your leverage!)... good chance platform has more political pull than you, too!
Often times they just ship it and say "your problem". Meanwhile, they also insist that we do not modify their code, or else take full responsibility. There's more motivation to finger point, or push responsibility on to your customer (including other internal teams).
Mind you that in my industry this all pushes the limits of ethics, and maybe our obligations dictated by regulations. Of course, we do everything we can to meet our obligations and do the right thing. However, you're left with the decision to leave in poor quality code (or missing work products), or modify / test code you probably will struggle to understand. It will also be difficult to understand the implications, eg. for other modules.
Could you elaborate on the part about platform teams?
In my experience, it's worked out terribly when a company treats a platform team as a catch-all for any back-end service regardless the business domain. On the other hand, it's seemed to work reasonably well when the team has a narrow (and very clear definition) of which services they own and why.
That said, I've never worked directly within a platform team as an engineer, so maybe it just appears that way as an outsider.
I'd would not call it "virtuous cycle" more of vicious cycle.
Well intentioned I doubt. Unless it means McKinsey style well intentioned: hammering about re-orgs and reforms while generating ton of consulting revenue with it.
> refactoring into a fetish (honestly there are people at my dayjob who have been "refactoring" the entire 3 years i've been here) and now hes blogged multiple times about platform teams
Side point here…
I absolutely love refactoring sometimes. Like taking a messy chunk of code with deeply nested conditionals a d simplifying it, which makes it more readable and extendable. You also have very clear requirements as well (whatever the original code did). There are few things in software development that bring me more joy than that.
But its often not necessary. And furthermore, refactoring is often not of that type. Sometimes people rewrite things because they don’t understand how they work (and writing is easier than reading). Or because they completely misjudge the value of the refactor.
Microservices are essential at any large company. Can you imagine if Facebook was still a PHP monolith? They'd need 5,000,000 instances to run the app.
> microservices are essential at any large company
There in lies the rub, the majority of engineers don't work at companies that need the kind of scale that Facebook has.
The majority of us work at smaller shops that can get by fine without all the overhead that microservices introduce. The problem that I see is that there are to many folks not weighing the pros and cons of the architectural decisions they are making, and are just joining the cargo cult.
On a small-medium vertical team at a large company? Probably-maybe not.
Many, many engineers (and managers maybe even more so) immediately jump to micro-services (often resume-driven development) where most companies have poor tooling and poor security knowledge that turn them into a living productivity-crippling nightmare to work with.
If that were true, then moving to a microservice architecture they'd need 5 million instances of the microservice that replaces the part of the monolith that required 5 million instances in the first place, plus a couple of millies for all the other services.
It depends on the granularity. I see projects where 3 developers create 13 micro services. Basically they buy all the complexity without having any benefits.
I generally hate the term “micro services”. They are just “services” and nothing new.
As someone who used to be part of a platform team, and is now actively working in the same organization to fight the creation of any more platform teams, I very much disagree with the article that this is a good working model.
What we have found is that platform teams are a huge point of frustration for product teams, since they cause unnecessary and unmanageable coupling between different business divisions, leading to impossible to balance priorities. What we've seen happen over and over is that product A needs Feature A, and product B needs Feature B, and they both need it tomorrow, and the platform team only has resources for one. And since Product A and Product B are in different business divisions, you end up needing to involve a senior VP or even an officer to make a prioritization decision for a simple software feature, and everyone gets frustrated by the process.
What we're striving towards instead is an "inner source" model, where a platform is collaboratively developped and maintained by multiple product teams. Each product team is then empowered to build new features as needed into the platform, and others can reuse them.
Of course, the platform needs some architects and overall review, and all teams will not equally participate. But the key point is to encourage this collaboration mode, where no one team is a bottleneck for multiple products.
The inspiration for this structure is obviously the open-source world, where multiple corporations and foundations collaborate on a single code base without needing to rely on a single platform vendor to provide features for them.
> What we're striving towards instead is an "inner source" model, where a platform is collaboratively developped and maintained by multiple product teams. Each product team is then empowered to build new features as needed into the platform, and others can reuse them.
I have seen this fail (once myself and once through coworker experiences) as well, IMO the best approach is to have as little shared infrastructure between teams as possible even if it means more work in the end
IMO the best approach is to have as little shared infrastructure between teams as possible even if it means more work in the end
That's the route to shipping features faster for sure, but it costs more, requires duplication of effort, and is really painful when you discover two teams built the same thing and you want to unify on one version rather than paying to run and maintain both. Plus, when the cost cutting comes in the bad times, you find whole teams are axed because you don't need two sets of people doing the same job. The remaining engineers then have to unify the features but without the resources to get the work done and without access to the institutional knowledge required to understand the version they didn't build themselves.
If you're an engineer in a FAANG it's not a bad solution to the shipping issue. If you're anywhere else it's a road to absolute chaos.
This sounds like fundamentally a business problem (we have finite resource, is product A or B our priority?) and trying to solve it with choices in the technology division. Well, here be dragons.
Based on my experiences, the idea that a genuine bottleneck in tech is being made visible to customer division management and senior management, is generally an authentic way to ensure realism creeps back into the prioritisation and funding processes.
A platform team might have a very unevenly distributed workload. Some weeks nothing has to be done on the platform, everything works fine. Then all of a sudden 2 teams request some extra features. How many people do you put in the platform team?
Either you put too many in there, and they are basically idle the time in between. Or you put a limited team, and once even a single team requests something they will have to wait for the developments, let alone 2 requests at the same time.
The OP describes an excellent solution: the team that needs the feature, pays for it in man-hours. In fact, those developers that need it, are pretty familiar with using it, so most of the time capable of doing some developments on it. They can also schedule in the work in their project. The teams that will need it afterwards will get it for free.
Sure, it's fundamentally a business problem. But it's very hard to do product planning when you need to take into account global priorities for the platform team. Plus, you need to spend time to even estimate the cost of Feature A and Feature B, so you actually end up with several rounds of discussions where you first prioritise spending time on the estimation, then the feature itself etc. Not to mention, if this is a deep platform feature, it's probably only a prerequisite for an actual customer facing-feature that requires additional work, which gets delayed by all the estimation work and so on.
Additionally, this overall means that no one can plan in advance any features that depend on the platform, since global business priorities are constantly shifting.
Not to mention the political angle, where more adept business leaders can always push through their pet platform features to the detriment of what are more obvious business goals. Or at least, it will always seem this way to teams whose requirements keep falling through the cracks.
This is good insight and matches with my experience. As a product-focused engineer/manager, it’s too often I’ve found my teams giving up and taking into our own hands what “should be” owned by platform teams, while they work on things nobody seems to see value in.
> What we're striving towards instead is an "inner source" model, where a platform is collaboratively developped and maintained by multiple product teams.
If a platform has shared ownership, then decisions will get implemented by cohorts only thinking for themselves and thus damaging the long-term roadmap. All systems, especially complex ones, need sole owners or they will devolve into what is essentially a pyramid of doom at the product level.
Those "long-term roadmaps" are, in my experience, created by self-important architects who are far removed from the realities and needs of everyday developers. Full of lofty ideals, grandiose abstractions, and "silver-bullet-itis".
Good communication across teams and good design (i.e. flexible, maintainable, extensible) tends to alleviate the tunnel vision associated with each team building what works for them.
You can still have code/design reviews by other stakeholders. It just means that if something is really high priority for you but not for them you can just do it yourself.
I've had to wait weeks for one-line changes because somebody else had to do them.
> What we've seen happen over and over is that product A needs Feature A, and product B needs Feature B, and they both need it tomorrow, and the platform team only has resources for one. And since Product A and Product B are in different business divisions
That seems like an unnecessary coupling between different products.
I understand this concept of a platform team mainly within one (larger) product. But maybe it's the same thing, just on a different scale.
> What we're striving towards instead is an "inner source" model, where a platform is collaboratively developped and maintained by multiple product teams. Each product team is then empowered to build new features as needed into the platform, and others can reuse them.
I think it breaks down at a certain point of complexity. The platform itself is complex enough that it's very difficult for product teams to understand it holistically. I've seen this multiple times when the product teams were still able to make the changes as needed, but over the long term this approach created a hot mess of platform features which didn't align to each other and no single person understood.
Sounds like you were working at a terrible organisation, not that there is anything wrong with running a platform team. Teams with stressful deadlines, having to involve VIPs or architects at the org to make decisions rather than giving teams autonomy to make their own decisions.
Not sure I'd like to use a platform with a code base that has the whole organisation contributing to it. Sounds like a design-by-committee dumpster fire. Teams need ownership of their product to perform effectively. You can claim that this is collective ownership of the platform, but in reality no one will feel any responsibility for the platform when things don't work or don't align entirely with the particular thing they are currently working on.
And have you ever looked at the source code for most open source software? It's an absolute mess that requires the collective effort of large numbers of people to keep it going. It works, but it's not exactly a good or effective model for internal software.
I don't think the org is/was dysfunctional. Tight deadlines are a reality of certain business sectors, especially when selling big expensive B2B software, where you have to sell something in your customers' budget windows or lose the opportunity. And I don't see how a platform team could be empowered to decide on its own which of two competing business goals, from two separate business groups, to pursue, and which to postpone for months.
Related to collective dev/ownership, that is a risk, I absolutely agree. That is going to be the hardest part for sure, maintaining a uniform standard and a high bar for contributions to a common platform.
Finally, I believe the general opinion is that open-source software is higher quality and especially has higher code quality than internal company software. I can't say I studied any directly comparable open-source projects VS our internal tools, but code quality definitely varies significantly between our own internal projects, with some extremely clean and well written, while others are messes of hasty bug fixes and accumulated tech debt.
I have seen platform teams work extremely well. I have also seen them fail spectacularly. The main reasons? The organization's ability to identify, or hire, the kind of developers that do great on platform teams. At the very least, the senior members need to have a mix of empathy and technical excellence that is often hard to find, and when you only have one of the two (or at worst, neither!), the platform fails to gain any traction, or is mandated, and ends up being a noose to productivity instead of a boon.
Sometimes the platform team is staffed by people that have been there forever, as a sort of semi-promotion. But when they know everything, it's easy to have little interest in the difficulties of learning internal concepts: After all, the learning has already been done. This makes the tools be technically capable, and intractable. Other times, the team is easy to get along with, but what they deliver isn't very good at all, and the lack of quality is papered with social skills.
You need people capable of understanding the problem other teams have, and their architectural constraints, and deliver something that will save them time, and they'll prefer to use over some open sourced hodgepodge. They need to think of upgrade paths, or live in a low-repo environment where the platform team can upgrade things for everyone. The customer service attitude should be immaculate, as to make people be happy to ask for help, yet be so good at documentation, or at simple enough architecture, as to make that customer service load be light. Many places can't hire people that meet those kinds of profiles at all, as someone like that will basically excel in most roles in most companies. So yes, you end up with the technically capable, yet gruff guys that nobody wants to talk to: The equivalent of Seinfeld's Soup Nazi... and that's if at least they are very good.
Most team topologies will work if your company is full of empathetic heroes though, so platform teams might not even be needed if you really are that good at hiring.
I think a critical component of making platform teams work is to allow internal competition between the platform team and the stream-aligned team using some other technology. In this way the platform team is a stream-aligned team whose customers are internal, and they have to win or lose within their market (internal teams). For example, a stream-aligned team can either use platform team CI/CD or github actions.
What I've seen is that platform teams work well in either small scale (because focus is clear to all) and large scale (because of this internal competitive dynamic) but are extremely hard to execute between the two.
Sure, there are different ways of building platforms, and some leave the end-users with fewer options than others. But ultimately, if you have a separation between platform and product teams, the product teams will depend on the platform team to provide a better fishing rod to catch a bigger fish, they aren't empowered to enhance their own rod. And if the product teams are empowered to modify the platform, then you're actually implementing another model than what the article and book seem to suggest.
> What we've seen happen over and over is that product A needs Feature A, and product B needs Feature B, and they both need it tomorrow, and the platform team only has resources for one.
Is this any different from a product team where there's competing demands between Customer A and Customer B?
In some ways, it's the same situation. However, the big difference is that I'm not a colleague to the customers waiting for this feature. If they get frustrated, (1) I don't get to see them at coffee time being frustrated with my team, and (2) they have a quick fix for their frustration: but from someone else. Internal customers don't typically have that luxury.
I recognise the friction points very well. It's very frustrating and limits product team velocity. That said, in my experience this is mainly due to a misinterpretation and mis-implementation of platform teams. Too often, platform teams are forced upon product teams. This misses an essential element of a platform, optionality. A platform should be a jumping board, that people can choose to accelerate development. When a platform is made mandatory it misses essential feedback mechanisms, such as rate of adoption, for it to steer in the right direction. While the rate of adoption is still often seen as a metric for a platform team's success, the mandate to enforce the platform onto product teams is fundamentally corrupting. In addition, the tools to truly accelerate development are not the same as time progresses. Without optionality, there is never the incentive to sunset anything the platform provides. Deviations of technology/pattern/solution use are often seen as negative aspects of the product team's performance, but rarely reflect back on the platform team's output.
TLDR; platform teams without product team's freedom to deviate (optionality) is corrupt and can destroy a large chunk of engineering velocity.
That is one important aspect, absolutely. But I don't think it's the main point. There are normally good reasons for which the platform itself is necessary and useful. Working around it, while a good option to have, is rarely ideal - or else you fall back to the model where every team invents and builds its own tools. Renouncing (or worse, forking) the whole platform because a feature can't be prioritized to be added to it is not exactly a great outcome.
I recognize this problem, although my experience was a lot worse. It seems to me that this one is very, very hard for many people to realize:
> Platform teams, however, need to build their services as products themselves, with a deep understanding of their customer's needs.
If there is no malfunction like a monopoly or a scam, as a customer I just choose a different product if one is not meeting my needs. The same must be true of a platform, and if it happens a lot that its customers are choosing something else then its time for some hard reflection. This is not just an ideal, but a hard requirement, and something that a lot of orgs just don't have.
So, what do I need as a developer? I can make excellent use of open source libraries within even a couple of minutes, but somehow for my enterprise platform I need to fire up a request and wait for weeks or months to even get to play with it. When I need a tiny little change I can't do anything myself, I need to request it and it is filed in the backlog. The same change, would I have composed it out of open source libraries or had cloud access myself, I could make in minutes. Now it takes days, weeks and I even experienced many months of waiting for a simple task I can do in 5 minutes. Thus, making use of the platform meant fighting for higher cloud privileges so we could do it ourselves and ship at least within a couple of month instead of a year, and dancing around bizarre compliance regulations, mostly in order to either evade or pleasing a chain of risk owners to satisfy audits required for certification. At some point we became almost incapable of shipping.
Not every platform team is as kafkaesque as this though, and it doesn't need to be.
We tried inner sourcing as well and it was quite hard honestly, because each team was just focused on their own goals and contributing to the shared platform or libraries was often an extra investment that defacto penalized their achievements - which _did_ have repercussions on their individual performance reviews. Furthermore, it was done in such an ad-hoc way that it was quite hard to get something sane off the ground. I think you do need a kind of dedicated ownership.
Best experience was in a team where we did everything ourselves and had the required expertise. Second-best was in really close collaboration with a platform team where we also had members going from one team to the other. But even that team failed to build the features we really needed as a product team.
Open source is a good model, and there is one essential thing that platform teams need to provide their users that open source has: autonomy. I have actually never seen this in a platform team in the org I am talking about.
I can use a library, drop it, change it or exchange it for another one. The same _must_ be possible with a platform. It needs to be something that helps its users, not limits them in any way, and it can't ever be rammed down their throats. If your platform doesn't work for me, I should have the autonomy to just use a third party to get the job done. If your magic abstraction over AWS doesn't work, give me an AWS account with admin access and I do it myself. Bonus points if its not an all-or-nothing and I can just use what works and build the rest myself.
If you don't have time to build feature B because feature A is more important, everything I need to build feature B myself should be readily available. For example, I must be able to just fork the platform, build feature B, and when done 'upstream' it to the platform again.
Theoretically yes, but since they weren't directly responsible for customer interactions, they couldn't do much more than coordinate with product PMs. Which wouldn't work if two product PMs for two different products had fundamentally different views of what is more important for the business.
I love team topologies, it's really good. My one problem with it is that it kind of stops at the level of structuring a team and doesn't do much about the higher level of management of the organisation. Which needs a deep look just as much as how you structure your teams and if you have platform teams and whatnot. Hell, you could argue that the main reason why agile failed, was that it never addressed how to challenge and change upper-ish management.
But its understandable, cause that shit is hard, its political, and its difficult with the current feudal entities that companies are
There's a lot that I like about this book. Splitting up the mandate between platform and products teams, and eliminating friction, letting each team be good at their thing is I think an efficiency many companies indeed could benefit from.
But I've also seen this book promoted heavily within an org, and the one core strength kept feeling like a core weakness that mades me incredibly sad, about how isolated it made work.
It doesn't insist it has to be so, but the org I saw that was so excited for TEam Topologies loved how it continually stressed independence of teams. And as a direct result, I've seen cooperation, coordination, & cross-team planning plummet. In ways that keep having suboptimal plans get put into action with bad outcomes. Stream aligned teams would turn into complicated subsystem teams, after they created something complicated and gnarly while being stream/product aligned, and unchecked.
I think the topologies here are quite useful framings, and as Fowler says the idea of trying to reduce cognitive complexity is an interesting one we haven't heard well represented before. And is probably due given how impractical making each team truly full stack devops has become, as the CI/CD/observability stack complexity has expanded. But I caution so much against the messages this book gives management, which is that stream/product aligned teams just need to be racehorses with blinders on & interference is bad. The book glories the stream aligned team, the value creator, and makes everyone else auxiliary, which is sort of true & great. But the book's glorification of execution speed doesn't leave much space for how and where cross-team wisdom happens, what kind of processes you have there. Broader cross-team architecture reviews, brainstorming, systems planning, systems coordination aren't well captured here: most teams need a strong collaboration mode to build good software that fits the architecture well. But the book only really regards a single collaboration need: those of platform teams to get feedback to ease their Developer's Experience.
The missing element was ubuntu. If you want to go fast, go alone. If you want to go far, go together. - African Proverb
How big was your company? I think beyond a certain size, independence of the team is the prize. You are not alone, for you have your mates. Beyond your team, trying to collaborate is like herding cats. Every team has its own priorities, everything moves more slowly, and the more stakeholders there are, the greater the risk of the project falling through. Bezos' stipulation that teams communicate through interfaces was a stroke of genius. This created a standard which allowed teams to self serve.
I agree that teams must collaborate to go farther. One team can only do so much. Management should make sure all the teams stay aligned and hold them all accountable, but the teams themselves should still strive to be independent.
It sounds like what your organization needed was a project manager to co-ordinate or a forum for the teams to share information.
Every large company I've worked at on implementing the ideas from this book that was successful put a lot of effort into supporting the chapters/guilds/communities of practice or whatever you want to call them that encouraged this cross-team collaboration. Sure every team had their DevSecOps person, but that person was also a member of the chapter that met regularly and maintained an active channel on Slack or similar to help each other out. Along with a standard framework for picking new things to minimise the sprawl of tooling where possible.
I’ve experienced something akin to the isolation you experienced in a relatively mid sized (about 300 devs) org. While team isolation made it possible to move fast, there was a lot of chaos and duplicated effort all over the place, lessons learned was promptly forgotten etc.
I think what made it kinda work is when they introduced “infra” dev teams that tried to see what the global problems were and solve them on a library/infra level.
Networking was one terraform module away, kafka integration was solved with in house abstractions, design systems, knowledge bases, etc. While not perfect there was a sense that “if its too gnarly a solution, ask around, somebody probably already solved it more elegantly”.
They would talk to all the teams and if someone developed an elegant package/lib/idea they would promote it to other teams.
Key was to have people working explicitly on tech sharing and global problem solving. Ended up quite a nice team environment by the time I was leaving the company, though it took years to get to that point.
The book was good and when I read it, it was helpful as it came out just as I was going from manager of a team to having to split that team and take over a few more in a group.
Plus Manuel, one of the authors is a really friendly guy that pushed the Lisbon tech meetup scene forward around 10 years ago when nobody was having any regular local meetups and he was traveling between Spain and Portugal on the regular to make them happen (or so I remember). I presented my first proper talk in one of his meetups and I remember that going well and giving me a lot of confidence for other things later on. Definitely read it if you need to think about groups of teams!
My problem with Fowler is his tendency to reduce complex problems into concepts that can be represented in simple, abstract models/diagrams with weak analogies (e.g. the "streams" here, test pyramids, microservices, various bits of Agile), which are enthusiastically consumed by both inexperienced engineers and decision makers in management: a deadly combination.
When much of the industry adopts these ideas, and the resulting inefficiencies become the baseline, it is virtually impossible for anyone to effectively detect those inefficiencies. It becomes a "is everyone crazy, or am I crazy?" situation, until some sort of disruption (e.g. widespread project/startup failures) forces the industry to paradigm shift back to sanity.
This is a solid book, but there's one concept in it that has drastically changed how I lead and collaborate across teams.
The book describes 3 types of problems/areas: simple, complicated, and complex. The definitions of simple/complicated/complex has shifted for me a bit since I read it, but this is how I currently describe it:
Simple - the solution is clear and is fairly low risk
Complicated - the problem is well framed, but the solution isn't clear
Complex - the problem itself isn't really clear
This 3 tiered framing lends itself to so many useful structures and interactions. Junior/Senior/Staff scope of responsibility fits remarkably well. Sprint planning / Quarterly planning / Executive planning fits.
The biggest thing it has helped with is recognizing that a team needs space to explore complex problems. Those teams need to operate very differently from a standard product stream team and recognizing that need is such an important thing.
I was about to give some feedback and quote my favourite author Jorge Luis Borges, but then I realised Martin has already done something similar:
George Box neatly quipped: "all models are wrong, some are useful". Thus Team Topologies is wrong: complex organizations cannot be simply broken down into just four kinds of teams and three kinds of interactions. But constraints like this are what makes a model useful. Team Topologies is a tool that impels people to evolve their organization into a more effective way of operating, one that allows stream-aligned teams to maximize their flow by lightening their cognitive load.
So I am thankful that there are people like Martin Fowler who are willing to write up and remind some of us, who have perhaps forgotten more than others have learnt, that we may continue to avoid the consequences of that aphorism: “Those who cannot remember the past are condemned to repeat it.”
God I hate the phrase "high performance teams". Its one of those buzzwords, that doesn't have a clear meaning, but most of the time it turns into "people that deliver a lot but also work a lot of overtime at a killer pace". Most organisations don't even have a clear picture of what performance means, so it defaults to deliver more and work more ...
In the article he identified the authors are moving the platform concept to a "grouping" instead of a team. This is a very recent development that mostly only people who are active in the community know about.
Well when the whole world is reading, you only publish what is really "solid" ... and that means ... not that ground-breaking. So I think you are right, he is catching up. Just I think the motivation is more "do not screw my reputation/company up" instead of the "out of the game too long".
Readit News
Almost everything he hawks causes misery for software developers, like his whole multi-year spin for Microservices, he's turned refactoring into a fetish (honestly there are people at my dayjob who have been "refactoring" the entire 3 years i've been here) and now hes blogged multiple times about platform teams - which - having been on one - don't work. I see him as a very damaging character.
Well intentioned, i'm sure, but I don't think he has good ideas.
Most of my nightmares with microservices comes from where there has been > 1 per team. That came from people who read his "microservices are great!" blog post and just thought that they should build as many as possible. I used to hate him because of that - because it was a logical response to his blog post and people would use it to justify their technical decisions - with argument-from-authority.
In another company I worked at each team maintained ONE microservice and it worked really well.
Because it was a corporate environment with all the usual power politics, control freakery and lack of effective inter-team communication, even though it was more work overall and definitely a technically inferior solution, it did a neat end run around those systemic corporate issues by giving teams relative freedom of development, freedom of deployment and a deliberately simple, loosely coupled, standardized JSON interface to microservices run by other teams whom they didn't talk to much.
So, I get it, but I lost a lot of respect for him over his inability to parameterize his suggestions and the fact that he didn't seem to initially really understand why microservices worked so well for him.
Micro services are an attempted solution for having highly bespoke infrastructure that required difficult, long, manual, fragile deployments that were high risk, and thus didn’t happen often and delayed business value. The problem was that they saw that issue and thought it was monolithic applications, not just having bad infrastructure. 99% of the benefits of devops with almost none of the drawbacks are done by having all of the standard dev ops stuff like Cicd, containers, software defined infra, etc and just doing it on the normal monolith.
Eventually you do need to break things up, but the micro services single functionality, single team per service type of micro services goes way to far and harms more than it helps.
To me, it's the Clean Code of web services.
But its a bit like a health influencer telling people to stay hydrated and blame him for his fans damaging themselves by trying to drink 10 liters a day.
On the other hand I don't know that having people dedicated to refactoring is necessarily ridiculous. At least "in theory, given unlimited resources".
That's kinda dumb, but the book is super good, I learned a whole lot from it, and apply the principles basically all the time.
He's been banging the drum on testing for longer, and that makes up for a lot of craziness (I'm with you on microservcies, except where you have 1k+ developers where they might make sense).
I’d love to hear more about your experience with a platform team and why you say they don’t work.
- Business hates and underfunds these teams because they are often blamed as being a blocker because of the previous statement. They are a pure cost center too. Career suicide joining them.
- Unless you have a very motivated team lead you will end up doing the work of the most aggressive and loud team leads from other teams. Anyone good at playing politics will dictate your roadmap.
- Given the first statement, how do you hire someone for these teams? If they are responsible for platform evolution, upgrades, CI + CD, making frameworks, writing scripts, improving developer experience PLUS you are the backstop for the full backlog of every other team so need a lot of domain expertise. You would need to pay someone a lot to do this right? Well no - because of point 2.
The idea of a platform team is one of those weird things that sounds obvious and clearly needed - until you marry the idea to that of any org that has layers of management + politics and a misunderstanding of the long tail effects of platform work.
An interesting thing that happened at our company when we ended up having a platform team was that the "Can do" attitude of our previously homogenous group of mobile devs stopped over night - "That's a platform role" became a common refrain.
I think the issue is a lot of people read his stuff and then blindly think it all applies to them and microservice themselves into a corner without actually thinking through if it actually applies to their situation. Same goes for various blogs from FAANG companies about scaling. Hardly anyone has scaling issues that they have, yet technologists like the bright and shiny new thing so they end up adopting over complicated solutions to their simple problems.
The team types he talks about in this post I see working well in some companies and situations but he's not saying "thou shall have a platform team and it solves all your woes". The first paragraph states the primary team is a "stream aligned team" which I take to mean a product team that is responsible for app(s) from top to bottom - UX, UI, backend, scaling, support, etc.
What your employees and customers observe and perceive is the reality, not what Google, Martin Fowler or Bill Gates prescribe.
Have any openings?
I'm not sure I completely understand "platform" in this context. I've worked on teams that tried to write a common platform, then delegate to more specialized teams for specific implementations.
I have often hated working under these conditions. These problems are often true for software suppliers whether internal and external. There is a disconnect, and weirdly platform is not motivated to provide good quality code with whatever analysis / testing is needed. It can be even worse if the supplier is internal, and it's known that the company doesn't want to use someone external (where's your leverage!)... good chance platform has more political pull than you, too!
Often times they just ship it and say "your problem". Meanwhile, they also insist that we do not modify their code, or else take full responsibility. There's more motivation to finger point, or push responsibility on to your customer (including other internal teams).
Mind you that in my industry this all pushes the limits of ethics, and maybe our obligations dictated by regulations. Of course, we do everything we can to meet our obligations and do the right thing. However, you're left with the decision to leave in poor quality code (or missing work products), or modify / test code you probably will struggle to understand. It will also be difficult to understand the implications, eg. for other modules.
It's a very awkward situation.
In my experience, it's worked out terribly when a company treats a platform team as a catch-all for any back-end service regardless the business domain. On the other hand, it's seemed to work reasonably well when the team has a narrow (and very clear definition) of which services they own and why.
That said, I've never worked directly within a platform team as an engineer, so maybe it just appears that way as an outsider.
I'd would not call it "virtuous cycle" more of vicious cycle.
Well intentioned I doubt. Unless it means McKinsey style well intentioned: hammering about re-orgs and reforms while generating ton of consulting revenue with it.
Be careful around software evangelists
Side point here…
I absolutely love refactoring sometimes. Like taking a messy chunk of code with deeply nested conditionals a d simplifying it, which makes it more readable and extendable. You also have very clear requirements as well (whatever the original code did). There are few things in software development that bring me more joy than that.
But its often not necessary. And furthermore, refactoring is often not of that type. Sometimes people rewrite things because they don’t understand how they work (and writing is easier than reading). Or because they completely misjudge the value of the refactor.
Classic "my experience trumps yours".
There in lies the rub, the majority of engineers don't work at companies that need the kind of scale that Facebook has.
The majority of us work at smaller shops that can get by fine without all the overhead that microservices introduce. The problem that I see is that there are to many folks not weighing the pros and cons of the architectural decisions they are making, and are just joining the cargo cult.
On a small-medium vertical team at a large company? Probably-maybe not.
Many, many engineers (and managers maybe even more so) immediately jump to micro-services (often resume-driven development) where most companies have poor tooling and poor security knowledge that turn them into a living productivity-crippling nightmare to work with.
If that were true, then moving to a microservice architecture they'd need 5 million instances of the microservice that replaces the part of the monolith that required 5 million instances in the first place, plus a couple of millies for all the other services.
I generally hate the term “micro services”. They are just “services” and nothing new.
The same people are now saying creating microservices is a sign of incompetence and the smart thing to do is create monoliths.
But not to sound ridiculous, they bring a spin to it: they now say they create modules within the monolith.
Software engineering is like fashion. It goes full circle every X years.
What we have found is that platform teams are a huge point of frustration for product teams, since they cause unnecessary and unmanageable coupling between different business divisions, leading to impossible to balance priorities. What we've seen happen over and over is that product A needs Feature A, and product B needs Feature B, and they both need it tomorrow, and the platform team only has resources for one. And since Product A and Product B are in different business divisions, you end up needing to involve a senior VP or even an officer to make a prioritization decision for a simple software feature, and everyone gets frustrated by the process.
What we're striving towards instead is an "inner source" model, where a platform is collaboratively developped and maintained by multiple product teams. Each product team is then empowered to build new features as needed into the platform, and others can reuse them.
Of course, the platform needs some architects and overall review, and all teams will not equally participate. But the key point is to encourage this collaboration mode, where no one team is a bottleneck for multiple products.
The inspiration for this structure is obviously the open-source world, where multiple corporations and foundations collaborate on a single code base without needing to rely on a single platform vendor to provide features for them.
I have seen this fail (once myself and once through coworker experiences) as well, IMO the best approach is to have as little shared infrastructure between teams as possible even if it means more work in the end
That's the route to shipping features faster for sure, but it costs more, requires duplication of effort, and is really painful when you discover two teams built the same thing and you want to unify on one version rather than paying to run and maintain both. Plus, when the cost cutting comes in the bad times, you find whole teams are axed because you don't need two sets of people doing the same job. The remaining engineers then have to unify the features but without the resources to get the work done and without access to the institutional knowledge required to understand the version they didn't build themselves.
If you're an engineer in a FAANG it's not a bad solution to the shipping issue. If you're anywhere else it's a road to absolute chaos.
Based on my experiences, the idea that a genuine bottleneck in tech is being made visible to customer division management and senior management, is generally an authentic way to ensure realism creeps back into the prioritisation and funding processes.
Either you put too many in there, and they are basically idle the time in between. Or you put a limited team, and once even a single team requests something they will have to wait for the developments, let alone 2 requests at the same time.
The OP describes an excellent solution: the team that needs the feature, pays for it in man-hours. In fact, those developers that need it, are pretty familiar with using it, so most of the time capable of doing some developments on it. They can also schedule in the work in their project. The teams that will need it afterwards will get it for free.
It's actually an excellent solution.
Additionally, this overall means that no one can plan in advance any features that depend on the platform, since global business priorities are constantly shifting.
Not to mention the political angle, where more adept business leaders can always push through their pet platform features to the detriment of what are more obvious business goals. Or at least, it will always seem this way to teams whose requirements keep falling through the cracks.
If a platform has shared ownership, then decisions will get implemented by cohorts only thinking for themselves and thus damaging the long-term roadmap. All systems, especially complex ones, need sole owners or they will devolve into what is essentially a pyramid of doom at the product level.
Good communication across teams and good design (i.e. flexible, maintainable, extensible) tends to alleviate the tunnel vision associated with each team building what works for them.
I've had to wait weeks for one-line changes because somebody else had to do them.
That seems like an unnecessary coupling between different products.
I understand this concept of a platform team mainly within one (larger) product. But maybe it's the same thing, just on a different scale.
> What we're striving towards instead is an "inner source" model, where a platform is collaboratively developped and maintained by multiple product teams. Each product team is then empowered to build new features as needed into the platform, and others can reuse them.
I think it breaks down at a certain point of complexity. The platform itself is complex enough that it's very difficult for product teams to understand it holistically. I've seen this multiple times when the product teams were still able to make the changes as needed, but over the long term this approach created a hot mess of platform features which didn't align to each other and no single person understood.
Not sure I'd like to use a platform with a code base that has the whole organisation contributing to it. Sounds like a design-by-committee dumpster fire. Teams need ownership of their product to perform effectively. You can claim that this is collective ownership of the platform, but in reality no one will feel any responsibility for the platform when things don't work or don't align entirely with the particular thing they are currently working on.
And have you ever looked at the source code for most open source software? It's an absolute mess that requires the collective effort of large numbers of people to keep it going. It works, but it's not exactly a good or effective model for internal software.
Related to collective dev/ownership, that is a risk, I absolutely agree. That is going to be the hardest part for sure, maintaining a uniform standard and a high bar for contributions to a common platform.
Finally, I believe the general opinion is that open-source software is higher quality and especially has higher code quality than internal company software. I can't say I studied any directly comparable open-source projects VS our internal tools, but code quality definitely varies significantly between our own internal projects, with some extremely clean and well written, while others are messes of hasty bug fixes and accumulated tech debt.
Sometimes the platform team is staffed by people that have been there forever, as a sort of semi-promotion. But when they know everything, it's easy to have little interest in the difficulties of learning internal concepts: After all, the learning has already been done. This makes the tools be technically capable, and intractable. Other times, the team is easy to get along with, but what they deliver isn't very good at all, and the lack of quality is papered with social skills.
You need people capable of understanding the problem other teams have, and their architectural constraints, and deliver something that will save them time, and they'll prefer to use over some open sourced hodgepodge. They need to think of upgrade paths, or live in a low-repo environment where the platform team can upgrade things for everyone. The customer service attitude should be immaculate, as to make people be happy to ask for help, yet be so good at documentation, or at simple enough architecture, as to make that customer service load be light. Many places can't hire people that meet those kinds of profiles at all, as someone like that will basically excel in most roles in most companies. So yes, you end up with the technically capable, yet gruff guys that nobody wants to talk to: The equivalent of Seinfeld's Soup Nazi... and that's if at least they are very good.
Most team topologies will work if your company is full of empathetic heroes though, so platform teams might not even be needed if you really are that good at hiring.
What I've seen is that platform teams work well in either small scale (because focus is clear to all) and large scale (because of this internal competitive dynamic) but are extremely hard to execute between the two.
Is this any different from a product team where there's competing demands between Customer A and Customer B?
TLDR; platform teams without product team's freedom to deviate (optionality) is corrupt and can destroy a large chunk of engineering velocity.
> Platform teams, however, need to build their services as products themselves, with a deep understanding of their customer's needs.
If there is no malfunction like a monopoly or a scam, as a customer I just choose a different product if one is not meeting my needs. The same must be true of a platform, and if it happens a lot that its customers are choosing something else then its time for some hard reflection. This is not just an ideal, but a hard requirement, and something that a lot of orgs just don't have.
So, what do I need as a developer? I can make excellent use of open source libraries within even a couple of minutes, but somehow for my enterprise platform I need to fire up a request and wait for weeks or months to even get to play with it. When I need a tiny little change I can't do anything myself, I need to request it and it is filed in the backlog. The same change, would I have composed it out of open source libraries or had cloud access myself, I could make in minutes. Now it takes days, weeks and I even experienced many months of waiting for a simple task I can do in 5 minutes. Thus, making use of the platform meant fighting for higher cloud privileges so we could do it ourselves and ship at least within a couple of month instead of a year, and dancing around bizarre compliance regulations, mostly in order to either evade or pleasing a chain of risk owners to satisfy audits required for certification. At some point we became almost incapable of shipping.
Not every platform team is as kafkaesque as this though, and it doesn't need to be.
We tried inner sourcing as well and it was quite hard honestly, because each team was just focused on their own goals and contributing to the shared platform or libraries was often an extra investment that defacto penalized their achievements - which _did_ have repercussions on their individual performance reviews. Furthermore, it was done in such an ad-hoc way that it was quite hard to get something sane off the ground. I think you do need a kind of dedicated ownership.
Best experience was in a team where we did everything ourselves and had the required expertise. Second-best was in really close collaboration with a platform team where we also had members going from one team to the other. But even that team failed to build the features we really needed as a product team.
Open source is a good model, and there is one essential thing that platform teams need to provide their users that open source has: autonomy. I have actually never seen this in a platform team in the org I am talking about.
I can use a library, drop it, change it or exchange it for another one. The same _must_ be possible with a platform. It needs to be something that helps its users, not limits them in any way, and it can't ever be rammed down their throats. If your platform doesn't work for me, I should have the autonomy to just use a third party to get the job done. If your magic abstraction over AWS doesn't work, give me an AWS account with admin access and I do it myself. Bonus points if its not an all-or-nothing and I can just use what works and build the rest myself.
If you don't have time to build feature B because feature A is more important, everything I need to build feature B myself should be readily available. For example, I must be able to just fork the platform, build feature B, and when done 'upstream' it to the platform again.
But its understandable, cause that shit is hard, its political, and its difficult with the current feudal entities that companies are
But I've also seen this book promoted heavily within an org, and the one core strength kept feeling like a core weakness that mades me incredibly sad, about how isolated it made work.
It doesn't insist it has to be so, but the org I saw that was so excited for TEam Topologies loved how it continually stressed independence of teams. And as a direct result, I've seen cooperation, coordination, & cross-team planning plummet. In ways that keep having suboptimal plans get put into action with bad outcomes. Stream aligned teams would turn into complicated subsystem teams, after they created something complicated and gnarly while being stream/product aligned, and unchecked.
I think the topologies here are quite useful framings, and as Fowler says the idea of trying to reduce cognitive complexity is an interesting one we haven't heard well represented before. And is probably due given how impractical making each team truly full stack devops has become, as the CI/CD/observability stack complexity has expanded. But I caution so much against the messages this book gives management, which is that stream/product aligned teams just need to be racehorses with blinders on & interference is bad. The book glories the stream aligned team, the value creator, and makes everyone else auxiliary, which is sort of true & great. But the book's glorification of execution speed doesn't leave much space for how and where cross-team wisdom happens, what kind of processes you have there. Broader cross-team architecture reviews, brainstorming, systems planning, systems coordination aren't well captured here: most teams need a strong collaboration mode to build good software that fits the architecture well. But the book only really regards a single collaboration need: those of platform teams to get feedback to ease their Developer's Experience.
The missing element was ubuntu. If you want to go fast, go alone. If you want to go far, go together. - African Proverb
I agree that teams must collaborate to go farther. One team can only do so much. Management should make sure all the teams stay aligned and hold them all accountable, but the teams themselves should still strive to be independent.
It sounds like what your organization needed was a project manager to co-ordinate or a forum for the teams to share information.
I think what made it kinda work is when they introduced “infra” dev teams that tried to see what the global problems were and solve them on a library/infra level.
Networking was one terraform module away, kafka integration was solved with in house abstractions, design systems, knowledge bases, etc. While not perfect there was a sense that “if its too gnarly a solution, ask around, somebody probably already solved it more elegantly”.
They would talk to all the teams and if someone developed an elegant package/lib/idea they would promote it to other teams.
Key was to have people working explicitly on tech sharing and global problem solving. Ended up quite a nice team environment by the time I was leaving the company, though it took years to get to that point.
Plus Manuel, one of the authors is a really friendly guy that pushed the Lisbon tech meetup scene forward around 10 years ago when nobody was having any regular local meetups and he was traveling between Spain and Portugal on the regular to make them happen (or so I remember). I presented my first proper talk in one of his meetups and I remember that going well and giving me a lot of confidence for other things later on. Definitely read it if you need to think about groups of teams!
When much of the industry adopts these ideas, and the resulting inefficiencies become the baseline, it is virtually impossible for anyone to effectively detect those inefficiencies. It becomes a "is everyone crazy, or am I crazy?" situation, until some sort of disruption (e.g. widespread project/startup failures) forces the industry to paradigm shift back to sanity.
The book describes 3 types of problems/areas: simple, complicated, and complex. The definitions of simple/complicated/complex has shifted for me a bit since I read it, but this is how I currently describe it:
Simple - the solution is clear and is fairly low risk
Complicated - the problem is well framed, but the solution isn't clear
Complex - the problem itself isn't really clear
This 3 tiered framing lends itself to so many useful structures and interactions. Junior/Senior/Staff scope of responsibility fits remarkably well. Sprint planning / Quarterly planning / Executive planning fits.
The biggest thing it has helped with is recognizing that a team needs space to explore complex problems. Those teams need to operate very differently from a standard product stream team and recognizing that need is such an important thing.
It feels like he's only catching up with what many have been doing for decades building high performance teams.
On the other side: You can be perfectly right ;)