The tech lead started out by complaining about how he has to do an untested unplanned release today because another team made some urgent changes. He's the only person who knows how to release it. The other team didn't communicate until today that a release is necessary.
We've done two other releases in the past month and both required a day of troubleshooting to fix issues.
Both of us have been working at this company for about 3 years and we both have over a decade of experience in software development.
When he finished complaining, I started asking questions and making suggestions about how we can improve things.
- Push back on the team that needs these urgent changes. Let them learn to do the release.
- Deny the release since they didn't communicate earlier.
- Improve the release process.
Everything I suggested was just flatly denied as impossible.
- The other team doesn't know how to do the release.
- He wants to be a "team player" so he can't deny the release.
- Project managers will never allocate time to improve the release process.
I feel strange because I've seen this same thing for my whole career and I still try fight for what's right when others appear to moan and carry on.
However, my experience tells me that bringing this stuff to my manager is even worse. My manager doesn't know anything about the code, my project, or the release project. He may assume it's complaining for the sake of complaining. It has been used as ammunition in reviews against me.
Learned helplessness sucks and I wish I could do more. I don't think either of the suggestions in the article are feasible for many ICs. Teams are ambivalent to making improvements, and retrospectives carry very little weight. Managers are above the fray and won't be held responsible for by people below them.
The tech lead started out by complaining about...When he finished complaining...
My manager doesn't know anything about the code, my project, or the release project.
IOW, your team has no leadership and no management. Your team lead may be called that, but sounds like he doesn't lead at all. Your manager may be called that, but he doesn't manage either. This happens a lot when people are given responsibility without any authority (probably the case for you "team lead"), or authority over things they don't understand (probably the case for your "manager").
The bad news is that you're part of a disorganized mob. The good news is that there is a leadership vacuum that, if you play your cards right, you could fill. If you want. Of course, even if you succeed, it's possible no one will care about your accomplishments. And you will probably make enemies. It's possible you'll be rewarded but given your description, I view that as unlikely.
As a side note, suggesting to the team lead that he lead and do X, Y or Z may appear to him as if you want him to take on risks with little upside. He's probably thought about doing those things anyway but just decided it's not worth the trouble. It's not all that surprising that he doesn't want to rock the boat.
> The good news is that there is a leadership vacuum that, if you play your cards right, you could fill. If you want.
Perhaps. I thought so once in exactly this situation, and it led to an incredible amount of frustration, and burnout. And others were hurt the same way. That leadership vacuum was very real, but nobody wanted it to be filled by anyone already in the company, it turned out.
Eventually we got a very competent new team lead from outside and he was accepted by everyone immediately. For one of the teams anyway; the other one is still a complete mess. I just make sure I'm on the right one.
Unfortunately this. I really like how you have articulated the problem. Disorganized mob. This mob also loves moving from one team to another inside the company once they deem to think they exhaust the resources in the former and once the product is dead. I think it happens sooner or later in enterprise. It sucks.
I've worked on a good number of teams just like yours. I always told myself things would be different when I was in charge. Then I was in charge.
I learned pretty early on that I have political capital, and very little of it. Instead, I have to have a manager that's pretty buck wild when I tell them I need them to be, and I have to deliver them some wins that will grease their wheels too. Good managers are hard to come by though, and if they get noticed they're promoted and gone in a couple of years.
Personally, I think the transactional nature of software positions is what largely holds us back. If I get promoted, then there'll be a new lead, and this cycle will start all over again.
This hit so close to home for me. At my last two jobs, I've simply burnt out from hitting this sort of wall over and over.
Everything I suggested was just flatly
denied as impossible
[...]
I feel strange because I've seen this same thing
for my whole career and I still try fight for
what's right when others appear to moan and carry on.
For anybody (100% rightfully!) wondering if it's my fault: (1) I've been consistently praised in reviews as a good communicator (2) I always operate by the maxim "don't just complain -- instead, offer solutions" and I see that you do too. (edit: That's not to say that I couldn't be communicating things better. Certainly, I don't believe I've reached some level of perfection there)
I think the reality is that this kind of change can't originate from the in-the-trenches folks at the bottom of totem pole. Management will never let individuals waver from their short term goals.
The only times I've seen developer pain-points successfully addressed in a sustainable way, it's because there was a dedicated team allocated to that sort of thing: a "developer experience" team, or some equivalent.
(Although, the issues you ran into are kind of an org issue in addition to a developer experience issue)
>The only times I've seen developer pain-points successfully addressed in a sustainable way, it's because there was a dedicated team allocated to that sort of thing: a "developer experience" team, or some equivalent.
I came to this conclusion after watching the place I work at for 2 and a half years fail to implement any of the grand ambitions management had in their heads. We wanted code review, pull requests, build pipelines, automated releases, updated OS's, logging, disaster recovery, you name it. No one had the experience, time, or energy to implement it, so it falls on one or two juniors (me) to quickly learn and implement while learning. No one is hired with experience in any of these topics, only fresh-grads that will replicate the garbage you're trying to get away from are hired, because they will on paper be producing more features for less financial cost.
>I think the reality is that this kind of change can't originate from the in-the-trenches folks at the bottom of totem pole.
I'm a manager. Please. PLEASE don't believe this. In-the-trenches folks ARE powerful. Really. I promise. I can't implement change if YOU don't do it. I don't know what sucks if YOU don't tell me.
This is why I .. struggle to function in society. There are tribal forces at play that are beyond me. Or require me to play games I don't want to play.
Social tissue is a strange medium and most of the time it's a friction generating engine. People complain, time passes on, nothing happens, repeat.
None of these are games: they're really just an expression of the energy you need to spend to change a rigid structure. And the more rigid, the more energy, just like a metal bar you'd be trying to fold.
You can change things but you must do it. You must put your reputation at risk, expense time that you wont be compensated for, go through conflicts you'd rather avoid, make lifelong enemies wether you succeed or not. Then, you changed the rigid structure once more and the next guy will either try to put it back where it was and face less resistance or fold it one more time and face even more.
You cannot teach a rigid structure agility just like you cannot teach iron to be silk, and the structure is rigid for a reason: size, cost, regulation, cultural beliefs, rot due to time, past mistakes (in my experience the largest factor: a mistake 10 years ago can justify 3 days a week of useless processes today) etc. You can change the structure but it's CEO-level work to change its rigidity.
Ironically, the only way to minimize the "game playing" is by mastering them. There will always be people out there whose only real skill is manipulating the people around them, and who will deploy that skill at full force.
I'm not saying you should manipulate the people around you... but the twin powers of, "understanding whats going on around you" and "documenting everything", go a long way.
I left my last contracting project around six weeks ago because of encountering this attitude - yet again.
I wrote my first custom business program 40 years ago. Since then, I've only been on two teams that didn't exhibit this attitude.
At this point, I really don't want to continue programming for a living, precisely because so many people adamantly refuse to fix problems with painfully obvious (and quick and cheap) solutions.
I've given up on trying to get people to change their behavior. If I do continue programming, my approach will be to try to find one of those rare teams that actually changes and improves.
A hazard of meetings is that the best debater owns the show. It's quite possible that the tech lead thought his idea was straightforward enough that he didn't expect it to be debated, and wasn't prepared for it.
I don't introduce new information at meetings unless I am ready to back it up, which usually takes an hour of preparation for every hour of meeting. If time is of the essence, it's often better to settle things in hallway conversations or via web chat.
> that bringing this stuff to my manager is even worse ...
This.
While I currently have a manager that mostly understands such complaints, I have learned that even then, it amounts to nothing. Because he can't change material things without invoking higher-ups, and they won't be brought on board. For one thing because that would require a lower level manager to create tasks for a higher manager. And that's not how management works in many companies: A lower level manager didn't get to be a lower level manager by creating "work" for higher-ups. He got to be a lower level manager by absorbing the complaints from the ground force, assuring them that their voice is heard while at the same time shielding the higher-ups.
Maybe I've just been in sh*tty IT companies all my life, but I'm seeing this, and similar pattern for nearly 30 years now, and that's why I'm now searching for a job that is as far away from the potential to create overbearing complexity as possible. Even to be point where salary be damned.
You don’t know other aspects about his job. If the work is easy and the pay is good there’s no point in quitting unless you aren’t working from home or there’s a substantially better offer.
Can't your tech lead suggest that he'll do the release under the condition that someone from that other team pairs with him so that they learn how to do it?
Yeah, at a previous job, I had inherited responsibility for maintaining a particular poorly tested (and hard to test—lots of API dependencies), widely-used library. Most changes were contributed by users and I managed by reviewing new code and tests as carefully as I could.
At one point, one particularly hard-pressed user wanted to make really sweeping changes to the library and had limited time. I stalled and stalled to try and review all their changes and convince myself they were going to work, and they got more and more agitated, and my TL (who was himself frustrated by how much of my time this was consuming) told them "you can merge whatever you want if you take over maintaining it". So that's what happened, and everybody left happy.
I regularly re-read the opening few pages of the Pragmatic Programmer as a mantra to myself, because what you have written here is very relevant.
> It is your life. You own it. You run it. You create it.
> Many developers we talk to are frustrated. Their concerns are varied. [...] And the answer we give is always the same.
> "Why can't you change it?"
> But, for some reason, developers seem to resist change. They hunker down, and hope things will get better.
> [...]
> So here's the most important tip in the book.
> Tip 3: You Have Agency
> Does your environment suck? Is your job boring? Try to fix it. But don't try forever. As Martin Fowler says, "You can change your organization, or you can change your organization."
When I read the situation you describe, the best thing would be to acknowledge where the tech lead is emotionally, and sympathize, because it sounds like they simply wanted to sound off about it, not actually change it at all. It's always easy to say no to why something won't work, and it sounds like a classic drama triangle situation, with someone coming in as a persecutor, you stepped in as a potential rescuer, then you became a victim because obviously your suggestions won't work.
You, however, don't need to accept the situation, and you can try to change it if you wish. Too many people (including myself, hence why I always re-read those few lines) fall into the trap of thinking that someone else should make this or that change, make a decision, or that you require approval or authority. This is far less true than people believe.
As someone who has stepped into a software manager role for the first time from being a software engineer for most of my career, my view is that I'm not always there to solve the problem, but to give space for people to solve the problems themselves. It helps if I know what's going on, or understand the issues, but I trust my team, and those I work with, and would rather get them together with the right people to discuss it through, and work out a way forward.
I appreciate I don't know the situation at your organization, the battles you fought, the things that have been said. Change is never easy, and there's always resistance, but you do have agency, as does your tech lead. Never forget it. Good luck.
I’ve been there. It sounds like you’re starting to blame yourself for their failings. You’re not sure if you’re a little crazy or they are. They’re gaslighting you. Get. Out. As. Soon. As. Possible.
If they’re using your valid concerns as issues on performance reviews, then the company values making low quality software, not their employees. Get out of the sausage factory and get into a place that values their employees and product. You’ll be much happier.
For a person of many years of experience your friend is the problem. He has agreed to push untested code which is his responsibility, enabled other teams to work out of schedule and him producing dangerous results. All these without any indication that he sought approval from upper management. Why would you use the word "complaining" when communicating an issue? State the facts and how they diverge from the formal process, flag parts of the process that are problematic due to lack of resources of consistency, put them in an email with a "default" course of action if not answered and without using "I" or "They", use passive voice (tone). Keep the score in writing as an informal calendar linked to the respective communications. Cc the manager. Obtain bliss, people will fret then they will forget.
If the team is disfunctional or the manager bad consider changing work. Not only due to the dysfunction but because a good team and manager is part of ones' mentoring, and it hampers ones' growth. At least now you gain some reflexes that will assist you in the future, if you chose to move on.
>He's the only person who knows how to release it. The other team didn't communicate until today that a release is necessary.
Also these deployment tasks are quite repetitive, it's a big strain, I can understand your pain. I have managed to automate myself out of this, at my current job: most deployment tasks involve the following steps: Commit the change, open a pull request, wait for the CI build to complete and download the build log, extract an image id from the CI build log, put it in some other repository and commit the change. It is possible to automate the first three steps with the github api (if your shop is using github), I have an example here: https://github.com/MoserMichael/githubapitools . The rest of it is easily scriptable.
Most of the time managers are trying to be helpful, and most of the time they do in fact understand quite a lot of the background of a project.
I'd suggest to try to see the situation from their perspective, which you maybe already did, but sometimes this helps understanding a perceived lack of action.
It’s not good if they’re paralyzed by this though. If the manager automatically says no to anything that may contain some element of risk, I will stop telling them.
I have seen this happen (and in a few cases the cause for this to happen as well).
All code (or releases, for that matter) are not created equal. There are some teams which are working on something which is more critical to the business of the company, at that moment, and there are other teams which may be able to absorb some tasks to let the first team focus on what's needed. Not saying that this was the situation with the OP, but this usually falls in the "greater good" category. I have seen organizations where inter-team boundaries are fought over and defended vigorously and others where these are more fluid and accommodative. I am not sure that the firm boundary teams end up doing better work or be happier overall.
What you're doing wrong, very honestly, is you suggest and 2 of your suggestions are losses (denying release).
Dont suggest, make enemies: send emails to both teams saying now it must be like that, lay the plan, pose a meeting to go through the plan and get yelled at. Once you get yelled at by the complainypants because what you tell them to do is too hard, now you can go to your manager and explain him they complain for the sake of complaining. The tech lead will get a earful and either fight you to his grave or admit defeat and let you do your reform grumbling waiting for you to fail.
I do that in a giant bank that cannot move without 10 approvers to everything, and the more these helpless teams fight me the more management put me in their midst to reform them ... and Im just a silly dev nobody and I can tell you the wonders one can do by just bulldozing. There are things to do if you fail eventually, which is to teach the higher ups to enjoy trying and failing, but that requires a bit of weaseling charisma :D
It's a fun game to play, until you realize it's not your job to manage the managers and your co-workers, and that you're underpaid for having that extra headache :)
From my experience you need to play a small political game, no way around it, what you want to do costs money and presents risk. If you believe it is necessary you need to find supporters in the management floor. Convince someone who can actually give you backup and help you get this done. If you can't find anyone like that, then either work on the project underground (risky, more work and less credit for you) or let it go (or leave).
> We've done two other releases in the past month and both required a day of troubleshooting to fix issues.
I feel you, it's especially frustrating when your product has such fragile and bespoke testing that things break on a regular basis. Then, us devs have to spend valuable time just troubleshooting.
Maybe I misunderstand, but if a product breaks regularly under "fragile and bespoke" testing, won't it break just as often, and probably more often, under robust and comprehensive testing? And if the product fails tests, who better to "just" troubleshoot the problems than the developers? (Perhaps you mean things break after nominal testing and subsequent release, but that's just shifting the time when the issues are detected -- the product still has the issues either way. And if the issues will adversely and/or significantly affect the customer, I imagine one's employer would prefer that you set aside your non-troubleshooting valuable time and fix the issues.)
It may be, based on my experience as that tech lead who declines everything, that the tech lead has already thought of all this many times over many years, has tried to implement the required changes, yet gets blocked by higher ups. And it may even be that those higher ups sometimes know it too, and deny him based on their higher ups.
A lot of times learned helplessness is true helplessness with learned resignation.
I don't think it's "learned helplessness". It is a rational, calculated tradeoff decision. In many situations, it is more painful to fix the underlying layers of technical debt, and more time consuming to fight the push back from the various teams that own different parts of the technical stack. It's far easier to just silence the pagers while on pager duty for that week.
It's also caused by misaligned incentives. At most BigCo, people don't get promoted for fixing technical debt. They get promoted for launching shiny new features and products.
Technical debt is all of the forms of self-sabotage that don't have clear metrics associated with them.
If they had clear, unimpeachable metrics, then you could increase your status by fixing them. Since they don't, few people engage with them. Worse, engaging with some of these problems can lose you status, and so tackling them becomes a form of self-sacrifice.
Quite often "technical debt" from an engineer perspective is an asset from the leadership perspective. Make of it what you want, but code which is bad from a customer perspective usually doesn't live until it's legacy.
And sometimes the technical debt cloud disappears after fully understanding the business rules and special cases which were the root cause for the code in question. This often gets overlooked and may (edit) lead to refactoring/rewrite approaches which ultimately fail. As everyone knows, not a rare occurrence.
Oh this hits close to home. At my BigCo I have seen tremendous amount of great tools/product landed in the trash bin, because it was only interesting until somebody get the promotion… Immense amount technical dept and lack of viable business model lead to fast death of great ideas… What a huge waste…
As a member of a core infra/"foundation" team, the biggest drain on my soul is the number of other engineers that are helpless, or never learned how to find solutions on their own. They never search wikis, look for similar posts on internal groups, or even read the error message from the tool that tells them exactly how to fix the problem they're asking about. When the culture has become "google everything", but you can't google for internal tools/tech problems, suddenly folks have no idea how to read error messages, debug a stack trace, or solve any problem without hand-holding from a senior engineer. I've been at BigCo for nine years now, and it has only gotten worse as the size of the company has grown exponentially.
Dealing with the infrastructure is your job. You know the ins and outs. It doesn't seem complicated to you.
For someone who is dealing with the application logic, having to context switch to infrastructure is painful. I have no idea how you have set things up. When I read your documentation I'm just baffled at the sheer amount of complexity. I don't want to deal with it. It's not my full time job. I have other things to worry about. I have no mental space to deal with this headache. I will just ask you so I can move on to do my actual job - which is already taking up over 90% of my mental capacity.
When I setup my own infrastructure, I make it as simple and stupid as possible, precisely because I have no mental slack to deal with all the complexity that I see devops people deal with on a daily basis.
At most jobs I've been to, the complexity I've seen in infrastructure is mind boggling. My intuition is that this complexity is not actually needed if you remove all the cruft and think from first principles about what the actual problem is you are trying to solve.
However, people in devops seem to enjoy building and maintaining this kind of complexity, and even take pride in how everything is orchestrated.
The drain I am talking about is a complete lack of due diligence on behalf of those asking for my help (and my limited time/attention).
When the error message contains a clear message and a URL with step-by-step instructions, and the person in question just pastes the full error message including the URL, and asks "what do I do?"
When a problem can be solved by responding with "please follow the instructions in the error message you sent me".
When someone makes an internal group post asking "how do I do X?" and the pinned post that was right below the submit button contains a summary and link to detailed step-by-step instructions titled "how to X".
> My intuition is that this complexity is not actually needed if you remove all the cruft and think from first principles about what the actual problem is you are trying to solve.
Have you ever worked on infrastructure? This comment is wrong on soo many levels.
Totally agree..
As a rails dev I'm asked to dig into ops.. Not once is the JS team asked to build out the rails code, or fix things in it when it glitches. The api business logic falls down, and thats on me.
But when ops is glitchy, I have to blow half my day faking interest in docker? I mean best case, I learn how ops works, and then what. I become the one they tap when ops is swamped, then before I know it, I'm doing ops.
And I quit, because as much as I love the ops team, I hate the gig. I'd hate to be a designer, or a BA. It's not where I find joy.
So when I hear people say.. Just follow these steps to trouble shoot, I wonder if it would be cool to tell that to our users..
Infrastructure is like application dev in that it’s a bundle of histories of feature work over time and with many contributors, and many customers with competing wants and not enough time to satisfy everyone.
Granted a primary goal should be to have simple APIs/workflows for app devs to use, but complicated insides is as natural as any advanced system.
I am working at company that just won a contract with a BigCo and it has to be one of the most frustrating experiences I have ever had. I was asked to test a configuration change and I was able to demonstrate that it worked locally in 15 minutes. Unfortunately, testing locally isn't enough so now I need to test it in the lab, but BigCo is so compartmentalised that it can weeks even to organise a meeting where all the stakeholder can get together and talk about what needed to be done to get the configuration changed in the lab.
We've had working sessions where the moment there was an issue, the developers at BigCo said that they need to talk to someone on some team and they wrap up the meeting for the day -- even if we had just got started. It's gotten to the point where we will have project managers in the working session -- essentially babysitting -- to stop the devs from just giving up.
The amount of stonewalling I've seen is insane. We were trying to address an issue with our integration and I was told that a particular configuration on the client side was impossible, or maybe that we needed a new license, or perhaps we needed to contract to another vendor and it's going to cost half a million dollars, or whatever other excuses they could come up with.
So I asked them what product BigCo used, I looked up the documentation, I sent an email with 12 steps on how to make the configuration change, I got on a video call with my contact at BigCo (and his boss to make sure he showed up) and I literally walked him through the setting up the configuration.
I just don't understand how you get to the point where that's business as usual.
I know your frustration and pain, but as someone frequently on the other side of the equation trying to seek out answers, it is often difficult-to-impossible.
Just this past week I was working through some infra setup challenges, I had to piece together half a dozen different documents, several README files, and a few archived slack threads. These sources often are outdated or even contradicted each other. Even after all that and some earnest trial and error I still had to ask for help.
It's fine to want to point to the docs, but when you do that you better have good docs you're pointing to. As an infra team, often your "API" is your docs and templates. Searching slack threads and piecing together bad docs isn't a scalable solution for every dev to repeat individually. But neither is pinging an individual with every question. Teams need to invest in good documentation.
No, no. This sounds completely different. You searched out all the information you could, and still had no idea. That’s great (I mean, not great, ideally the necessary information would have been available, but that’s on the people providing the information).
The problem is people that have not searched or tried at all, and then ask their senior to please do their job for them.
I agree with this and would add Slack to the equation. It's just faster to ping a senior engineer on Slack than to spend 5 minutes looking into an issue.
I've also seen new hires burn an entire week because they were afraid to ask a question.
But I think the balance has definitely shifted towards asking too much help...
When I started at BigCo, the recommended policy was to spend two hours trying to solve it yourself, including searching the wiki, debugging code, etc. If you hadn't made any progress in that time period, then ask someone from your own team or make a post in a related internal support group. Only after exhausting that route, and getting no help, should you escalate to the team/oncall that owns the tool/service you are having problems with. It seems that culture has been lost.
This reminds me of a story I've read some time ago, about some monkeys and bananas.
A group of monkeys in a room. In the center of the room is a tall pole with a bunch of bananas suspended from the top.
Every time a monkey tries to reach the bananas it is hit with a torrent of cold water from an overhead shower. Eventually, the monkeys learn that something bad will happen if they climb up the pole so they just sit and don’t even try again.
If a new monkey is added to replace an old monkey, it will of course try to go for the bananas. But the original monkeys will grab and drag it down before the punishment is even triggered. After a while, the monkey gets the message and it stops trying.
Repeat the action of adding a new monkey in place of an old one enough times and you will reach a point where none of the monkeys know why they shouldn't get the bananas but they still won't try.
The story concludes by stating that even if we remove the shower at this point it won't really make a difference. The learned behavior is already deeply ingrained in the culture of the group.
Funny enough, I've seen this type of issue in many human teams/projects. And usually the inertia is far too strong for just a couple of engineering "monkeys" to radically change things if they don't get support from a few management "monkeys".
yep, that's why I called it a story, not a scientific experiment. It could even be read as a kind of anecdote of parable.
While oversimplified, it can serve as a good example on how an initially rational practice could easily became dogma even when the initial stimulus is no longer present.
Part of my day job in the past 2 years has been to consult a client in refactoring/rewriting an old platform. The client's old team is still in place and even if we've added some new people just to try and change the mindset, the inertia of the spaghetti codebase coupled with the client's short and mid term financial goals as well as the "traditions" of the old team has stopped us from enacting any groundbreaking changes.
I guess the moral is that there are many forces at play in this kind of setup and stories like these can help in framing some of them in a fun way.
This article is bashing BigCo while letting move-fast-and-break-things Startup inc. off the hook.
Just because the fires are different every day at Startup Inc. doesn't mean we're not being conditioned to endure suffering.
Case in point, a previous BigCo on call roster that I was required to partake in, sucks, makes you lose will to live, keeps you up at night, nothing documented, on-call phone ringing 5-7 of the nights in the week.
I tried to improve this process and was mildly sucessful even if that was just to get these incidents recorded.
Fast forward to my next job at Startup Inc. and their on call roster, sucks, makes you lose will to live, keeps you up at night, nothing documented, on-call phone ringing less but still often enough and with higher urgency.
The source of the two problems was different most of the time, one being a lack of training and effectively giving children knives.
The other being moving fast and breaking things resulting in things breaking after it was declared that version x.x went out without any issues just like the CEO wanted.
> Hold managers accountable: one of the key responsibilities of leaders is to create positive change for their teams. Once you notice a situation where you think everyone is in learned helplessness mode, make sure to notify your manager and follow-up until the problem is addressed.
"Hey boss, I noticed that Bob and Joe are really helpless at their jobs and that you just kinda watch it happen."
Yeah what could possibly go wrong with that?
I'm not saying the thinking is wrong, but a little idealistic. Not everyone works at BigCo where it is impossible to get fired and you only ever get transferred. Nevermind most of the people in this position are going to have extremely short tenture.
I don't know if that was intended to be snarky, but yes, that's exactly what you should do, and people do in fact do that, and it often does work (although probably not as quickly as you'd like).
How else am I supposed to know Bob and Joe are doing bad work? Should I count their LOC/day? Pore over their code and rely on my own (10 years out of date) opinion of it?
In my humble opinion, there is one (1) good way to measure developer productivity, and that is to ask their teammates, and if you won't tell me your teammate is awful, you can't really complain that I don't act on it. Of course it's possible that you tell me and I don't do anything, but at least then it's my fault and not yours.
It is somewhat meant to be snarky, but then again the original quote is exceedingly naive.
It describes very well the phenomenon but gives advice that I can only see ever working at BigCo. Where your team is huge and if you screw up there's another one down the hall.
The truth is most places that exhibit this level of helplessness often include a small team of 5 or 6, each with plenty of tenure. With no other lateral teams to move to.
Now you want the new guy to walk up to the boss and tell him that his drinking buddies for the past decade are the reason things are falling apart and he's an ineffective manager for not noticing it. You can play with the wording but that's the message.
In the real world we have political factors like seniority, favoritism, nepotism, tenure, and nevermind external factors.
In the real world the boss doesn't actually care if the team is working well or not. He only cares that the perception of the team is positive. If the new guy threatens the perception, he's gotta go. Nobody cares if it gets fixed. If nobody knows it's broken there's no need to fix it.
Let's have less articles like this one where we basically fire ourselves ostensibly for virtue signaling and more articles about how to socially engineer your way to the fucking top. Because let's face it, that's what it actually takes.
If you work in a small company, pretty much any boss will be happy to hear feedback like "Hey, it looks like there's an impediment to smooth work, and the team has gotten used to it, but it's costing us". Bonus points if you propose a solution.(If your manager isn't happy to hear that, it's time to leave, because small companies closing their eyes to this have a tendency to go down in flames)
But you'll definitely need to learn to take the blame out of your communication. Learned helplessness is a technical term, it doesn't mean "Bob and Joe are helpless". And your manager not noticing a problem is hardly "you just kinda watch it happen".
Frame it as a possible improvement to what the team can do - because that's what it is. Yes, you might get turned down. And if you repeatedly get turned down, you have a choice to make.
Another possible approach is "model, document and share"[1]:
> Imagine you’ve started a new job as an engineering manager, and the teams around you are too busy to use a planning process. You’ve mentioned to your peers a few times that you’ve seen Kanban work effectively, but folks tried it two years ago and are still upset whenever the word is mentioned: it just doesn’t work here.
> Your first reaction might be to confront this head on, but it takes a while to build credibility after starting a new job. Sure, you’ve been hired for your experience so they respect your judgement, but it’s a hard sell to convince someone that your personal experience should invalidate their personal experience.
Readit News
The tech lead started out by complaining about how he has to do an untested unplanned release today because another team made some urgent changes. He's the only person who knows how to release it. The other team didn't communicate until today that a release is necessary.
We've done two other releases in the past month and both required a day of troubleshooting to fix issues.
Both of us have been working at this company for about 3 years and we both have over a decade of experience in software development.
When he finished complaining, I started asking questions and making suggestions about how we can improve things. - Push back on the team that needs these urgent changes. Let them learn to do the release. - Deny the release since they didn't communicate earlier. - Improve the release process.
Everything I suggested was just flatly denied as impossible. - The other team doesn't know how to do the release. - He wants to be a "team player" so he can't deny the release. - Project managers will never allocate time to improve the release process.
I feel strange because I've seen this same thing for my whole career and I still try fight for what's right when others appear to moan and carry on.
However, my experience tells me that bringing this stuff to my manager is even worse. My manager doesn't know anything about the code, my project, or the release project. He may assume it's complaining for the sake of complaining. It has been used as ammunition in reviews against me.
Learned helplessness sucks and I wish I could do more. I don't think either of the suggestions in the article are feasible for many ICs. Teams are ambivalent to making improvements, and retrospectives carry very little weight. Managers are above the fray and won't be held responsible for by people below them.
My manager doesn't know anything about the code, my project, or the release project.
IOW, your team has no leadership and no management. Your team lead may be called that, but sounds like he doesn't lead at all. Your manager may be called that, but he doesn't manage either. This happens a lot when people are given responsibility without any authority (probably the case for you "team lead"), or authority over things they don't understand (probably the case for your "manager").
The bad news is that you're part of a disorganized mob. The good news is that there is a leadership vacuum that, if you play your cards right, you could fill. If you want. Of course, even if you succeed, it's possible no one will care about your accomplishments. And you will probably make enemies. It's possible you'll be rewarded but given your description, I view that as unlikely.
As a side note, suggesting to the team lead that he lead and do X, Y or Z may appear to him as if you want him to take on risks with little upside. He's probably thought about doing those things anyway but just decided it's not worth the trouble. It's not all that surprising that he doesn't want to rock the boat.
Perhaps. I thought so once in exactly this situation, and it led to an incredible amount of frustration, and burnout. And others were hurt the same way. That leadership vacuum was very real, but nobody wanted it to be filled by anyone already in the company, it turned out.
Eventually we got a very competent new team lead from outside and he was accepted by everyone immediately. For one of the teams anyway; the other one is still a complete mess. I just make sure I'm on the right one.
If you don't want to become a leader, what do you do in a disorganized mob situation? Leave?
> The other team doesn't know how to do the release.
> He wants to be a "team player" so he can't deny the release.
> Project managers will never allocate time to improve the release process.
and your good reasons not to push internally for a better system?
> My manager doesn't know anything about the code, my project, or the release project.
> He may assume it's complaining for the sake of complaining.
> It has been used as ammunition in reviews against me.
In fact the team leader is a "hero" who extinguished a fire caused by other department.
There is a self-reinforcing aspect to learned helplessness in teams, as you've pointed out.
I learned pretty early on that I have political capital, and very little of it. Instead, I have to have a manager that's pretty buck wild when I tell them I need them to be, and I have to deliver them some wins that will grease their wheels too. Good managers are hard to come by though, and if they get noticed they're promoted and gone in a couple of years.
Personally, I think the transactional nature of software positions is what largely holds us back. If I get promoted, then there'll be a new lead, and this cycle will start all over again.
I think the reality is that this kind of change can't originate from the in-the-trenches folks at the bottom of totem pole. Management will never let individuals waver from their short term goals.
The only times I've seen developer pain-points successfully addressed in a sustainable way, it's because there was a dedicated team allocated to that sort of thing: a "developer experience" team, or some equivalent.
(Although, the issues you ran into are kind of an org issue in addition to a developer experience issue)
I came to this conclusion after watching the place I work at for 2 and a half years fail to implement any of the grand ambitions management had in their heads. We wanted code review, pull requests, build pipelines, automated releases, updated OS's, logging, disaster recovery, you name it. No one had the experience, time, or energy to implement it, so it falls on one or two juniors (me) to quickly learn and implement while learning. No one is hired with experience in any of these topics, only fresh-grads that will replicate the garbage you're trying to get away from are hired, because they will on paper be producing more features for less financial cost.
I'm a manager. Please. PLEASE don't believe this. In-the-trenches folks ARE powerful. Really. I promise. I can't implement change if YOU don't do it. I don't know what sucks if YOU don't tell me.
I NEED you to help me make things better.
Social tissue is a strange medium and most of the time it's a friction generating engine. People complain, time passes on, nothing happens, repeat.
You can change things but you must do it. You must put your reputation at risk, expense time that you wont be compensated for, go through conflicts you'd rather avoid, make lifelong enemies wether you succeed or not. Then, you changed the rigid structure once more and the next guy will either try to put it back where it was and face less resistance or fold it one more time and face even more.
You cannot teach a rigid structure agility just like you cannot teach iron to be silk, and the structure is rigid for a reason: size, cost, regulation, cultural beliefs, rot due to time, past mistakes (in my experience the largest factor: a mistake 10 years ago can justify 3 days a week of useless processes today) etc. You can change the structure but it's CEO-level work to change its rigidity.
Ironically, the only way to minimize the "game playing" is by mastering them. There will always be people out there whose only real skill is manipulating the people around them, and who will deploy that skill at full force.
I'm not saying you should manipulate the people around you... but the twin powers of, "understanding whats going on around you" and "documenting everything", go a long way.
I wrote my first custom business program 40 years ago. Since then, I've only been on two teams that didn't exhibit this attitude.
At this point, I really don't want to continue programming for a living, precisely because so many people adamantly refuse to fix problems with painfully obvious (and quick and cheap) solutions.
I've given up on trying to get people to change their behavior. If I do continue programming, my approach will be to try to find one of those rare teams that actually changes and improves.
I don't introduce new information at meetings unless I am ready to back it up, which usually takes an hour of preparation for every hour of meeting. If time is of the essence, it's often better to settle things in hallway conversations or via web chat.
This.
While I currently have a manager that mostly understands such complaints, I have learned that even then, it amounts to nothing. Because he can't change material things without invoking higher-ups, and they won't be brought on board. For one thing because that would require a lower level manager to create tasks for a higher manager. And that's not how management works in many companies: A lower level manager didn't get to be a lower level manager by creating "work" for higher-ups. He got to be a lower level manager by absorbing the complaints from the ground force, assuring them that their voice is heard while at the same time shielding the higher-ups.
Maybe I've just been in sh*tty IT companies all my life, but I'm seeing this, and similar pattern for nearly 30 years now, and that's why I'm now searching for a job that is as far away from the potential to create overbearing complexity as possible. Even to be point where salary be damned.
Quit. This is a sign of a broken and unfixable workplace.
What... does he know?
Fine, he doesn't get into the code itself. But the project and its dependencies? Yikes.
At one point, one particularly hard-pressed user wanted to make really sweeping changes to the library and had limited time. I stalled and stalled to try and review all their changes and convince myself they were going to work, and they got more and more agitated, and my TL (who was himself frustrated by how much of my time this was consuming) told them "you can merge whatever you want if you take over maintaining it". So that's what happened, and everybody left happy.
> It is your life. You own it. You run it. You create it.
> Many developers we talk to are frustrated. Their concerns are varied. [...] And the answer we give is always the same.
> "Why can't you change it?"
> But, for some reason, developers seem to resist change. They hunker down, and hope things will get better.
> [...]
> So here's the most important tip in the book.
> Tip 3: You Have Agency
> Does your environment suck? Is your job boring? Try to fix it. But don't try forever. As Martin Fowler says, "You can change your organization, or you can change your organization."
When I read the situation you describe, the best thing would be to acknowledge where the tech lead is emotionally, and sympathize, because it sounds like they simply wanted to sound off about it, not actually change it at all. It's always easy to say no to why something won't work, and it sounds like a classic drama triangle situation, with someone coming in as a persecutor, you stepped in as a potential rescuer, then you became a victim because obviously your suggestions won't work.
You, however, don't need to accept the situation, and you can try to change it if you wish. Too many people (including myself, hence why I always re-read those few lines) fall into the trap of thinking that someone else should make this or that change, make a decision, or that you require approval or authority. This is far less true than people believe.
As someone who has stepped into a software manager role for the first time from being a software engineer for most of my career, my view is that I'm not always there to solve the problem, but to give space for people to solve the problems themselves. It helps if I know what's going on, or understand the issues, but I trust my team, and those I work with, and would rather get them together with the right people to discuss it through, and work out a way forward.
I appreciate I don't know the situation at your organization, the battles you fought, the things that have been said. Change is never easy, and there's always resistance, but you do have agency, as does your tech lead. Never forget it. Good luck.
If they’re using your valid concerns as issues on performance reviews, then the company values making low quality software, not their employees. Get out of the sausage factory and get into a place that values their employees and product. You’ll be much happier.
If the team is disfunctional or the manager bad consider changing work. Not only due to the dysfunction but because a good team and manager is part of ones' mentoring, and it hampers ones' growth. At least now you gain some reflexes that will assist you in the future, if you chose to move on.
Also these deployment tasks are quite repetitive, it's a big strain, I can understand your pain. I have managed to automate myself out of this, at my current job: most deployment tasks involve the following steps: Commit the change, open a pull request, wait for the CI build to complete and download the build log, extract an image id from the CI build log, put it in some other repository and commit the change. It is possible to automate the first three steps with the github api (if your shop is using github), I have an example here: https://github.com/MoserMichael/githubapitools . The rest of it is easily scriptable.
I'd suggest to try to see the situation from their perspective, which you maybe already did, but sometimes this helps understanding a perceived lack of action.
Dont suggest, make enemies: send emails to both teams saying now it must be like that, lay the plan, pose a meeting to go through the plan and get yelled at. Once you get yelled at by the complainypants because what you tell them to do is too hard, now you can go to your manager and explain him they complain for the sake of complaining. The tech lead will get a earful and either fight you to his grave or admit defeat and let you do your reform grumbling waiting for you to fail.
I do that in a giant bank that cannot move without 10 approvers to everything, and the more these helpless teams fight me the more management put me in their midst to reform them ... and Im just a silly dev nobody and I can tell you the wonders one can do by just bulldozing. There are things to do if you fail eventually, which is to teach the higher ups to enjoy trying and failing, but that requires a bit of weaseling charisma :D
You can try but once there’s resistance is probably a better idea to find something else. Let management figure out why people keep quitting.
I don’t like giving this recommendation out but the other way can make for a career limiting move.
I feel you, it's especially frustrating when your product has such fragile and bespoke testing that things break on a regular basis. Then, us devs have to spend valuable time just troubleshooting.
They broke it they have to fix it. You need the light shining on that team, not strive to fix it yourself.
This is called being a toxic employee, I've come to think of that as a good thing but it's definitely not a career move.
Good story though, can definitely relate
Deleted Comment
A lot of times learned helplessness is true helplessness with learned resignation.
It's also caused by misaligned incentives. At most BigCo, people don't get promoted for fixing technical debt. They get promoted for launching shiny new features and products.
If they had clear, unimpeachable metrics, then you could increase your status by fixing them. Since they don't, few people engage with them. Worse, engaging with some of these problems can lose you status, and so tackling them becomes a form of self-sacrifice.
Big new features are, it’s infinitely easier to build process improvements into new projects vs fighting for improvements in existing areas.
And sometimes the technical debt cloud disappears after fully understanding the business rules and special cases which were the root cause for the code in question. This often gets overlooked and may (edit) lead to refactoring/rewrite approaches which ultimately fail. As everyone knows, not a rare occurrence.
Dealing with the infrastructure is your job. You know the ins and outs. It doesn't seem complicated to you.
For someone who is dealing with the application logic, having to context switch to infrastructure is painful. I have no idea how you have set things up. When I read your documentation I'm just baffled at the sheer amount of complexity. I don't want to deal with it. It's not my full time job. I have other things to worry about. I have no mental space to deal with this headache. I will just ask you so I can move on to do my actual job - which is already taking up over 90% of my mental capacity.
When I setup my own infrastructure, I make it as simple and stupid as possible, precisely because I have no mental slack to deal with all the complexity that I see devops people deal with on a daily basis.
At most jobs I've been to, the complexity I've seen in infrastructure is mind boggling. My intuition is that this complexity is not actually needed if you remove all the cruft and think from first principles about what the actual problem is you are trying to solve.
However, people in devops seem to enjoy building and maintaining this kind of complexity, and even take pride in how everything is orchestrated.
To me it's just a headache.
When the error message contains a clear message and a URL with step-by-step instructions, and the person in question just pastes the full error message including the URL, and asks "what do I do?"
When a problem can be solved by responding with "please follow the instructions in the error message you sent me".
When someone makes an internal group post asking "how do I do X?" and the pinned post that was right below the submit button contains a summary and link to detailed step-by-step instructions titled "how to X".
Have you ever worked on infrastructure? This comment is wrong on soo many levels.
it's basically "beautiful mind" syndrome, and it's steeped deeply in ego.
now if on the other hand that teacher is coming from a place where it was hard for him, he probably has a sense of empathy and is willing to teach.
But when ops is glitchy, I have to blow half my day faking interest in docker? I mean best case, I learn how ops works, and then what. I become the one they tap when ops is swamped, then before I know it, I'm doing ops.
And I quit, because as much as I love the ops team, I hate the gig. I'd hate to be a designer, or a BA. It's not where I find joy.
So when I hear people say.. Just follow these steps to trouble shoot, I wonder if it would be cool to tell that to our users..
Infrastructure is like application dev in that it’s a bundle of histories of feature work over time and with many contributors, and many customers with competing wants and not enough time to satisfy everyone.
Granted a primary goal should be to have simple APIs/workflows for app devs to use, but complicated insides is as natural as any advanced system.
We've had working sessions where the moment there was an issue, the developers at BigCo said that they need to talk to someone on some team and they wrap up the meeting for the day -- even if we had just got started. It's gotten to the point where we will have project managers in the working session -- essentially babysitting -- to stop the devs from just giving up.
The amount of stonewalling I've seen is insane. We were trying to address an issue with our integration and I was told that a particular configuration on the client side was impossible, or maybe that we needed a new license, or perhaps we needed to contract to another vendor and it's going to cost half a million dollars, or whatever other excuses they could come up with.
So I asked them what product BigCo used, I looked up the documentation, I sent an email with 12 steps on how to make the configuration change, I got on a video call with my contact at BigCo (and his boss to make sure he showed up) and I literally walked him through the setting up the configuration.
I just don't understand how you get to the point where that's business as usual.
Simple, just get 100 junior developers working on the same product, and make the Project Managers’s manage them.
Just this past week I was working through some infra setup challenges, I had to piece together half a dozen different documents, several README files, and a few archived slack threads. These sources often are outdated or even contradicted each other. Even after all that and some earnest trial and error I still had to ask for help.
It's fine to want to point to the docs, but when you do that you better have good docs you're pointing to. As an infra team, often your "API" is your docs and templates. Searching slack threads and piecing together bad docs isn't a scalable solution for every dev to repeat individually. But neither is pinging an individual with every question. Teams need to invest in good documentation.
The problem is people that have not searched or tried at all, and then ask their senior to please do their job for them.
I've also seen new hires burn an entire week because they were afraid to ask a question.
But I think the balance has definitely shifted towards asking too much help...
Study a bit, WRITE my discoveries down
Craft a message with issues and background clear to readers (that includes myself when I forgot about this)
Send message to selected recipients, continue studying depends on free time available and issue importance
If they don't read the error messages from their tools, they are not engineers: they are idiots.
You should try to change team or company then. Don't believe all SWEs suck. There are pretty good coders out there :)
A group of monkeys in a room. In the center of the room is a tall pole with a bunch of bananas suspended from the top.
Every time a monkey tries to reach the bananas it is hit with a torrent of cold water from an overhead shower. Eventually, the monkeys learn that something bad will happen if they climb up the pole so they just sit and don’t even try again.
If a new monkey is added to replace an old monkey, it will of course try to go for the bananas. But the original monkeys will grab and drag it down before the punishment is even triggered. After a while, the monkey gets the message and it stops trying.
Repeat the action of adding a new monkey in place of an old one enough times and you will reach a point where none of the monkeys know why they shouldn't get the bananas but they still won't try.
The story concludes by stating that even if we remove the shower at this point it won't really make a difference. The learned behavior is already deeply ingrained in the culture of the group.
Funny enough, I've seen this type of issue in many human teams/projects. And usually the inertia is far too strong for just a couple of engineering "monkeys" to radically change things if they don't get support from a few management "monkeys".
While oversimplified, it can serve as a good example on how an initially rational practice could easily became dogma even when the initial stimulus is no longer present.
Part of my day job in the past 2 years has been to consult a client in refactoring/rewriting an old platform. The client's old team is still in place and even if we've added some new people just to try and change the mindset, the inertia of the spaghetti codebase coupled with the client's short and mid term financial goals as well as the "traditions" of the old team has stopped us from enacting any groundbreaking changes.
I guess the moral is that there are many forces at play in this kind of setup and stories like these can help in framing some of them in a fun way.
Just because the fires are different every day at Startup Inc. doesn't mean we're not being conditioned to endure suffering.
Case in point, a previous BigCo on call roster that I was required to partake in, sucks, makes you lose will to live, keeps you up at night, nothing documented, on-call phone ringing 5-7 of the nights in the week.
I tried to improve this process and was mildly sucessful even if that was just to get these incidents recorded.
Fast forward to my next job at Startup Inc. and their on call roster, sucks, makes you lose will to live, keeps you up at night, nothing documented, on-call phone ringing less but still often enough and with higher urgency.
The source of the two problems was different most of the time, one being a lack of training and effectively giving children knives.
The other being moving fast and breaking things resulting in things breaking after it was declared that version x.x went out without any issues just like the CEO wanted.
https://blog.danielna.com/talks/pushing-through-friction/
> Hold managers accountable: one of the key responsibilities of leaders is to create positive change for their teams. Once you notice a situation where you think everyone is in learned helplessness mode, make sure to notify your manager and follow-up until the problem is addressed.
"Hey boss, I noticed that Bob and Joe are really helpless at their jobs and that you just kinda watch it happen."
Yeah what could possibly go wrong with that?
I'm not saying the thinking is wrong, but a little idealistic. Not everyone works at BigCo where it is impossible to get fired and you only ever get transferred. Nevermind most of the people in this position are going to have extremely short tenture.
How else am I supposed to know Bob and Joe are doing bad work? Should I count their LOC/day? Pore over their code and rely on my own (10 years out of date) opinion of it?
In my humble opinion, there is one (1) good way to measure developer productivity, and that is to ask their teammates, and if you won't tell me your teammate is awful, you can't really complain that I don't act on it. Of course it's possible that you tell me and I don't do anything, but at least then it's my fault and not yours.
It describes very well the phenomenon but gives advice that I can only see ever working at BigCo. Where your team is huge and if you screw up there's another one down the hall.
The truth is most places that exhibit this level of helplessness often include a small team of 5 or 6, each with plenty of tenure. With no other lateral teams to move to.
Now you want the new guy to walk up to the boss and tell him that his drinking buddies for the past decade are the reason things are falling apart and he's an ineffective manager for not noticing it. You can play with the wording but that's the message.
In the real world we have political factors like seniority, favoritism, nepotism, tenure, and nevermind external factors.
In the real world the boss doesn't actually care if the team is working well or not. He only cares that the perception of the team is positive. If the new guy threatens the perception, he's gotta go. Nobody cares if it gets fixed. If nobody knows it's broken there's no need to fix it.
Let's have less articles like this one where we basically fire ourselves ostensibly for virtue signaling and more articles about how to socially engineer your way to the fucking top. Because let's face it, that's what it actually takes.
If you work in a small company, pretty much any boss will be happy to hear feedback like "Hey, it looks like there's an impediment to smooth work, and the team has gotten used to it, but it's costing us". Bonus points if you propose a solution.(If your manager isn't happy to hear that, it's time to leave, because small companies closing their eyes to this have a tendency to go down in flames)
But you'll definitely need to learn to take the blame out of your communication. Learned helplessness is a technical term, it doesn't mean "Bob and Joe are helpless". And your manager not noticing a problem is hardly "you just kinda watch it happen".
Frame it as a possible improvement to what the team can do - because that's what it is. Yes, you might get turned down. And if you repeatedly get turned down, you have a choice to make.
> Imagine you’ve started a new job as an engineering manager, and the teams around you are too busy to use a planning process. You’ve mentioned to your peers a few times that you’ve seen Kanban work effectively, but folks tried it two years ago and are still upset whenever the word is mentioned: it just doesn’t work here.
> Your first reaction might be to confront this head on, but it takes a while to build credibility after starting a new job. Sure, you’ve been hired for your experience so they respect your judgement, but it’s a hard sell to convince someone that your personal experience should invalidate their personal experience.
[1] https://lethain.com/model-document-share/