My LinkedIn title has been “Tech Debt Loan Shark” for years and it has garnered some of the funniest cold opens from recruiters over the years. I don’t think there’s a real problem with admitting technical debt at healthy engineering orgs.
I call myself a “Loan Shark” because if you aren’t given an option by leadership to take the time to do something right, “admitting” technical debt is less about “taking blame” than it is about covering your ass. When that debt comes to bite the company later, bring your receipts and use them as leverage to do some proper technical investment driven by the engineers on the ground floor.
That's hilarious! Does that make Management "Technical Debt CDO Traders"? :)
"Loan Shark" nicely anchors the business consequence of piling debt on top of debt. The interest rate ain't gonna be your safe little bond interest rate, it's gonna be whatever the situation says it is.
I also don’t think tech debt is a problem in and of itself. And I’ve worked a number of startups with serious tech debt that was holding back velocity.
I really enjoy the challenge of plotting a course out of a hole like that. However in my experience without management buy-in it just won’t happen. Even if you as an engineer do the planning work to make it as easy as possible to pick off low hanging fruit, you’ll typically be seen as “off task”. You’ll get lip service about how code quality matters, because without that they’ll scare away the other devs. But they’ll be hoping you give up and just focus on delivering what the management team is asking for each quarter.
It takes a ton of patience and management hand-holding, on the scale of 2+ years to shift the culture. And you have to be willing to forgo any chance at personal advancement for that time. Because you’re “off task”.
I think you could take that approach and eventually get an advancement bump around the 3-4 year timeframe but it’s not the fastest way to advance by any means.
And the total failure rate for an endeavor like that is pretty high. Maybe more than half.
We run on a debt-based economy so it should be no surprise that consciously issuing and managing tech debt works for growth in similar ways IMO.
Sure, it's possible to live debt-free and manage finances or architecture frugally - and I consider that a noble thing - but you will have a hard time keeping up with those who keep covering up their debt with more and larger debt.
Yes, and I think that is what makes risk assessment hard.
All debt packages risk. However, the kind of debt/risk perception we are personally conditioned around in daily life does not translate to software. Managing software debt requires investment banker-like thinking, rather than the mindset of a credit card or mortgage holder.
People always have stories about technical debt at work and open source, etc. Does anyone have stories of projects with no technical debt? Perfectly maintainable code?
We like to blame bizdev and sales for putting pressure on us, but that feels like a cop-out to me. Maybe I'm just a bad programmer but even if you took out the deadlines and "requirements" I'll probably still ship code with some warts here and there. Complaining about people in other positions is no way to improve.
Sorry, maybe a bit of an extreme take. Am still interested in stories of especially maintainable codebases, as I'm probably over-correcting here.
I do a lot of technical due diligence for companies in a bunch of different contexts (private equity pre-acquisition, post acquisition, management change over, etc). I have found that tech debt is highly correlated to the business goals and tech organization structure, and has to be evaluated in those contexts.
Startups you want to see with fairly high tech debt. Tech debt there is a very often a sign of shipped code that is meeting some customer need. You do need to watch out for insane tech debt that may be hard to undo later on (running everything in an ancient proprietary language for example).
Startups with low tech debt often are devs navel gazing without a lot of revenue coming in. This is surprisingly true almost all the time.
Very large companies will tend to have a mashup of all kinds of technology that is never fully untangled. A very large Corp with little tech debt is exceptionally rare. If it ain’t broke don’t fix it is the rule.
Mid sized corps and late stage startups seem to be the sweet spot. Fairly low tech debt systems that is providing high value is still possible. This is where you have enough maturity to know what “good” looks like, and enough developers, momentum and budget to implement the vision.
One small anecdote of a project at work. The oldest current project. All our projects are fairly simple and similar. They all use libraries and templates. This one is the last of the early projects before those templates and libraries. It's the easiest to manage. Things rarely go wrong. No forced upgrades because of a library vulnerability or template update. Bugs have been always been straightforward to find and fix because everything is procedural and clear. No abstraction 7 layers deep. Just clean, simple code.
Code doesn’t have to be perfectly maintainable and „elegant“. It just has to be useful i.e. good enough. If it solves your user‘s problem, that’s good enough. If your development speed can keep up as you add new requirements, that’s good enough.
Doesn’t mean you should ship untested code because that would decrease both the value to the user and your development speed. There are projects out there that go into the other extreme, entering the realm of premature abstraction and accidental complexity. In my experience that’s even worse because in that case it’s hard to convince stakeholders that there even is a problem.
It‘s a tough pill to swallow for devs with perfectionist inclinations like me, but at the end what matters is providing value to the user. In fact, I think technical debt will always be present as long as requirements change, and as the saying goes, change is the only constant in software engineering. It‘s not something to be completely avoided as that’s impossible. It just needs to be managed in a pragmatic way to keep up velocity.
Every engineering team I've worked with virtually every engineer complains about technical debt - regardless of how much time is allocated to manage it. I'm not suggesting it isn't important to take care of it, just that sometimes it comes across as the boy who cried wolf.
Technical debt is entered into deliberately, as a trade-off for time that must be paid back, with interest. There's no technical debt that isn't "self-admitted".
Technical debt is entered into deliberately, as a trade-off for time that must be paid back, with interest.
The implication here is that if only you had enough time you wouldn't have any tech debt. I don't think that's true.
Some of the tech debt is due to not having enough knowledge about the future of your system; what it needs to do, how it'll be used, etc. You have to make a decision about the tech now and sometimes that decision turns out to be wrong, or at least incomplete, which leaves you with some debt that will need paid off some time.
I also believe some tech debt is unavoidable due to churn from outside of the system. Best practises for a given part of an app aren't fixed (eg React class components shifting to functional components), so sometimes something that was once good can turn into debt over time with no input from you.
Publicly self-admitted, in this case. Apart from that: if I inherit the code base of a bad programmer on my job, with a lot of technical debt, which part of it is a deliberate trade-off? It's still "time that must be paid back, with interest", but how it came to be is not part of the definition. The term is merely meant to help confer the problem to non-technical people, who _then_ can make a conscious decision to accrue technical debt. If they do so out of incompetence, it's still technical debt.
> but how it came to be is not part of the definition
It's possible the people working at the time were not conscious of the trade-offs they were making. But it still happened and that's how it came to be.
> if I inherit the code base of a bad programmer on my job
It sucks, but it's almost guaranteed to be the case, to some degree. Think from the company perspective. You're thinking too much from your personal perspective, which is kind of irrelevant to the company. You're not using the term just for you.
I think the trade-off is at the company level. They hired a worse programmer to save money/time back then and accelerate development in the short term.
Agreed, just like real debit it isn't necessarily bad as long as you have a valid plan on how to repay it. And, like real debt, taking on technical debt may allow you to do things that wouldn't be possible otherwise (getting a product to market in time for example), similar to how if you refuse to borrow money to buy a house / car but save up for it instead you'll probably have to wait a long time before having it.
The problem though is that in the financial world there are rules and often laws about how much debt is acceptable and a bank will generally refuse to let you get into dangerous levels of debt whereas there are no such rules concerning technical debt so a company can keep piling it on until it eventually causes a collapse.
Not always. Technologies get old and have to be replaced over time. Tech debt is now mostly treated as a part of a product and developers do not decide what to do about it, but bussines through roles like product owners.
My own project has a number of TODO: comments in it. Some of them could be accurately described as unfixed bugs. Others are more like 'feature requests'.
When I am in 'bug fix' mode, I will try to locate and fix a few of them. I try to allocate a day now and then to focus on cleaning them up.
When you are in startup mode, trying to find the right product-market fit you cannot afford to make the code perfect. You have to get it to the point where potential customers find real value in what it can currently do. Shortcuts are a byproduct of that process.
Even if you can't get to it right away, a quick note of the issue in the code itself can make it easier to address it later when the problem is not fresh in your mind (or if someone else is tasked with doing it).
Not sure if the code is the best place to track technical debt. Whenever a compromise made that results in tech debt, a ticket should be created in the issue tracker. That way it's easier to get an overview about the project's health.
in my experience no one doubts that the debt exists, rather people just don't see the same value in addressing it, and aren't easily convinced by arguments they've already been ignoring for years or decades
i'm one of two feature devs on a team of four. over three years the team wrote a few hundred tickets for debt. when the client was acquired and we finally got space to address debt, non-technical stakeholders asked us to nuke the list and write a much shorter one that we can prioritize by complexity and customer impact. the debt is now being addressed in the same way as new build, because that's how we treated it; much like new build, it won't be completed to our satisfaction
in my experience the only thing that works, short of a major incident that devs forecasted and offered to prevent, is buying time--either to just write better code up front, or to use on _undirected_ remediation
This kind of "debt" is seldom "technical". It is a symptom of all manner of factors. In software organisations the symptoms of trouble become visible in codebases and/or behaviour of systems. Thinking about the artifacts as the problem area tends to cause fixing of symptoms and fringe causes, not systemic causes. As far as I've experienced, this sort of "narrow solving due to narrow framing" is endemic in software. It takes a lot of culture building for things like Gemba walk to work in practice, i.e. sure, root about for bad smells because that's how you discover something's fishy, but then work at a systems level to root out the systemic causes and malincentives.
Gemba means "the actual place" (where value is created).
The core idea behind the Gemba walk is a follows:
> Gemba (aka Walkie Lookie) walks denote the action of going to see the actual process, understand the work, ask questions, and learn. It is also known as one fundamental part of Lean management philosophy.
> Taiichi Ohno, an executive at Toyota, led the development of the concept of the Gemba Walk. The Gemba Walk is an opportunity for staff to stand back from their day-to-day tasks to walk the floor of their workplace to identify wasteful activities. The objective of Gemba Walk is to understand the value stream and its problems rather than review results or make superficial comments.
Readit News
I call myself a “Loan Shark” because if you aren’t given an option by leadership to take the time to do something right, “admitting” technical debt is less about “taking blame” than it is about covering your ass. When that debt comes to bite the company later, bring your receipts and use them as leverage to do some proper technical investment driven by the engineers on the ground floor.
"Loan Shark" nicely anchors the business consequence of piling debt on top of debt. The interest rate ain't gonna be your safe little bond interest rate, it's gonna be whatever the situation says it is.
I really enjoy the challenge of plotting a course out of a hole like that. However in my experience without management buy-in it just won’t happen. Even if you as an engineer do the planning work to make it as easy as possible to pick off low hanging fruit, you’ll typically be seen as “off task”. You’ll get lip service about how code quality matters, because without that they’ll scare away the other devs. But they’ll be hoping you give up and just focus on delivering what the management team is asking for each quarter.
It takes a ton of patience and management hand-holding, on the scale of 2+ years to shift the culture. And you have to be willing to forgo any chance at personal advancement for that time. Because you’re “off task”.
I think you could take that approach and eventually get an advancement bump around the 3-4 year timeframe but it’s not the fastest way to advance by any means.
And the total failure rate for an endeavor like that is pretty high. Maybe more than half.
Still, it is fascinating work.
Please elaborate.
- Open to connecting about tech debt?
- What's the going rate on a tech debt loan these days?
- Tech Debt Loan Blue Whale Opportunity
Sure, it's possible to live debt-free and manage finances or architecture frugally - and I consider that a noble thing - but you will have a hard time keeping up with those who keep covering up their debt with more and larger debt.
All debt packages risk. However, the kind of debt/risk perception we are personally conditioned around in daily life does not translate to software. Managing software debt requires investment banker-like thinking, rather than the mindset of a credit card or mortgage holder.
We like to blame bizdev and sales for putting pressure on us, but that feels like a cop-out to me. Maybe I'm just a bad programmer but even if you took out the deadlines and "requirements" I'll probably still ship code with some warts here and there. Complaining about people in other positions is no way to improve.
Sorry, maybe a bit of an extreme take. Am still interested in stories of especially maintainable codebases, as I'm probably over-correcting here.
Startups you want to see with fairly high tech debt. Tech debt there is a very often a sign of shipped code that is meeting some customer need. You do need to watch out for insane tech debt that may be hard to undo later on (running everything in an ancient proprietary language for example).
Startups with low tech debt often are devs navel gazing without a lot of revenue coming in. This is surprisingly true almost all the time.
Very large companies will tend to have a mashup of all kinds of technology that is never fully untangled. A very large Corp with little tech debt is exceptionally rare. If it ain’t broke don’t fix it is the rule.
Mid sized corps and late stage startups seem to be the sweet spot. Fairly low tech debt systems that is providing high value is still possible. This is where you have enough maturity to know what “good” looks like, and enough developers, momentum and budget to implement the vision.
Doesn’t mean you should ship untested code because that would decrease both the value to the user and your development speed. There are projects out there that go into the other extreme, entering the realm of premature abstraction and accidental complexity. In my experience that’s even worse because in that case it’s hard to convince stakeholders that there even is a problem.
It‘s a tough pill to swallow for devs with perfectionist inclinations like me, but at the end what matters is providing value to the user. In fact, I think technical debt will always be present as long as requirements change, and as the saying goes, change is the only constant in software engineering. It‘s not something to be completely avoided as that’s impossible. It just needs to be managed in a pragmatic way to keep up velocity.
Programmers who aren't you, and those damn MBAs ruining your flowstate
The implication here is that if only you had enough time you wouldn't have any tech debt. I don't think that's true.
Some of the tech debt is due to not having enough knowledge about the future of your system; what it needs to do, how it'll be used, etc. You have to make a decision about the tech now and sometimes that decision turns out to be wrong, or at least incomplete, which leaves you with some debt that will need paid off some time.
I also believe some tech debt is unavoidable due to churn from outside of the system. Best practises for a given part of an app aren't fixed (eg React class components shifting to functional components), so sometimes something that was once good can turn into debt over time with no input from you.
The term "tech debt" means deliberate choices to compromise the system in a controlled way, to achieve a purpose. E.g. speed to market.
If it's not that, then it's not tech debt.
It's possible the people working at the time were not conscious of the trade-offs they were making. But it still happened and that's how it came to be.
> if I inherit the code base of a bad programmer on my job
It sucks, but it's almost guaranteed to be the case, to some degree. Think from the company perspective. You're thinking too much from your personal perspective, which is kind of irrelevant to the company. You're not using the term just for you.
Technical debt isn't inherently bad. Otherwise it would be called "techinical loss".
The problem though is that in the financial world there are rules and often laws about how much debt is acceptable and a bank will generally refuse to let you get into dangerous levels of debt whereas there are no such rules concerning technical debt so a company can keep piling it on until it eventually causes a collapse.
Deleted Comment
When I am in 'bug fix' mode, I will try to locate and fix a few of them. I try to allocate a day now and then to focus on cleaning them up.
When you are in startup mode, trying to find the right product-market fit you cannot afford to make the code perfect. You have to get it to the point where potential customers find real value in what it can currently do. Shortcuts are a byproduct of that process.
Even if you can't get to it right away, a quick note of the issue in the code itself can make it easier to address it later when the problem is not fresh in your mind (or if someone else is tasked with doing it).
i'm one of two feature devs on a team of four. over three years the team wrote a few hundred tickets for debt. when the client was acquired and we finally got space to address debt, non-technical stakeholders asked us to nuke the list and write a much shorter one that we can prioritize by complexity and customer impact. the debt is now being addressed in the same way as new build, because that's how we treated it; much like new build, it won't be completed to our satisfaction
in my experience the only thing that works, short of a major incident that devs forecasted and offered to prevent, is buying time--either to just write better code up front, or to use on _undirected_ remediation
I always tell POs that if they want X done until Y day, that is going to result in tech debt.
When they are asking later why Z doesn't work properly or why it takes time, I just remind them of that technical debt they've agreed on.
edit: use better words
https://en.wikipedia.org/wiki/Gemba
Gemba means "the actual place" (where value is created).
The core idea behind the Gemba walk is a follows:
> Gemba (aka Walkie Lookie) walks denote the action of going to see the actual process, understand the work, ask questions, and learn. It is also known as one fundamental part of Lean management philosophy.
> Taiichi Ohno, an executive at Toyota, led the development of the concept of the Gemba Walk. The Gemba Walk is an opportunity for staff to stand back from their day-to-day tasks to walk the floor of their workplace to identify wasteful activities. The objective of Gemba Walk is to understand the value stream and its problems rather than review results or make superficial comments.