I think the general public (and by that I mean including software engineers too) overestimate the likelihood of a huge screw-up leading to being fired like they do in the movies, if the screw-up is neither (1) malicious/intentional in nature, nor (2) demonstrates that you're grossly incompetent for the job.
Most huge screw-ups happen to well-intentioned, knowledgeable software engineers, who simply made an honest mistake.
The correct way to handle it, on the company/management's perspective, is not to fire the person who made the mistake, but to allow them to correct it (perhaps with help from others). And that is indeed what happens in most cases. There are certainly poorly managed companies who would fire someone in these scenarios, but they should be less common than otherwise.
I'm not going to name any names: in the late 00s/early 10s I worked in one of the highest-profile, high-growth tech startups of its era, and I've personally made a blunder that corrupted literally millions of user records in the database. This incident was known internally as one of the most disastrous technical things that happened in the company's history, among a few others. The nature of the product was one of very quickly updating data, and updates were critically important (e.g. is affected by user spends) and hence restoring from DB backups of even the night before was unfeasible. There was irreparable damage where a whole team of us had to spend the next few weeks painstakingly hand-fixing data for users, and coming up with algorithms/code to fix these things as users use the product as they go. As you expect in this anecdote, I did not get fired, I was part of the team that worked tirelessly following this incident to fix user data, and I continued to have a good, growing career in my remaining time in this company (the next few years).
Work long enough and a person will screw up. That's how things go. And if you never screw up, then you're not really pushing yourself. Personally, I've deleted customer tables, updated the whole table when meaning to update one record, even shipped our private key once. There are probably even more I can't remember, and others from people that I helped fix (one time a boss locked us all out of the db - only fix was restoring lol) In all cases I became a better engineer by owning the mistake quickly, fixing the issue, and then putting processes in place to avoid it in the future. I think it shaped who I was to be a better engineer, manager, and owner.
The only time I've seen people fired on the spot is when they lie. After seeing someone fired, I asked one of the owners at an old company I worked at why that was the line. He responded that even the best performers make mistakes, but if he couldn't trust someone, he couldn't work with them. This was many years ago, and as I've grown in my career I have found it to be a pretty good line to draw.
> A young executive had made some bad decisions that cost the company several million dollars. He was summoned to Watson’s office, fully expecting to be dismissed. As he entered the office, the young executive said, “I suppose after that set of mistakes you will want to fire me.” Watson was said to have replied,
> “Not at all, young man, we have just spent a couple of million dollars educating you.” [1]
In most European countries it is certainly hard to fire people on a whim like in US and countries with similar weak labour laws.
I have seen often enough puzzled looks from team managers from such countries, when leeding European teams for the first time, and routinely get a NO for whatever crazy requests they happen to do.
Depending on which country, there still are mechanisms in place to fire people for what's called "culpable reasons" where I live, but there's a mandated layoff period (15 days here) after being fired to leaving the company. You get paid a certain fraction of what you'd otherwise be paid during that period and so on.
So yes, they can fire you if you (maliciously) screw up, but it's not like they just escort you out of the building on the spot.
> I think the general public (and by that I mean including software engineers too) overestimate the likelihood of a huge screw-up leading to being fired like they do in the movies
I’ve done some volunteer mentoring for a while. Fear of getting fired for small things is very widespread among younger generations right now. As far as I can tell, some of them build their mental model of the office from a combination of movies, Reddit posts, TikTok rage bait, and stories on social media. They’re so convinced that every corporation is an evil entity set on destroying their lives that small mistakes are catastrophized into career-ending moves.
The saddest part is when they make a small mistake and then let it snowball into lies and manipulation to cover it up. In the program I’m familiar with I don’t recall any stories of people being fired for singular honest mistakes, but there have been plenty of stories shared where people made a mistake and then caused far more problems by lying about it or even doing things like trying to attack and discredit people who witnessed the mistake as a defensive measure.
A large part of it stems from layoff culture. None of us know if this is going to be the last day at work. Is it rational? No, layoffs typically hit randomly and with little predictability, but that's not salve to those who worry about their jobs
There are, I think, different norms and risk levels in different types of employment.
Somewhat ironically, it's much easier to get fired from a minimum wage fast food service job than it is to get fired from a six-figure-salary job at Google.
The expense and time required to hire a well-paid, highly qualified engineer is much bigger than to hire someone to do a burger-flipping job. Firing is writing off these expenses.
Your point aside, that particular example is not the best since it's a lot easier to quickly land a customer in the hospital as a random fast food worker than as a random software engineer.
You're not really a sysadmin until you take down prod and not really a network engineer until you cut off your arm (drop a router connect with no way back in) or cause an STP storm. I'm sure there are more examples...
The joys of learning how tech works in the real world.
in one of my first jobs we used to call that "the kalidas problem", after an old indian legend about the great playwright kalidas being a fool in his youth, and cutting off the branch of a tree while sitting on it. https://www.indiaparenting.com/the-legend-of-kalidas.html
My team is in charge of infrastructure so most of us do everything. Sometimes I get to take down prod AND cut myself off from the network in one move.
Why is every service on this site alerting? All I did was plug something in. Maybe that cable wasn't labelled correctly after all, proof is in the results.
I've made my fair share of mistakes at my current employer. Took down business critical infrastructure, dropped productions tables, accidently placing test orders for clients and not getting them cancelled quick enough
Not once have I ever been reprimanded for my actions (Out side of being the butt of a few jokes and having to write a few "sorry" emails), they aren't something to be proud off but I'm definitely a better developer now because of these mistakes.
Mistakes happen, that's life, important thing is how you deal with the issues.
After dropping the production table, I had the fun job of restoring the backups and then having to scour through logs to rebuild any missing information between the last back up and the incident. I have never felt anywhere near as sick in my life, that was a fun Monday.
When I dropped the table and realised what I had done I went to my boss with my tail between my legs and just said "I fucked up", he didn't get angry and just said well get to work fixing it and so I did.
This kind of thinking is super destructive too because you'll just panic, turtle, and stop communicating which makes it impossible to get anything done.
I made a somewhat similar blunder to the original poster at a job some years ago. I got a dressing down from the CEO and that's it, went back to work and never heard of it again. As far as I can tell it had no effect on my career at that company.
In the software engineering industry, what I have seen is people not afraid of being fired due to a mistake, but to be afraid of the cost of their mistake in their next performance review. Make two mistakes (not necessarily in a row) and you are putting yourself in a dangerous corner (e.g., you may be considered to be lay off in the next round of lay offs).
> I think the general public (and by that I mean including software engineers too) overestimate the likelihood of a huge screw-up leading to being fired
I've seen this misconception perpetuated by management even in countries with very strong worker protection laws. Works particularly great on people who already have strong work ethics and internal fears, it stokes that fire. Employees would work as if any failure could lead to termination, especially in emergencies. Those managers use this as a "motivational" tool (as much as a hammer to the face can motivate you). They can squeeze more results out of their teams and maybe edge out a promotion.
It's illuminating to see when the people understand the pressure is many times fake, that even the internal SLAs are more relaxed than what they fear.
While I agree in principle, in this particular example the author was a new-ish hire who included an easter egg for no particular reason, which included copyrighted content that required all the existing CD's to be destroyed.
Great share! What do they say, "You can't make an omelette without breaking eggs". I think people can go through life trying to never make mistakes and ultimately I think that just holds them back.
After having introduced an Easter egg and being called out for it, the author states:
I became a cautionary tale though and would occasionally
warn off the new hires who might have had an inkling to do
something similar. And true to my word, I would tread very
carefully from that day on with an eye to what Apple HQ
would think about any of my actions — and potential
consequences (intended or not).
It is very likely that management weighed the author's value to the organization against the cost, real or perceived, to rectify this particular situation. An additional potential value which was ultimately realized is the author became an extension of organizational policy "at ground level."
IMHO, this is an optimal resolution and should be applauded. Management reaped a 20-year reward and the author kept his job.
Reminds me of a story (which I've heard in slightly varying versions) that happened in the early days of IBM. A man had made a costly mistake, but Thomas Watson didn't fire him, saying "we just spent all that money educating you!"
People learn by screwing up, and those can be the best lessons.
In one of my early jobs, I spammed a bunch of the company customers, trying to promote my music. This was before the Internet (1987 or so), and before spam, so I was a “proto-spammer.”
It became a watershed in my life, and I took my work and career, very, very seriously, after that.
> Reminds me of a story (which I've heard in slightly varying versions) that happened in the early days of IBM. A man had made a costly mistake, but Thomas Watson didn't fire him, saying "we just spent all that money educating you!"
> People learn by screwing up, and those can be the best lessons.
Precisely.
Had the author been fired for his actions, the only people who could learn from the situation would be those employed at that time. As each cycled out of the company, this lesson would be lost.
By addressing the error in judgement while retaining the author, the organization ensured the lesson would be passed on to each new generation of engineers without having to prescribe the policy dictatorially.
This is also a much better way of handling the problem than OP's blunder on the article.
Berating in an office might teach one person. Not berating and admitting process failure in public can be educational for the whole company.
Reminds me of Gitlab's database deletion thing a few years ago. Not only they spent money educating the one engineer, by making the story public they also educated their other employees, and dare I say the whole industry. I still hear people referring to that story from time to time.
I wouldn't say it's an optimal resolution, because it was a complete overreaction by management in the first place. The optional resolution would have been that the author was gently but firmly told "that isn't acceptable here, don't do it again". There was no need for his manager to tear him a new one over such a minor incident.
To answer the author's question: because the crayon picker rules! It's genuinely useful.
The article covers this but I think it's worth saying again: this was a different era in software development. In 1995 "shipping" meant literally "shipping": they had to produce real inventory, and then get it out into a bunch of different distribution channels. A problematic Easter egg caught too late could ultimately require an actual physical recall.
Also, there was at least one scientific graphics package from the late 1980s (I used it under BSD4.3 on microvaxen[0]) that not only had a crayon-based color picker, but used it for a simple calibration mechanism, since you could just buy a cheap box of crayons and adjust the screen colors to match. So there was at least one contemporary existence proof of "Crayola probably isn't going to go after you for this".
[0] I've forgotten the name; the main thing I remember is that your window was dimensioned in 0.0-1.0 floating point, not pixels. Some searching suggests it might have been PHIGS or GKS... and turns up a Tugboat article about FoilTeX using crayola color names similarly in 1992, so it was a more popular hack than I realized at the time.
We used to have to physically fly out a technician to each of our customers office to install software updates and fixes. Those bugs were very expensive to patch!
This is an example of classic software engineering. It's when the person writing the code understands why they're doing it, succeeds in that, then tastefully adds a little more, and has some fun, which in turn makes the product delightful.
Not to dismiss the importance of a strong Product lead, but often the role is a permanent stopgap for engineers who lack the talent and discipline demonstrated in this article. (Easter egg aside.)
One of the deliberate outcomes of putting a product lead in place is to train the engineer to fully understand the product. For any vertical market or B2B product the chance of hiring engineers who understand the business domain is virtually zero so they need clear guidance to build what customers want.
Usually what you end up with is a "three blind men and the elephant" situation; the product is simply too large for any one person to "fully" understand. Humans have a finite context window too.
So ending up with a product manager who is standing far enough away to see the whole elephant, but not in detail, while the engineers have detailed views of their part of the product but not the whole thing, is a reasonable position.
I found the linked article about his interview even more fascinating!
I heard once about a mechanical engineer who brought printed photos of his previous projects to the interview instead of a typed resume, to good results. Once you’ve had to interview someone, you realize many times the interviewer is as nervous as the interviewee, so breaking the ice is valuable.
As an embedded developer I do this regularly and encourage others to do the same. I bring a binder, each page has a large photo of a project or PCB and lists the core technologies involved.
I also bring a small case with a few previous projects from other jobs, ones that I can legally share outside an NDA, and a power supply.
It's been critically handy in some situations when I'm asked if I've worked with a certain technology, chip, protocol, etc. I can either flip to the page with a relevant project and start pointing (which also jogs my memory), or plug in the related device and fire it up. It definitely gets their attention and stands out.
Only corporate legal could freak out over the idea of hiding a single stanza of an 80 year old poem across a dozen resource names where they'll never ever be seen by the average user. If that's not fair use it should be.
You don't want to end up with a court trying to figure out whether that usage constitutes fair use. Taking a stand on whether it "should be" fair use isn't what company resources are for.
Legal troubles are exactly why you have a legal team. Uber didn't ask for permission, neither did OpenAI. Now they're worth billions. Microsoft and Apple do their own fairshare of probably borderline illegal things too.
The author of this post is a regular commenter on HN, JKCalhoun. Nice post, JKCalhoun! I love the three dozen dongles hanging on the wall of Dithering Heights.
The versions of these color pickers that were included in builds of Copland contain the strings "Hey what?" (for the HSL picker) and "Scheherazade" (for the crayon picker). Might those have been some of yours?
Blast from the past! A handful of sacred Power Macs in my elementary school computer lab had the shareware version of Glider installed ("Glider Trial", or "Trail" [sic] as my spelling-challenged classmates called it who were unfamiliar with trialware) and every one of us jockeyed for one of these machines as we filed into the room so that we could play it. Circa 1999 I somehow got my mom to ask a hapless Wal-Mart employee whether they made Glider from Casady and Greene written by John Calhoun for Windows. We got a weird look and a devastating "no". Sometime in the next year or two my best friend and I fixed up his Grandpa's old Mac Plus which happened to have an older full version of Glider on it which made me quite happy.
Wait, that is him? JKCalhoun? I probably should read more closely.
Jeez JKCalhoun is one of my favorite commenters on HN. He is very knowledgeable about Apple stuffs and I knew he probably worked in Apple for a long, long time just by reading his replies. He is one of the people that I look up to (i.e., when I don't feel burned out).
It's weird how... servile... this person sounds. They're wondered why they weren't fired, when their prank had no negative side effects and the only issue was a department head being a dick about it? hey think the lesson here was that they needed to learn professionalism? They should be bemoaning the world of power-tripping unreasonable bosses. No need to warp your idea of what's right around what someone got mad at you about.
> They're wondered why they weren't fired, when their prank had no negative side effects and the only issue was a department head being a dick about it?
There are plenty of companies where your boss being upset with you can lead to you being fired irrespective of whether it's logically justified. The author doesn't seem servile so much as green (which they admit and talk about quite a bit).
> No need to warp your idea of what's right around what someone got mad at you about.
Part of being an adult is the ability to mentally square how the world ought to be with how it actually is. It is perfectly reasonable to acknowledge that a decision you made wasn't optimal in retrospect even if it was the theoretically-correct one. This isn't "warping" anything as much as it is getting on with your life.
Carl Sagan had sued Apple the prior year for merely using his name as a project codename that was never going to be published anywhere or included in any customer product. John had placed an excerpt of a still-copyrighted text into a resource fork that would ship to customers—and in fact had already been burned to discs.
I think it would have been an overreaction to fire him, but it was absolutely within the realm of plausible outcomes.
> sued Apple the prior year for merely using his name as a project codename
This just highlights the fact that you can be sued for anything. For example, you could be sued for the same thing (even though CDs were destroyed) based on information from this blog.
It sounds like they had to scrap a CD run, so it wasn't exactly "no negative side effects." But yes, it's always a bad sign when the project you were hired for is canned and you get a new manager.
It sounded like they didn't have to scrap it, but did anyway? Or at least did not feel like it had to actually be justified to him.
> I remember him explaining how many CDs would have to be destroyed (had they already been created?) and the cost of those. I think I weakly explained that I had assumed that there were no copyright conflicts but he wasn’t hearing it. After that half-hearted defense I let him berate me and more or less accepted whatever fate was coming my way.
Yeah. I see what you are saying. I think the reason I am embarrassed about it is not that I am averse to offending upper management (my corporate overlords) but that I offended on petty grounds. An Easter egg is kind of an ego thing and I think that is what was embarrassing.
Readit News
Most huge screw-ups happen to well-intentioned, knowledgeable software engineers, who simply made an honest mistake.
The correct way to handle it, on the company/management's perspective, is not to fire the person who made the mistake, but to allow them to correct it (perhaps with help from others). And that is indeed what happens in most cases. There are certainly poorly managed companies who would fire someone in these scenarios, but they should be less common than otherwise.
I'm not going to name any names: in the late 00s/early 10s I worked in one of the highest-profile, high-growth tech startups of its era, and I've personally made a blunder that corrupted literally millions of user records in the database. This incident was known internally as one of the most disastrous technical things that happened in the company's history, among a few others. The nature of the product was one of very quickly updating data, and updates were critically important (e.g. is affected by user spends) and hence restoring from DB backups of even the night before was unfeasible. There was irreparable damage where a whole team of us had to spend the next few weeks painstakingly hand-fixing data for users, and coming up with algorithms/code to fix these things as users use the product as they go. As you expect in this anecdote, I did not get fired, I was part of the team that worked tirelessly following this incident to fix user data, and I continued to have a good, growing career in my remaining time in this company (the next few years).
The only time I've seen people fired on the spot is when they lie. After seeing someone fired, I asked one of the owners at an old company I worked at why that was the line. He responded that even the best performers make mistakes, but if he couldn't trust someone, he couldn't work with them. This was many years ago, and as I've grown in my career I have found it to be a pretty good line to draw.
> “Not at all, young man, we have just spent a couple of million dollars educating you.” [1]
[1] http://the-happy-manager.com/articles/characteristic-of-lead...
I have seen often enough puzzled looks from team managers from such countries, when leeding European teams for the first time, and routinely get a NO for whatever crazy requests they happen to do.
So yes, they can fire you if you (maliciously) screw up, but it's not like they just escort you out of the building on the spot.
I’ve done some volunteer mentoring for a while. Fear of getting fired for small things is very widespread among younger generations right now. As far as I can tell, some of them build their mental model of the office from a combination of movies, Reddit posts, TikTok rage bait, and stories on social media. They’re so convinced that every corporation is an evil entity set on destroying their lives that small mistakes are catastrophized into career-ending moves.
The saddest part is when they make a small mistake and then let it snowball into lies and manipulation to cover it up. In the program I’m familiar with I don’t recall any stories of people being fired for singular honest mistakes, but there have been plenty of stories shared where people made a mistake and then caused far more problems by lying about it or even doing things like trying to attack and discredit people who witnessed the mistake as a defensive measure.
Somewhat ironically, it's much easier to get fired from a minimum wage fast food service job than it is to get fired from a six-figure-salary job at Google.
The joys of learning how tech works in the real world.
in one of my first jobs we used to call that "the kalidas problem", after an old indian legend about the great playwright kalidas being a fool in his youth, and cutting off the branch of a tree while sitting on it. https://www.indiaparenting.com/the-legend-of-kalidas.html
Why is every service on this site alerting? All I did was plug something in. Maybe that cable wasn't labelled correctly after all, proof is in the results.
Not once have I ever been reprimanded for my actions (Out side of being the butt of a few jokes and having to write a few "sorry" emails), they aren't something to be proud off but I'm definitely a better developer now because of these mistakes.
Mistakes happen, that's life, important thing is how you deal with the issues.
After dropping the production table, I had the fun job of restoring the backups and then having to scour through logs to rebuild any missing information between the last back up and the incident. I have never felt anywhere near as sick in my life, that was a fun Monday.
When I dropped the table and realised what I had done I went to my boss with my tail between my legs and just said "I fucked up", he didn't get angry and just said well get to work fixing it and so I did.
I've seen this misconception perpetuated by management even in countries with very strong worker protection laws. Works particularly great on people who already have strong work ethics and internal fears, it stokes that fire. Employees would work as if any failure could lead to termination, especially in emergencies. Those managers use this as a "motivational" tool (as much as a hammer to the face can motivate you). They can squeeze more results out of their teams and maybe edge out a promotion.
It's illuminating to see when the people understand the pressure is many times fake, that even the internal SLAs are more relaxed than what they fear.
Deleted Comment
IMHO, this is an optimal resolution and should be applauded. Management reaped a 20-year reward and the author kept his job.
People learn by screwing up, and those can be the best lessons.
In one of my early jobs, I spammed a bunch of the company customers, trying to promote my music. This was before the Internet (1987 or so), and before spam, so I was a “proto-spammer.”
It became a watershed in my life, and I took my work and career, very, very seriously, after that.
> People learn by screwing up, and those can be the best lessons.
Precisely.
Had the author been fired for his actions, the only people who could learn from the situation would be those employed at that time. As each cycled out of the company, this lesson would be lost.
By addressing the error in judgement while retaining the author, the organization ensured the lesson would be passed on to each new generation of engineers without having to prescribe the policy dictatorially.
Berating in an office might teach one person. Not berating and admitting process failure in public can be educational for the whole company.
Reminds me of Gitlab's database deletion thing a few years ago. Not only they spent money educating the one engineer, by making the story public they also educated their other employees, and dare I say the whole industry. I still hear people referring to that story from time to time.
The article covers this but I think it's worth saying again: this was a different era in software development. In 1995 "shipping" meant literally "shipping": they had to produce real inventory, and then get it out into a bunch of different distribution channels. A problematic Easter egg caught too late could ultimately require an actual physical recall.
It's not surprising people freaked out over it.
[0] I've forgotten the name; the main thing I remember is that your window was dimensioned in 0.0-1.0 floating point, not pixels. Some searching suggests it might have been PHIGS or GKS... and turns up a Tugboat article about FoilTeX using crayola color names similarly in 1992, so it was a more popular hack than I realized at the time.
Not to dismiss the importance of a strong Product lead, but often the role is a permanent stopgap for engineers who lack the talent and discipline demonstrated in this article. (Easter egg aside.)
This is politics baked into the heirarchy, not a lack of "talent and discipline".
I had devs who told me they don’t care about product features. "Just tell me what to do, it’s my job to implement."
Usually what you end up with is a "three blind men and the elephant" situation; the product is simply too large for any one person to "fully" understand. Humans have a finite context window too.
So ending up with a product manager who is standing far enough away to see the whole elephant, but not in detail, while the engineers have detailed views of their part of the product but not the whole thing, is a reasonable position.
I heard once about a mechanical engineer who brought printed photos of his previous projects to the interview instead of a typed resume, to good results. Once you’ve had to interview someone, you realize many times the interviewer is as nervous as the interviewee, so breaking the ice is valuable.
https://engineersneedart.com/blog/interview/interview.html
I also bring a small case with a few previous projects from other jobs, ones that I can legally share outside an NDA, and a power supply.
It's been critically handy in some situations when I'm asked if I've worked with a certain technology, chip, protocol, etc. I can either flip to the page with a relevant project and start pointing (which also jogs my memory), or plug in the related device and fire it up. It definitely gets their attention and stands out.
The versions of these color pickers that were included in builds of Copland contain the strings "Hey what?" (for the HSL picker) and "Scheherazade" (for the crayon picker). Might those have been some of yours?
https://rezmason.net/chattin_with_jkcalhoun/copland_colorpic...
Hats off to you, Mr. Calhoun!
Jeez JKCalhoun is one of my favorite commenters on HN. He is very knowledgeable about Apple stuffs and I knew he probably worked in Apple for a long, long time just by reading his replies. He is one of the people that I look up to (i.e., when I don't feel burned out).
I can't recall the "Hey what?" though.
There are plenty of companies where your boss being upset with you can lead to you being fired irrespective of whether it's logically justified. The author doesn't seem servile so much as green (which they admit and talk about quite a bit).
> No need to warp your idea of what's right around what someone got mad at you about.
Part of being an adult is the ability to mentally square how the world ought to be with how it actually is. It is perfectly reasonable to acknowledge that a decision you made wasn't optimal in retrospect even if it was the theoretically-correct one. This isn't "warping" anything as much as it is getting on with your life.
I think it would have been an overreaction to fire him, but it was absolutely within the realm of plausible outcomes.
This just highlights the fact that you can be sued for anything. For example, you could be sued for the same thing (even though CDs were destroyed) based on information from this blog.
Copyright infringement lawsuits are a real thing and can include both the offending company (Apple) and all parties identified as potential violators.
In other words, management may have saved this guy's ass from being named in a very costly lawsuit.
> I remember him explaining how many CDs would have to be destroyed (had they already been created?) and the cost of those. I think I weakly explained that I had assumed that there were no copyright conflicts but he wasn’t hearing it. After that half-hearted defense I let him berate me and more or less accepted whatever fate was coming my way.