> When was the last time you had to debug an ancient codebase without documentation or help from a team?
All the time. 300-400k SLOC in C++. Legacy in the sense that there were no tests of any kind. Little-to-no documentation. Solo developer at the tiny company. Fix bugs and add features while keeping the system available to the tens of thousands of users.
A more recent example: here’s a patch for a critical feature we need. It was written over a year ago. The original author isn’t available anymore. You can write the code from scratch or try to resurrect the patch against master.
Being able to jump into a project and lead people towards some goal is definitely a skill for senior developer positions. Yes, you generally have a team you can lean on and have the ability to do research and all that. But how do you show that you can do all that in an interview?
Agree with the conclusion that a good thing to test for is for problem-solving.
The tech side depends a lot on what you’re doing. Although it gets ridiculous and organizations get lazy with this part. You don’t need to be white boarding graph algorithms for a junior web developer role. If your application is a social networking role and you’re interviewing a senior developer or architect? Definitely. They’re going to be teaching this stuff and need to understand it at a deep level.
+1 for "all the time". Today I have been debugging a critical piece of the system which is written in Python (none of the rest of the system is) and largely hasn't been updated since 2020 and, you'll not be surprised, has no comments, no documentation, and a fucked up deployment system which makes me cry every time I have to think about it.
Last week I was debugging some similarly uncommented, undocumented, Go code from 2020 written by lunatics that is also a critical piece of the system.
+1 here too. The situation always comes up after some kind of fucked up acquisition. Nothing like inheriting an entire code base built on someone else's stack when all the original developers decided to quit or were laid off. Ancient versions of Python and Java, deployment done by pull and pray. Multiple versions of the same service in different places because they were half way through some modernization when it all came to an end. Fun stuff. Getting your bearings fast is a real skill.
> When was the last time you had to debug an ancient codebase without documentation or help from a team?
Heck, I have to debug the stuff that some idiot (me) wrote six months ago and it might as well have been someone else who wrote it for all I remember about how to debug it
So many times I'm reviewing code and really wish unpleasant words with the developer that created this horrible code. Problem is, people think I'm weird when I talk to myself like that.
Edit: apparently this wasn't written clearly enough that I was the original dev that I'm having words with???
When is the last time you’ve been trying to get help from another team, only to realize that the engineer you are talking to can’t write code, even with the AI help?
Real. Engineers who don't think they have this problem, are engineers who see their dependencies and the lower layers of their stack as ossified black boxes they "can't" touch, rather than something they can reach into and fix (or even add features to!) when necessary.
IMHO the willingness to "dig your way down" to solve a problem at the correct layer (rather than working around a bug or missing feature in a lower layer by adding post-hoc ameliorations in "your" code) is one of the major "soft skills" that distinguishes more senior engineers. This is one of those things that senior engineers see as "the natural thing to do", without thinking about it; and fixes that take this approach work subtly to ensure that the overall engineered system is robust to both changes up and down the stack, and to unanticipated future use-cases.
And contrariwise, fixes tending not to take this approach even when it'd be a very good idea, is one of the central ways that a codebase starts to "rot" when all the senior engineers quit or are let go/replaced with more-junior talent. When, for example, every input-parsing edge-case is "fixed" by adding one more naive validation regex in front of the "opaque, scary" parser — rather than updating the parser grammar itself to enforce those validation rules — your codebase is actively evolving into a Ball of Mud.
Of course, the tradeoff for solving problems at the "correct" layer, is that you/your company often ends up having to maintain long-lived, trivial, private hotfix-branch forks of various ecosystem software you use in your stack. More often than not, the upstreams of the software you use don't see your problem as a problem, and so don't want to take your fix. So you've just got to keep it to yourself.
(Funny enough, you could probably trivially measure an engineer's seniority through a tool that auths against their github profile and calculates the number of such long-lived trivial private hotfix branches they've ever created or maintained. Much simpler than a coding challenge!)
That's I'd the company you're at will let you take the time to "do it right". Usually that's not the case. They want features yesterday. You try and fix something properly by going down to the correct layer? "You shouldn't be doing that."
Jumping into an unknown codebase (which may be a library you depend on) and being able to quickly investigate, debug, and root cause an issue is an extremely invaluable skill in my experience
Acting as if this isn’t useful in the real world won’t help. The real world is messy, documentation is often missing or unreliable, and the person/team who wrote the original code might not be around anymore.
> Fix bugs and add features while keeping the system available to the tens of thousands of users.
Don't tens of thousands of users warrant more developers? Or having enough of a budget to work on tests, or other things to improve the developer experience and be able to work without lots of stress? That's unfortunate.
> > When was the last time you had to debug an ancient codebase without documentation or help from a team?
> All the time.
I guess the minor difference was that it was part of your paid job instead of a huge time sink to get hired.
This is the biggest issue I have with all these tasks is that they take precious time that could be utilized better. In my case: working on real open-source software instead of throwaway made-up problems. (Source: I had to spend 3 days to write a transaction engine toy once and then I didn't get the job for another reason.)
To reinforce, an task I had when interviewing was to solve an issue with a webapp written in a language I had never used and a webserver I had never used.
It wasn't that hard if you know the fundamentals (http, ssl). It was sorta fun and I got the job.
Even then, the senior developer/architect "social networking app" graph algorithms stuff has its limits as well. Mathematicians come up with great algorithms, not software developers; we just use 'em. And this makes sense because those are two entirely different roles.
Imo it's more important that you can break down an existing best-in-class algorithm for x task and have your prospective dev answer questions about it/how they would use it, etc. Expecting every senior dev to be a best-in-class top mathematician as well as having a handle on the ever-changing language/library/ops/etc stacks is just crazy.
I'd never expect a senior dev on my teams to just know something like that, but I would expect that given a task they can do their research, find good candidates and understand the pros/cons of each approach. Because our power is in our ability to learn & use; the stuff that we already know is a bonus.
+1. Jumping into an unfamiliar code base with no docs and no access to the original developers offers a lucrative bottomless pit of work for a skilled freelancer. I made this my specialty a decade ago.
Yeah I thought that was a weird thing to highlight. The "debug this!" kind of interview is pretty much my favorite, because it's by far the closest to what I do for like 90% of my time at work (when I'm not doing communication tasks...).
Don't understand one thing: I can read any code of any legacy code base, sure. What I cannot do is to asses if the code is supposed to do its job (e.g., the business logic behind it). Is the code I'm reading supposed to calculate the pro-rated salary according to the "law"? Just by reading the code, you cannot know that. You need help, either from other developers who perhaps know the codebase of from product experts that know the business logic. Or documentation, sure, but this is usually a luxury one doesn't have.
So, definitely, I always need help debugging any kind of code (unless it's rather code that doesn't deal with product features).
> What I cannot do is to asses if the code is supposed to do its job
You can, to a degree. If you can get the system under test you can make an assertion about how you think it works and see if it holds. If you know what the "law" is you can test whether the system calculates it according to that specification. You will learn something from making those kinds of assertions.
Working Effectively with Legacy Code by Michael Feathers goes into detail about exactly this process: getting untested, undocumented code people rely on into a state that it can be reliably and safely maintained and extended.
Depending on the situation I often recommend going further and use model checking. A language and toolbox like TLA+ or Alloy is really useful to get from a high-level, "what should the system do?" specification down to, "what does the system actually do?" The results are some times surprising.
You're right that at some level you do need to work with someone who does understand what the system should do.
But you can figure out what the system actually does. And that is de facto what the business actually does... as opposed to what they think it does, logic errors and all.
A good senior programmer, in my opinion, thinks above the code like this.
Update: interviewing for this kind of stuff is hard. Take-home problems can be useful for this kind of thing in the right context. What would you do differently?
another +1 from me -- my job, with some exceptions, is literally "plop this guy into an unfamiliar project and implement some user need be it a bugfix, feature, or (silently) a refactor.
agnostic programming flueny/software practices, including being comfortable debugging unfamiliar things or navigating uncharted waters without many domain experts to guide me, is the number one skill required in my role.
bonus points if i end up becoming an active maintainer to the project, or someone who can help other devs w it, the more times i dip into it.
+1 as well: 2M LOC codebase with little to no tests and a lot of lost knowledge due to disbanded teams, people moving on etc. In my experience very common state of affairs honestly.
Yeah I think OP's overall argument is right but this particular example isn't needed and has plenty of counter-examples. It's more like when was the last time you had to solve [insert leetcode problem here] without any resources, under time constraints, with someone judging you and staring over your shoulder at all times, at work. I find under these conditions I have maybe 40% of my usual brainpower..
"> When was the last time you had to debug an ancient codebase without documentation or help from a team?"
I mean, even if you have a team, you are hired to do specifically this kind of work. I'm confused what OP thinks a software job is if not EXACTLY this.
> > When was the last time you had to debug an ancient codebase without documentation or help from a team?
> All the time.
So you were forced to do it in a silo? No access to any type of documentation, whatsoever? Not able to refer to any C++ manuals or any other type of documentation? Not permitted to access the internet to look things up? Barred from using online forums to seek help?
A partner of a friend quit their job earlier this year. They then took 4-6 weeks to prepare for each interview with Big Tech companies (4-6 weeks for Meta, 4-6 weeks for Stripe, etc.). Along the way, they also took random interviews just to practice and build muscle memory. They would grind leetcode several hours a day after researching which questions were likely to be encountered at each Big Tech.
This paid off and they accepted an offer for L6/staff at a MAANG.
Talked to them this week (haven't even started the new role) and they've already forgotten the details of most of what was practiced. They said that the hardest part was studying for the system design portion because they did not have experience with system design...but now made staff eng. at a MAANG. IRL, this individual is a good but not exceptional engineer having worked with them on a small project.
Wild; absolutely wild and I feel like explains a lot of the boom and bust hiring cycles. When I watch some of the system design interview prep videos, it's just a script. You'll go into the call and all you need to do is largely follow the script. It doesn't matter if you've actually designed similar or more complex systems; the point of the system design interview is apparently "do you know the script"?
Watch these two back to back at 2x speed and marvel at how much of this is executed like a script:
Back when I had a job hiring people we created problems we could walk people through and see what they figured out on the fly/what they knew but didn't know they knew. That was what I was taught whiteboard problems were, not this lame leet code. But I grew up with both parents in 1980s/90s Santa Cruz tech. The current scene adopted the practice but made it exclusive when it was intended to be inclusive (because there wasn't a huge talent pool back then so they were looking for people they could grow into developers).
One of my hires was a physicist who didn't know any of the jargon and had a look of terror when she first started on our whiteboard problems. Once we led her to a comfortable spot and she got into it she started talking about tools she made to help with her physics work with zero dev knowledge (she was shocked when we called her tools software, she was like, but it wasn't REAL software, we were like hate to break it to you, but that was not only real software you wrote, but crazy impressive). The white board problems were a tool to highlight potential.
I get that these big companies just want drop in widgets not people and that is what they are searching for, I just think it's funny they are using something that was created for the exact opposite purpose.
Part of me is kind of glad I'm too old to be one of the cool kids and have no hope to land a job that does these sorts of code challenges.
Well it shows that you are ready to invest a lot of your own free time into being recruited there - i.e. you'll be a loyal and hard worker. And that you're smart enough to learn it all. So I would say it has a character + IQ component to it.
Of course 95%+ of it will usually be useless in your real-life work, to solve most of those problems in the time given you need to know the problem and the optimal solution to it so basically memorization with little thinking.
If you talk to hiring at some of these companies, it is intentionally designed to be this way so that it is fairly meritocratic.
In other words, anybody, regardless of what university they went to or what courses they took out what advantages or disadvantages they had, can learn this stuff in a couple of months if they have what it takes for the role. And because the skills are so standardized, the process is pretty differently objective.
It's not expected that they'll actually use the specific skills in the job. But it shows they can learn skills of that type and then perform them at a high level in a stressful interview situation. Which is a great signal for whether they can learn the skills needed for the specific project they wind up on and perform in a high stakes deadline scenario.
I'm not defending the system, but I am saying there's a clear logic behind it.
The thing I think is funny in all this is that hiring a new manager is fraught with a high amount of risk (more so than an engineer), but they don't have nearly the level of hurdles to get over. Does the company interview past employees of the manager? Did the manager applicant have alcohol, drug issues, or weird sexual things he did to his direct reports or others? Or, instead, would you enjoy his presence on a golf outing?
I know one manager who had issues with all three got hired at Google. So. Think of the poor HR person that will have to clean up that mess.
The idea of it being positioned as meritocratic is hilarious to me. As if having a process outside of the leetcode question bank will lead to any more bias. I'm generally of the belief that an interview process should be standardized in an attempt to reduce bias, but I disagree that it needs to be something people can study for to the extent that leetcode questions can be studied.
I've interviewed people with leetcode questions where my co-interviewers made the candidate's confidence in presenting the correct answer the tipping point for hiring (with women mostly being targeted in this category). Bias can happen in any process, and with "culture" frequently being a consideration it's hard to tell what should be justified. Meritocracy in the tech industry is mostly a joke outside the overachieving outliers.
> IRL, this individual is a good but not exceptional engineer having worked with them on a small project.
Have you worked with them since they went through this regiment? Doing a DS&A coding problem regiment + system design will change you as an engineer. You might be surprised how good they've become.
Also they say they've forgotten. But if they were able to get that position, they probably could do medium level leetcode problems. So, I'd doubt they've forgotten all of it. I'm pretty sure they'd still be able to solve easy level problems, which most people can't solve off the cuff. They also probably still know the complexity of a bunch of essential backend operations (search, sort, array and hash lookups, tree & graph traversals, etc).
No home study grind on puzzle problems is a substitute for years of practical experience, which is what most teams actually hope for in their higher-level engineers.
The point is not that the friend didn't pick up some implicit knowledge or become a sharper engineer than they were before grinding, it's that by exploiting the screening strategy, they got placed into a job they're not truly qualified for.
Are they bright enough to fake it until they make it? Maybe, but that's not going to be the case for many of the countless placements that were made like this, and hints at why both product and software quality is in bad shape these days.
> Have you worked with them since they went through this regiment? Doing a DS&A coding problem regiment + system design will change you as an engineer. You might be surprised how good they've become.
Having done this prep, I can't tell if this comment is sarcasm. Building real systems makes you a good engineer. Maintaining systems over a long period of time makes you a good engineer. Working with other experienced engineers makes you a good engineer.
Doing this prep you do learn a few things along the way, but it's contrived. You already know what you need to learn, which is often the hard part on the job. It's works as a filter since the ability to learn concepts is important, but it's usefulness continues to trend downwards as it becomes more standardized.
> It doesn't matter if you've actually designed similar or more complex systems
You know you've designed more complex systems. The interview has no way of knowing if that's true, or you're an actor following the "I've designed complex systems" script.
And acting talent is arguably more common among the population than programming talent.
I think there are ways to tell by simply asking deep questions about the system someone claims to have built, critiquing the design decisions, diving into why this technology over that technology.
When I've hired or been a part of a hiring process, I always place emphasis on a candidate's past projects and ask deep technical questions around those. I also always review their GitHub repos if one is provided in the profile and I will ask them deep questions about their repos, why they chose this tech or that, interesting pieces of code, design tradeoffs they made, etc.
A moderately smart person was selected for a good job perhaps over many many better possible hires simply because that person had the leisure to learn the game. Inefficient. But nice for that individual, naturally.
It’s like the bar exam for lawyers. It bears absolutely no relationship whatsoever to the actual job and the work you will be doing. It’s a pointless ritual. And the point of the above story is that the person performed the ritual and then promptly forgot all the words once they succeeded. It illustrates the pointlessness.
It took forever. Literally tens of thousands of dollars of opportunity cost, with a lot of risk of having nothing to show for it.
The bummer is that you're right, it actually is worth even this much investment, because these companies do pay extremely well. But it's still horrendously inefficient, because the companies are getting a very small improvement to their signal to noise ratio, at this great cost (which, notably, they don't bear).
"Seemingly smart person getting a good job" is a criteria for filling headcount during a boom, but regresses the industry's maturity by placing inexperienced people into critical roles.
The problem with this story is that people need to work with and rely on this person for responsibilities they're not yet qualified to meet. In some cases, a "smart person" will be able to grow into the role, but the road there is long and messy, which becomes a frustration for colleagues, supervisors, clients, users etc.
Because of the prolonged boom we just went through, the industry -- especially at FAANG's -- is now saturated with smart, naive people trying to fake it until they make it, leading to a gross decline in quality and consistency compared to where we have been and might otherwise be.
Apparently they expect people to work for free for more than a month to learn material they then won't use.
In my mind that's a rather nasty practice.
Not that I'm complaining. I'm happy to pick up people that are good at computers but wouldn't be able to pass that hurdle, and probably wouldn't hire anyone that has.
although its a different paradigm that I’m used to, I agree with this general concept
if you provide utility to the market, you shouldn’t need credentials or a corporate ladder to climb to prove it, it should be immediate compensation no matter how disparate
so despite how coveted tech compensation packages are at this tier of company, a seemingly smart person getting a good job after doing a contrived aptitude test does meet that criteria. other people outside the field (and within) have difficulty passing it and don't have the cognitive ability to study for it, or the financial stability to prioritize studying for it
I also agree that a less contrived aptitude test would be better
They passed over all the people who didn't study hard enough on random word problems, and then practice those types of problems sufficiently such they could hammer out a solution in under 20 minutes. Multiple aspects of the "leetcode interview" require practice outside of work experience for many people, despite them being great problem solvers.
I took a similar approach -- took 6 weeks off doing interview prep, then concurrently applied to three FAANG employers. Got offers from all three, the worst of which was a 36% raise over my former non-FAANG job.
> They said that the hardest part was studying for the system design portion because they did not have experience with system design
Your friend is not alone, and I don't understand what separates us. Even when I was early career, system design was my favorite.
My biggest issue is performance anxiety when there's a concrete problem to solve that I've never seen before, so coding sessions are absolute hell. Inability to focus, forgetting approaches I've known forever, etc.
Ask me to do system design though, and it's generally freeform, I can talk about different approaches, deep dive on the fly, and draw a nice diagram or many nice diagrams, and interviewers love me and think I'm smart. I've had interviewers who were suspicious after my bad coding performance try to nail me on esoteric implementation details but because I have actual, hard-earned experience I can pretty much explain any approach in my niche and then tell you all the tradeoffs. I can also quickly code up examples, etc. Never had a bad feedback on this portion.
Something about the complete detachment of the coding problem from real life is such a huge mental block and I am continuing to fight it 15 years into my career, after dozens of similar interviews.
Interviewers would be better off focusing on data fundamentals: data modeling, validation, schema design, indexing, sharding/distribution strategies. When you screw this stuff up, it can be very, very hard to fix. The "problems" cascade and build up. Garbage in, garbage out. I've seen databases, in production, with no primary keys or indexes. It is absolutely bonkers.
Is "cramming" the same as "learning"? Is knowing the chemical reaction between an acid and baking soda the same as being a world class patissier? Many of these exercises are like "demonstrate that you can make this dough rise" but not actually considering "can you make an artful and delicious pastry"; the two are totally different.
The most surprising thing when asked about their experience was that most of the details have already been lost before even starting the role.
The problem is not everyone has the chance to work on super-scale systems. So what are you going to do? Weed out anyone with the potential to become great at that just because they've never had the chance to try?
Demonstrating the ability to learn, even if it's just scripted, should be good enough. Granted, they might get passed over by someone with actual experience, but the lack of real experience shouldn't be a deal breaker. It's fine as long as the employer has realistic expectations of the employee.
Systems Design is IMO the most useless part of those interviews.
Interviewers are just trying to find someone who would design a system the same way they did. They used a queue? They want you to add one. They used serverless? Better say the word "Lambda" somewhere. They hate serverless? Ooops better stick to regular servers.
In the end yeah, it's about following a script, but also reading the room.
I recently ran an interview process for a relatively senior eng role at a tiny startup. Because I believe different interview methods work better for different people, I offered everyone a choice:
1. Do a takehome test, targeted to take about 4 hours but with no actual time limit. This was a non-algorithmic project that was just a stripped-down version of what I'd spent the last month on in actual work.
2. Do an onsite pairing exercise in 2 hours. This would be a version of #1, but more of "see how far we get in 2 hours."
3. Submit a code sample of pre-existing work.
Based on the ire I've seen takehome tests get, I figured we'd get a good spread between all three, but amazingly, ~90-95% of candidates chose the takehome test. That matches my preference as a candidate as well.
I don't know if this generalizes beyond this company/role, but it was an interesting datapoint - I was very surprised to find that most people preferred it!
Interesting. I think I slightly prefer take homes, mostly because they're not as hectic. The problem ends up being, that I don't have any guarantee that the other side will spend any time on it. At least with an in person, I know that the company had to incur a significant cost so I know my time is less likely to be wasted. There's already a growing asymmetry with job searches, and this is one more trend that accelerates that. I still have a non technical friend who's annoyed with me for not completing a take home years ago, because, while the main ask was obviously trivial, there was enough slop in the directions that I could kill hours trying to decide if something was good enough to submit. On top of that it was expecting some fairly narrow set of techs, that, while not difficult to pick up, were not something I was interested in trying to learn enough about for a chance to maybe help some company that "couldn't find any technical talent".
> The problem ends up being, that I don't have any guarantee that the other side will spend any time on it.
This is a key thing for me. I consider it a moral obligation to give material feedback to anyone who does a takehome test for me - to invest at least some serious time evaluating it. Likewise, I would never give a takehome test until the candidate has at least had a phone screen with someone at the company and there's some level of investment on both sides.
On the other hand, I know a few junior devs right now who are submitting resumes and getting sent takehome tests off-the-bat. And, of course, after spending hours on those challenges, they get only form-letter rejections. I understand why companies do that - there's a glut of junior devs right now and any early-career role gets flooded with resumes that you need to somehow pare down - but I still consider it unconscionable.
I understand why you might not want to risk dealing with that.
I'd favor 1 and 2 over 3, because I feel that you understanding the problem I'm solving gives a better context to appreciate my approach.
If I take it home and hand it later, you can compare my solution to yours (or others) and better grasp my coding style, even if it's different to yours, since we worked on essentially the same task.
If we're working as a pair, you can observe my thought process. I can better express my reasoning for picking certain idioms.
With past code, there's almost no context. It's just code that was written for some other project. In my opinion, this can be very useful in preliminary steps, when I'm sending out my resume and you need to validate that I can indeed write code. In later stages, I think it can be impressive if I have a working app and can walk you down the code for a particular feature. But who has working software laying around? We all have code that works with a few hacks, sorta...
I'd favor 1 over 2, because it reduces the probability to blunder. Between spending 2 hours on-site with a higher risk to mess up (due to pressure, Murphy's law, etc) and take the assignment home to work on it for an extra 2 hours, with ample time to freely research whatever I don't understand. It's a no-brainer for me.
I'm your average skilled introverted engineer. But after more than a dozen years of experience and problem solving, I'd go with #2. I feel i'd be able to explain myself much more easily, have to do much less work, and probably have much more ways to impress the interviewers with face to face. I have also been on the receiving side of take-home tests and I know how hard it is to impress someone with those.
When I hire I don’t need someone to impress me, he has to present himself as able to do the job.
I think a lot of devs think they have to hire someone that impresses them and we end up with insane tests and adversarial interviews.
Yeah of course there will be people who will blow through the task much faster than others with better quality than average and they will impress me and they will get the offer in first place - but often times they already get multiple offers and after negotiation they will pick some higher offer and we have to get someone we can afford to do the job.
I was expecting "3. submit a code sample" to be the overwhelming winner here - did anybody choose that? Seems like a no-brainer, since it's already done...
The code I've written in personal projects have been pretty messy -- whereas code I've written professionally tend to be better (not perfect), but of course I can't share that code.
On the employer side, I would prefer the take home assignment over existing code, unless the existing code was super high quality or highly relevant to the company.
Work samples are a trap. Real problems are always way more hairy than contrived ones, which is not appreciated by people who were just introduced to the problem 30s ago. Since you wrote it while trying to actually accomplish something rather than polish up a perfect little nugget to impress someone, you probably spent way less time per line.
As others have pointed out, there are a lot of problems:
- Many engineers don't write code outside of work and so don't have much code they can show off without breaching their employer's trust
- Those that do often don't have recent code.
- Those that have recent, personal code to show off have often written it to accomplish some goal and the code isn't necessarily a great example they feel like showing off
Really, the only people I've seen have good code samples are those who do extensive open-source work. And for those people, I'm probably already aware of that work when I checked their resume + opened their github.
Interesting! I like the idea of choice, but as a hiring manager it makes my problem harder. How do you compare the results from different choices equitably? I find trying to compare candidates fairly to be quite difficult, even when they have the exact same interview.
Last time I did a coding interview for real, I had the choice of any programming language, and could choose between 3 different problems to solve. I liked that quite a bit, and was offered the job. Being able to choose Python, instead of, say, C++ in a time-bound interview almost feels like cheating.
> as a hiring manager it makes my problem harder. How do you compare the results from different choices equitably?
That makes sense, and it's the perspective that's being drilled into a lot of us. Implicit bias and all that. But in my experience, the comparison problem has never been that big of an issue in practice. I guess it depends on the hiring climate, but I'm much more familiar with spending a lot of time going through mediocre candidates and looking for someone who's "good enough". And this is when the recruiter is doing their job, and I understand why each candidate made it to the interview stage. Sure, sometimes it's a hot position (or rather, hot group to work for) and we get multiple good candidates, but then the decisionmaking process is more about "do we want A, who has deep and solid experience in this exact area; or B, who blew us away with their breadth of knowledge, flexibility, and creativity?" than something like "do we want A who did great on the take-home test but was unimpressive in person, or B whose solution to the take-home test was so-so but was clearly very knowledgeable and experienced in person?" The latter is in a hypothetical case where everyone did the same stuff, just to make it easier to compare, but even in that setup that's an uncommon and uninteresting choice. You're comparing test results and trying to infer Truth from it. Test results don't give a lot of signal or predictive power in the first place, so if you're trying to be objective by relying only on normalized scores, then you're not working with much of value.
Take home tests or whiteboard tests or whatever are ok to use as a filter, but people aren't points along a one-dimensional "quality" line. Use whatever you have to in order to find people that have a decent probability of being good enough, then stop thinking about how they might fail and start thinking about what it would look like if they succeed. They'll have different strengths and advantages. Standardizing your tests isn't going to help you explore those.
in one interview for a python job i was allowed to submit solutions to tests in pike which i had more experience with, but none of the devs at the company had ever seen. for python they had an automated testsuite but i remember my interviewer telling me that the devs were pouring over my test results to verify them manually while we continued other parts of the interview. i got the job.
Yeah, that was the explicit tradeoff: I'm losing the ability to compare apples-to-apples. I decided it was worth the risks associated with that - that I'd wind up with two candidates who picked different options and I couldn't decide between them because I got different signal between them. As it turned out, it didn't really come up - nearly everyone chose the takehome. On a larger scale, though, you'd definitely have to grapple with that.
That's because these three methodologies test different things. Take-home is the best representative, but it's heavily skewed by the prize. Whiteboarding only test for algorithms knowledge and a bit of problem-solving. Paired programming relies more on communication. And the last two add time pressure that rarely exists in real world scenario.
The fact is, it all depends on the person you need and the team they will be slotted in. But it seems that the ones who know the criteria is rarely involved in these matters.
That's why I offered the choice: different people do well under different circumstances. I figured that it'd allow me to hire a great dev who hates timed challenges or a great dev that hates take-home busywork. In practice, though, everyone like the take-homes, so ¯\_(ツ)_/¯
As for feeding it into an LLM, I go back and forth on that. With their current capabilities, the project is slightly too complex for that to work. You could get a lot of help from an AI, but you still have to put the pieces together yourself. And I'm fine with that! If AI tooling makes you a faster/better dev on a test, I'd expect it to do the same in real work.
Longer-term, though, I worry that LLMs still won't be able to "write the whole thing" for real work, but will be able to "write the whole thing" for takehome tests. And as such, I'll have to figure out how to handle that.
Why would you even do any of that for a senior role? I wouldn't waste my time with it, and it shows you don't know how to interview/evaluate for a senior position.
> Why would you even do any of that for a senior role?
I used to think like that ... then we hired a batch of 3 "senior" devs who could not do simple, everyday coding tasks, even in their ostensible languages of preference. All had come with exceptional resumés and personal recommendations.
So now, no one at any level one gets hired without demonstrating that they can at least write a fizzbuzz and commit it to a repository.
Now, I doubt that senior engineers in other fields have to do this sort of thing. But, most other engineering fields have licensing/accreditation with accompanying post-graduate academic curricula. We don't (but probably should).
I've come around to the idea that all people who will be expected to write code as part of their job need to be evaluated for their ability to write code as part of the interview process. I've seen too many people write convincing and impressive resumes without a single ounce of technical ability to their name.
Why wouldn’t you? How should one evaluate a senior position, in your opinion? Are you suggesting a senior dev shouldn’t need to demonstrate any coding skill? (If not, why not?) Or are you suggesting that there are other faster ways to show coding skills?
I exclusively give pairing interviews, usually just 1 hr. The variance between good and bad candidates is amazing, and you wouldn't guess at all from resumes or casual conversations. Most candidates have given me very positive feedback. Why wouldn't you want to be interviewed like this?
Because all evidence[0] I've found has shown that work-sample tests and repeatable, structured indicators are the best indicator of job performance. I understand that a lot of devs think they can get a feel for a candidate by just having a good ol' conversation with them, but the body of study on the subject says that that's just wrong. And so I have people do a task that's as close to the actual job as humanly possible and evaluate them on that.
Because for a senior role you need to hire a person with senior role skills, I suppose. Just N years of experience at X company doesn't provide that. I'd rather look at someone's github repo than talk to their reference tbh.
The more people online complain about coding interviews, the more confident I am that they are the absolute best way to filter candidates for a software development job. Across the industry there are way too many talkers/pretenders/meeting schedulers and not enough people who can roll up their sleeves, jump into the code and actually get stuff done. And this problem becomes worse at higher levels. You can bitch about it all you want, but you aren't owed that cushy $500K/yr FAANG job. If you can't get yourself to brush up on basic programming and write some for loops then companies will simply move on to someone who will.
I am good at passing these interviews and am at a faang (will be moving to another one this month). These interviews are useless and provide a false signal on problem solving skills and people's abilities to learn things. The interviews specifically don't test whether or not somebody can roll up their sleeves and jump into code, because if it did why do I as a new hire have to explain so much about software engineering and debugging practices to people who have been here so long?
If you've actually worked at a large company, you'd know that 90% of the real work is done by like 5% of people (maybe even less). If the interviews worked this ratio would be so much better.
Yea, I happen to hate coding challenges, but not because they're hard--because they bias towards "people who have time to do coding challenges". That said you're absolutely right about how many phonies are in the industry and coding tests are probably the best we have to weed them out.
The problem I see in a lot of cases is putting too much emphasis on solving the problem in the interview, instead of working through it. Many interviewers claim that they care more about the latter, but in practice failing at the former ends up disqualifying you.
There are physics textbooks and YouTube videos everywhere and yet we aren't all physics experts. Existence of knowledge and accessibility of information does not guarantee everyone can learn to do something and it especially doesn't guarantee that everyone can learn to do something well.
Typical table stakes prep work for a high paying job with a lot of competition.
I’m serious. It’s not even guaranteed you’ll get the job if you do the above but everyone who passed the big tech interviews will acknowledge they fucking studied for it. What do expect here?
The ease with which interviews allow the unscrupulous to inflate their abilities has been well-known for long enough for it to be metagamed and exploited for profit and now the effects have become abundantly apparent. Imposters institutionally infested in this manner across industries have reached a critical threshold where their fundamental lack of competency can no longer remain hidden.
The companies ask that candidates learn how to solve algorithm problems and the candidates do it.
I would call it “everyone playing a game they agreed to play by the rules they agreed to play with.”
And I don’t know what you mean by “it’s no longer about skill.” It still takes a lot of skill to be able to solve hard algorithm problems, even if you took a course on how to solve them and practiced solving them for 6 months.
When you audition for an orchestra they give you the sheet music ahead of time. It doesn’t mean you have no skill if you practice first instead of just showing up to sight read the music.
Even if we buy that leetcode is good in general, what about for the increasingly more dominant variant, the ML/AI engineer?
How can we even ban them using LLMs in their "ML-leetcode" problem when the problem is on LLM tokenization or something?
Like, if I lose a candidate because they're experts at using cursor + claude to do their coding (i.e. they solve it by writing 3 prompts and filling in 2 errors which are easily spotted in a total of 10 minutes), despite them having NeurIPS caliber publications and open source code, did I filter out a fake grifter, or did I filter a top tier candidate?
I just don't buy that leetcode style problems make any sense for anyone whose doing anything involving LLMs, and like it or not, increasingly large amounts of the cushy 500K/yr FAANG+ jobs will involve LLMs going forward.
the current system just allows people who understand the system and prep to beat the exam to succeed instead. yet you still find the phonies get through, so is it really solving the original problem then?
outside the geography where the salaries are an order of magnitude less, we yet go through the pain because of companies copying each other's approach.
Agreed. And a lot of people will say that you can just memorize a bunch of leetcode problems, but I've always created custom coding problems when I've acted as an interviewer so good luck finding the solutions online.
And frankly, if they're able to adapt a random leetcode problem to be able to solve the coding test that I give them, then that's exactly the kind of adaptability that I'm looking for in a prospective software engineer.
> This is like asking a Ruby developer to debug PHP as a test of flexibility.
Sounds like an OK test to me. Great (senior) developers should be able to do that kind of thing. Categorizing yourself exclusively as "a Ruby developer" is a career trap.
In the short term. And then the entire industry experiences some kind of a technological shift, as it will for many more cycles, and you're jobless as early as the first wave.
It's ridiculous how developers mindlessly accept that you should constantly be learning to keep yourself relevant, but keep it shallow by just jumping from one tool to another, instead of encouraging deeper knowledge of generalizable patterns that stay relevant across waves of technological disruption.
That's either a fun or terrible exercise depending on who is administering and how. However, if you said it's a Ruby role and the candidate is good enough to be picky, you may scare them off when they think your description of the role was a lie.
I agree. I've had this happen often enough on the job that it's not a totally made up example. And usually you'll be one of two or three people in the whole company who is able and willing.
Debugging old DSL vendor specific languages or code so old using, frameworks and standards long out of fashion and support, that they are half way to being a different language.
Adding support for some back ported features or patching security holes in an old client or legacy stacks.
Or at a big company we had some escrow code from a much smaller partner that we ended up becoming responsible for.
Often getting the environment setup for proper debugging is more work than anything.
I interviewed for a Scala role one time despite never having written it professionally. I suppose it's obscure enough that they couldn't afford to be too picky.
It was a pair programming exercise and so with some help from the interviewer and the IDE I was able to fumble through to a working result. I agree it was fun and educational.
Most of my development experience is in C/C++ and Python.
Know what I'd do if the interviewer asked me to debug PHP? Pretty much return the question:
"I've never used PHP. Are there logging macros/functions defined somewhere? Where do I see the output? syslog maybe? Is there a debugger of some sort I can use? How do I run each `piece` of code in isolation?"
(I am assuming the job listing did not explicitly mention PHP experience. If it did, both myself and the recruiter would absolutely deserve to fail me for this interview).
being able to do that without significant time constraints isn't that bad imho, but inside of an interview?! That's borderline laughable, for me, as you're only going to be filtering out people based on whether they have any previous PHP exposure or not.
> These high-stress, solo coding assignments don’t reflect the actual job. Instead, they put developers in situations they’d never face in the workplace, where collaboration and support are standard. When was the last time you had to debug an ancient codebase without documentation or help from a team?
I have had to do this, on numerous occasions. Sometimes there is a system that no one who built it is left, the documentation doesn't exist or can't be found, and that's it. This is not a great example. Being able to debug a program, follow requests around and see what's happening is a good skill.
What this example is really bad at though is that you pigeonhole yourself into developers. If you have a legacy php application, you are weeding out the java, python, ruby, go etc developers that do have those skills but don't know php and will be slower at it. Despite them possibly being stronger developers given a couple months of hands on php.
> A “4-hour” assignment can quickly turn into 8, 10, or even more, just to ensure it’s in great shape.
This I agree with. Suggested time can easily be gamed, so someone who has more time can look like a better candidate.
When I interviewed at Stripe, they had a "debug this!" question ready to go in a number of languages. (I think the list was something like: ruby, java, python, javascript, go, maybe one or two more.) I thought this was pretty great, but also seemed pretty labor intensive.
That is the way to do it, good on Stripe. Its definitely labor intensive, but I think that you either need to outsource it or be big enough that you can source the talent to build a half-dozen plus idiomatic and kind of big applications to make it a good test.
> When was the last time you had to debug an ancient codebase without documentation or help from a team?
Literally every day this week.
Documentation? The code was last updated in June, the documentation was last updated in January - of 2022.
The team? The half that didn't get laid off two years ago all quit for greener pastures. Sure, there's a team that owns this project - along with three others, all of which they've contributed maybe a dozen lines to over the past year, and 11 of those were dependency updates.
I have no issue with doing this on a job, but i have issue with it in interviews. Not everyone is able to be comfortable in a strange environment with strange people. Give me a damn take-home test and stop talking amongst yourselves while i'm at the whiteboard.
> When was the last time you had to debug an ancient codebase without documentation or help from a team?
> This is like asking a Ruby developer to debug PHP as a test of flexibility
> If the job requires specific tech skills, test those skills
Sounds like someone failed a coding challenge.
In all seriousness, you need to understand that companies rarely implement their hiring practices for the sake of it. It is usually the best their collective minds could come up with. Telling them to get rid of it without attempting to understand what problem they're trying to solve and without offering alternatives, is, to say the least, not a productive use of anyone's time.
I find in software development we try to reinvent the wheel all too often instead of borrowing practices from other professions. Let's have a look at what civil engineers have to go through to get hired at half the salary of a software dev, and let's incorporate that into our practice. That will make everyone a happy trooper !
> In all seriousness, you need to understand that companies rarely implement their hiring practices for the sake of it. It is usually the best their collective minds could come up with.
Unlikely. It's usually what the most senior person either read about or experienced in the past. In fact I'd say that it is highly unlikely that they did any introspection at all as to what they are actually trying to accomplish.
They just did it like everyone else because they believe exactly what you do -- that someone somewhere created the process with intention.
I feel like we've so lost the plot with tech hiring that we settled on what appears at best to be a local maxima.
Folks on this forum mostly won't remember but about 20 years ago the in-vogue way of interviewing software engineers was to ask "puzzle questions" (e.g. "I have 100 ball bearings and two are a different weight than the rest....") and "lateral thinking" questions ("why are manhole covers round?") because that's what they did at Microsoft and everyone copied Microsoft.
I'm told this is still the common style of interview for mechanical engineers, which says something about what it's like to work in that industry too.
> In fact I'd say that it is highly unlikely that they did any introspection at all as to what they are actually trying to accomplish.
Based on the observations I've made of hiring managers I've worked with, what they're trying to accomplish is to provide the appearance to HR that they at least attempted an unbiased hiring process. The result often resembles a "literacy test" and I suspect one has to be very intentional about avoiding that to practically avoid it.
Readit News
All the time. 300-400k SLOC in C++. Legacy in the sense that there were no tests of any kind. Little-to-no documentation. Solo developer at the tiny company. Fix bugs and add features while keeping the system available to the tens of thousands of users.
A more recent example: here’s a patch for a critical feature we need. It was written over a year ago. The original author isn’t available anymore. You can write the code from scratch or try to resurrect the patch against master.
Being able to jump into a project and lead people towards some goal is definitely a skill for senior developer positions. Yes, you generally have a team you can lean on and have the ability to do research and all that. But how do you show that you can do all that in an interview?
Agree with the conclusion that a good thing to test for is for problem-solving.
The tech side depends a lot on what you’re doing. Although it gets ridiculous and organizations get lazy with this part. You don’t need to be white boarding graph algorithms for a junior web developer role. If your application is a social networking role and you’re interviewing a senior developer or architect? Definitely. They’re going to be teaching this stuff and need to understand it at a deep level.
Last week I was debugging some similarly uncommented, undocumented, Go code from 2020 written by lunatics that is also a critical piece of the system.
It hurts.
Where are these dev jobs where _don't_ have to figure out some mysterious issue in a barely maintained GitHub repo semi-regularly?
Heck, I have to debug the stuff that some idiot (me) wrote six months ago and it might as well have been someone else who wrote it for all I remember about how to debug it
Edit: apparently this wasn't written clearly enough that I was the original dev that I'm having words with???
IMHO the willingness to "dig your way down" to solve a problem at the correct layer (rather than working around a bug or missing feature in a lower layer by adding post-hoc ameliorations in "your" code) is one of the major "soft skills" that distinguishes more senior engineers. This is one of those things that senior engineers see as "the natural thing to do", without thinking about it; and fixes that take this approach work subtly to ensure that the overall engineered system is robust to both changes up and down the stack, and to unanticipated future use-cases.
And contrariwise, fixes tending not to take this approach even when it'd be a very good idea, is one of the central ways that a codebase starts to "rot" when all the senior engineers quit or are let go/replaced with more-junior talent. When, for example, every input-parsing edge-case is "fixed" by adding one more naive validation regex in front of the "opaque, scary" parser — rather than updating the parser grammar itself to enforce those validation rules — your codebase is actively evolving into a Ball of Mud.
Of course, the tradeoff for solving problems at the "correct" layer, is that you/your company often ends up having to maintain long-lived, trivial, private hotfix-branch forks of various ecosystem software you use in your stack. More often than not, the upstreams of the software you use don't see your problem as a problem, and so don't want to take your fix. So you've just got to keep it to yourself.
(Funny enough, you could probably trivially measure an engineer's seniority through a tool that auths against their github profile and calculates the number of such long-lived trivial private hotfix branches they've ever created or maintained. Much simpler than a coding challenge!)
Jumping into an unknown codebase (which may be a library you depend on) and being able to quickly investigate, debug, and root cause an issue is an extremely invaluable skill in my experience
Acting as if this isn’t useful in the real world won’t help. The real world is messy, documentation is often missing or unreliable, and the person/team who wrote the original code might not be around anymore.
> Fix bugs and add features while keeping the system available to the tens of thousands of users.
Don't tens of thousands of users warrant more developers? Or having enough of a budget to work on tests, or other things to improve the developer experience and be able to work without lots of stress? That's unfortunate.
So not necessarily any budget for more than they did.
I thought the page was satire at first.
> All the time.
I guess the minor difference was that it was part of your paid job instead of a huge time sink to get hired.
This is the biggest issue I have with all these tasks is that they take precious time that could be utilized better. In my case: working on real open-source software instead of throwaway made-up problems. (Source: I had to spend 3 days to write a transaction engine toy once and then I didn't get the job for another reason.)
It wasn't that hard if you know the fundamentals (http, ssl). It was sorta fun and I got the job.
Imo it's more important that you can break down an existing best-in-class algorithm for x task and have your prospective dev answer questions about it/how they would use it, etc. Expecting every senior dev to be a best-in-class top mathematician as well as having a handle on the ever-changing language/library/ops/etc stacks is just crazy.
I'd never expect a senior dev on my teams to just know something like that, but I would expect that given a task they can do their research, find good candidates and understand the pros/cons of each approach. Because our power is in our ability to learn & use; the stuff that we already know is a bonus.
Not just all the time — right now in the other browser tabs I'm ignoring in favor of this one lol
Deleted Comment
So, definitely, I always need help debugging any kind of code (unless it's rather code that doesn't deal with product features).
You can, to a degree. If you can get the system under test you can make an assertion about how you think it works and see if it holds. If you know what the "law" is you can test whether the system calculates it according to that specification. You will learn something from making those kinds of assertions.
Working Effectively with Legacy Code by Michael Feathers goes into detail about exactly this process: getting untested, undocumented code people rely on into a state that it can be reliably and safely maintained and extended.
Depending on the situation I often recommend going further and use model checking. A language and toolbox like TLA+ or Alloy is really useful to get from a high-level, "what should the system do?" specification down to, "what does the system actually do?" The results are some times surprising.
You're right that at some level you do need to work with someone who does understand what the system should do.
But you can figure out what the system actually does. And that is de facto what the business actually does... as opposed to what they think it does, logic errors and all.
A good senior programmer, in my opinion, thinks above the code like this.
Update: interviewing for this kind of stuff is hard. Take-home problems can be useful for this kind of thing in the right context. What would you do differently?
agnostic programming flueny/software practices, including being comfortable debugging unfamiliar things or navigating uncharted waters without many domain experts to guide me, is the number one skill required in my role.
bonus points if i end up becoming an active maintainer to the project, or someone who can help other devs w it, the more times i dip into it.
Surprisingly, both jobs were fine too.
If you didn't solve your problem in 45 minutes with zero assistance from tools or Google or colleagues, were you immediately fired?
I mean, even if you have a team, you are hired to do specifically this kind of work. I'm confused what OP thinks a software job is if not EXACTLY this.
> All the time.
So you were forced to do it in a silo? No access to any type of documentation, whatsoever? Not able to refer to any C++ manuals or any other type of documentation? Not permitted to access the internet to look things up? Barred from using online forums to seek help?
The OP says: "A “4-hour” assignment". This is take-home, so candidates are free to use any C++ manuals, documentation, AI, online forums, whatever...
A partner of a friend quit their job earlier this year. They then took 4-6 weeks to prepare for each interview with Big Tech companies (4-6 weeks for Meta, 4-6 weeks for Stripe, etc.). Along the way, they also took random interviews just to practice and build muscle memory. They would grind leetcode several hours a day after researching which questions were likely to be encountered at each Big Tech.
This paid off and they accepted an offer for L6/staff at a MAANG.
Talked to them this week (haven't even started the new role) and they've already forgotten the details of most of what was practiced. They said that the hardest part was studying for the system design portion because they did not have experience with system design...but now made staff eng. at a MAANG. IRL, this individual is a good but not exceptional engineer having worked with them on a small project.
Wild; absolutely wild and I feel like explains a lot of the boom and bust hiring cycles. When I watch some of the system design interview prep videos, it's just a script. You'll go into the call and all you need to do is largely follow the script. It doesn't matter if you've actually designed similar or more complex systems; the point of the system design interview is apparently "do you know the script"?
Watch these two back to back at 2x speed and marvel at how much of this is executed like a script:
- https://www.youtube.com/watch?v=4_qu1F9BXow
- https://www.youtube.com/watch?v=_K-eupuDVEc
One of my hires was a physicist who didn't know any of the jargon and had a look of terror when she first started on our whiteboard problems. Once we led her to a comfortable spot and she got into it she started talking about tools she made to help with her physics work with zero dev knowledge (she was shocked when we called her tools software, she was like, but it wasn't REAL software, we were like hate to break it to you, but that was not only real software you wrote, but crazy impressive). The white board problems were a tool to highlight potential.
I get that these big companies just want drop in widgets not people and that is what they are searching for, I just think it's funny they are using something that was created for the exact opposite purpose.
Part of me is kind of glad I'm too old to be one of the cool kids and have no hope to land a job that does these sorts of code challenges.
Of course 95%+ of it will usually be useless in your real-life work, to solve most of those problems in the time given you need to know the problem and the optimal solution to it so basically memorization with little thinking.
In other words, anybody, regardless of what university they went to or what courses they took out what advantages or disadvantages they had, can learn this stuff in a couple of months if they have what it takes for the role. And because the skills are so standardized, the process is pretty differently objective.
It's not expected that they'll actually use the specific skills in the job. But it shows they can learn skills of that type and then perform them at a high level in a stressful interview situation. Which is a great signal for whether they can learn the skills needed for the specific project they wind up on and perform in a high stakes deadline scenario.
I'm not defending the system, but I am saying there's a clear logic behind it.
I know one manager who had issues with all three got hired at Google. So. Think of the poor HR person that will have to clean up that mess.
I've interviewed people with leetcode questions where my co-interviewers made the candidate's confidence in presenting the correct answer the tipping point for hiring (with women mostly being targeted in this category). Bias can happen in any process, and with "culture" frequently being a consideration it's hard to tell what should be justified. Meritocracy in the tech industry is mostly a joke outside the overachieving outliers.
Have you worked with them since they went through this regiment? Doing a DS&A coding problem regiment + system design will change you as an engineer. You might be surprised how good they've become.
Also they say they've forgotten. But if they were able to get that position, they probably could do medium level leetcode problems. So, I'd doubt they've forgotten all of it. I'm pretty sure they'd still be able to solve easy level problems, which most people can't solve off the cuff. They also probably still know the complexity of a bunch of essential backend operations (search, sort, array and hash lookups, tree & graph traversals, etc).
The point is not that the friend didn't pick up some implicit knowledge or become a sharper engineer than they were before grinding, it's that by exploiting the screening strategy, they got placed into a job they're not truly qualified for.
Are they bright enough to fake it until they make it? Maybe, but that's not going to be the case for many of the countless placements that were made like this, and hints at why both product and software quality is in bad shape these days.
Having done this prep, I can't tell if this comment is sarcasm. Building real systems makes you a good engineer. Maintaining systems over a long period of time makes you a good engineer. Working with other experienced engineers makes you a good engineer.
Doing this prep you do learn a few things along the way, but it's contrived. You already know what you need to learn, which is often the hard part on the job. It's works as a filter since the ability to learn concepts is important, but it's usefulness continues to trend downwards as it becomes more standardized.
You know you've designed more complex systems. The interview has no way of knowing if that's true, or you're an actor following the "I've designed complex systems" script.
And acting talent is arguably more common among the population than programming talent.
When I've hired or been a part of a hiring process, I always place emphasis on a candidate's past projects and ask deep technical questions around those. I also always review their GitHub repos if one is provided in the profile and I will ask them deep questions about their repos, why they chose this tech or that, interesting pieces of code, design tradeoffs they made, etc.
The bummer is that you're right, it actually is worth even this much investment, because these companies do pay extremely well. But it's still horrendously inefficient, because the companies are getting a very small improvement to their signal to noise ratio, at this great cost (which, notably, they don't bear).
The problem with this story is that people need to work with and rely on this person for responsibilities they're not yet qualified to meet. In some cases, a "smart person" will be able to grow into the role, but the road there is long and messy, which becomes a frustration for colleagues, supervisors, clients, users etc.
Because of the prolonged boom we just went through, the industry -- especially at FAANG's -- is now saturated with smart, naive people trying to fake it until they make it, leading to a gross decline in quality and consistency compared to where we have been and might otherwise be.
In my mind that's a rather nasty practice.
Not that I'm complaining. I'm happy to pick up people that are good at computers but wouldn't be able to pass that hurdle, and probably wouldn't hire anyone that has.
if you provide utility to the market, you shouldn’t need credentials or a corporate ladder to climb to prove it, it should be immediate compensation no matter how disparate
so despite how coveted tech compensation packages are at this tier of company, a seemingly smart person getting a good job after doing a contrived aptitude test does meet that criteria. other people outside the field (and within) have difficulty passing it and don't have the cognitive ability to study for it, or the financial stability to prioritize studying for it
I also agree that a less contrived aptitude test would be better
Your friend is not alone, and I don't understand what separates us. Even when I was early career, system design was my favorite.
My biggest issue is performance anxiety when there's a concrete problem to solve that I've never seen before, so coding sessions are absolute hell. Inability to focus, forgetting approaches I've known forever, etc.
Ask me to do system design though, and it's generally freeform, I can talk about different approaches, deep dive on the fly, and draw a nice diagram or many nice diagrams, and interviewers love me and think I'm smart. I've had interviewers who were suspicious after my bad coding performance try to nail me on esoteric implementation details but because I have actual, hard-earned experience I can pretty much explain any approach in my niche and then tell you all the tradeoffs. I can also quickly code up examples, etc. Never had a bad feedback on this portion.
Something about the complete detachment of the coding problem from real life is such a huge mental block and I am continuing to fight it 15 years into my career, after dozens of similar interviews.
People generally don't even want you to go deep on anything, but you're can't stay to shallow.
It's just bsing, yes your absolutely right, it feels like memorizing a script and saying the right words at the right time
The most surprising thing when asked about their experience was that most of the details have already been lost before even starting the role.
Demonstrating the ability to learn, even if it's just scripted, should be good enough. Granted, they might get passed over by someone with actual experience, but the lack of real experience shouldn't be a deal breaker. It's fine as long as the employer has realistic expectations of the employee.
Deleted Comment
Interviewers are just trying to find someone who would design a system the same way they did. They used a queue? They want you to add one. They used serverless? Better say the word "Lambda" somewhere. They hate serverless? Ooops better stick to regular servers.
In the end yeah, it's about following a script, but also reading the room.
If he was even considered for L6 staff, then his on resume credentials already justified him at L5 Senior or L6 Staff.
The interview performance is what pushed him over the edge into L6.
It seems like you think they don't deserve a staff level position?
1. Do a takehome test, targeted to take about 4 hours but with no actual time limit. This was a non-algorithmic project that was just a stripped-down version of what I'd spent the last month on in actual work.
2. Do an onsite pairing exercise in 2 hours. This would be a version of #1, but more of "see how far we get in 2 hours."
3. Submit a code sample of pre-existing work.
Based on the ire I've seen takehome tests get, I figured we'd get a good spread between all three, but amazingly, ~90-95% of candidates chose the takehome test. That matches my preference as a candidate as well.
I don't know if this generalizes beyond this company/role, but it was an interesting datapoint - I was very surprised to find that most people preferred it!
This is a key thing for me. I consider it a moral obligation to give material feedback to anyone who does a takehome test for me - to invest at least some serious time evaluating it. Likewise, I would never give a takehome test until the candidate has at least had a phone screen with someone at the company and there's some level of investment on both sides.
On the other hand, I know a few junior devs right now who are submitting resumes and getting sent takehome tests off-the-bat. And, of course, after spending hours on those challenges, they get only form-letter rejections. I understand why companies do that - there's a glut of junior devs right now and any early-career role gets flooded with resumes that you need to somehow pare down - but I still consider it unconscionable.
I understand why you might not want to risk dealing with that.
I'd favor 1 and 2 over 3, because I feel that you understanding the problem I'm solving gives a better context to appreciate my approach.
If I take it home and hand it later, you can compare my solution to yours (or others) and better grasp my coding style, even if it's different to yours, since we worked on essentially the same task.
If we're working as a pair, you can observe my thought process. I can better express my reasoning for picking certain idioms.
With past code, there's almost no context. It's just code that was written for some other project. In my opinion, this can be very useful in preliminary steps, when I'm sending out my resume and you need to validate that I can indeed write code. In later stages, I think it can be impressive if I have a working app and can walk you down the code for a particular feature. But who has working software laying around? We all have code that works with a few hacks, sorta...
I'd favor 1 over 2, because it reduces the probability to blunder. Between spending 2 hours on-site with a higher risk to mess up (due to pressure, Murphy's law, etc) and take the assignment home to work on it for an extra 2 hours, with ample time to freely research whatever I don't understand. It's a no-brainer for me.
When I hire I don’t need someone to impress me, he has to present himself as able to do the job.
I think a lot of devs think they have to hire someone that impresses them and we end up with insane tests and adversarial interviews.
Yeah of course there will be people who will blow through the task much faster than others with better quality than average and they will impress me and they will get the offer in first place - but often times they already get multiple offers and after negotiation they will pick some higher offer and we have to get someone we can afford to do the job.
I was expecting "3. submit a code sample" to be the overwhelming winner here - did anybody choose that? Seems like a no-brainer, since it's already done...
On the employer side, I would prefer the take home assignment over existing code, unless the existing code was super high quality or highly relevant to the company.
As others have pointed out, there are a lot of problems:
- Many engineers don't write code outside of work and so don't have much code they can show off without breaching their employer's trust
- Those that do often don't have recent code.
- Those that have recent, personal code to show off have often written it to accomplish some goal and the code isn't necessarily a great example they feel like showing off
Really, the only people I've seen have good code samples are those who do extensive open-source work. And for those people, I'm probably already aware of that work when I checked their resume + opened their github.
Deleted Comment
Last time I did a coding interview for real, I had the choice of any programming language, and could choose between 3 different problems to solve. I liked that quite a bit, and was offered the job. Being able to choose Python, instead of, say, C++ in a time-bound interview almost feels like cheating.
That makes sense, and it's the perspective that's being drilled into a lot of us. Implicit bias and all that. But in my experience, the comparison problem has never been that big of an issue in practice. I guess it depends on the hiring climate, but I'm much more familiar with spending a lot of time going through mediocre candidates and looking for someone who's "good enough". And this is when the recruiter is doing their job, and I understand why each candidate made it to the interview stage. Sure, sometimes it's a hot position (or rather, hot group to work for) and we get multiple good candidates, but then the decisionmaking process is more about "do we want A, who has deep and solid experience in this exact area; or B, who blew us away with their breadth of knowledge, flexibility, and creativity?" than something like "do we want A who did great on the take-home test but was unimpressive in person, or B whose solution to the take-home test was so-so but was clearly very knowledgeable and experienced in person?" The latter is in a hypothetical case where everyone did the same stuff, just to make it easier to compare, but even in that setup that's an uncommon and uninteresting choice. You're comparing test results and trying to infer Truth from it. Test results don't give a lot of signal or predictive power in the first place, so if you're trying to be objective by relying only on normalized scores, then you're not working with much of value.
Take home tests or whiteboard tests or whatever are ok to use as a filter, but people aren't points along a one-dimensional "quality" line. Use whatever you have to in order to find people that have a decent probability of being good enough, then stop thinking about how they might fail and start thinking about what it would look like if they succeed. They'll have different strengths and advantages. Standardizing your tests isn't going to help you explore those.
Now add this to the fact that a vast majority of your applicants are going to be feeding your assignment directly into an assistive LLM.
As an interviewer, you really can't win.
- If you have a take-home test, people will either complain that it's too involved, and time consuming.
- If you do whiteboarding, people will claim that it's not representative of the actual job.
- If you do paired programming, people will claim it's unfair to those who don't test well under pressure.
The fact is, it all depends on the person you need and the team they will be slotted in. But it seems that the ones who know the criteria is rarely involved in these matters.
As for feeding it into an LLM, I go back and forth on that. With their current capabilities, the project is slightly too complex for that to work. You could get a lot of help from an AI, but you still have to put the pieces together yourself. And I'm fine with that! If AI tooling makes you a faster/better dev on a test, I'd expect it to do the same in real work.
Longer-term, though, I worry that LLMs still won't be able to "write the whole thing" for real work, but will be able to "write the whole thing" for takehome tests. And as such, I'll have to figure out how to handle that.
I used to think like that ... then we hired a batch of 3 "senior" devs who could not do simple, everyday coding tasks, even in their ostensible languages of preference. All had come with exceptional resumés and personal recommendations.
So now, no one at any level one gets hired without demonstrating that they can at least write a fizzbuzz and commit it to a repository.
Now, I doubt that senior engineers in other fields have to do this sort of thing. But, most other engineering fields have licensing/accreditation with accompanying post-graduate academic curricula. We don't (but probably should).
I exclusively give pairing interviews, usually just 1 hr. The variance between good and bad candidates is amazing, and you wouldn't guess at all from resumes or casual conversations. Most candidates have given me very positive feedback. Why wouldn't you want to be interviewed like this?
https://www.researchgate.net/publication/283803351_The_Valid...
If you've actually worked at a large company, you'd know that 90% of the real work is done by like 5% of people (maybe even less). If the interviews worked this ratio would be so much better.
"I want to work for your company"
"Ok, prove yourself"
"Sorry, I don't have time"
How do you expect that conversation to go from there? If you don't have time then make time. It isn't anyone's problem but your own.
- Cracking the coding interview.
- Elements of programming interviews in Java|Python|whatever...
- leetcode & other sides with paid premium subscriptions...
- mock interview bootcamps...
It's no longer about skill, it's only about gaming the system.
LLMs are another great example
I’m serious. It’s not even guaranteed you’ll get the job if you do the above but everyone who passed the big tech interviews will acknowledge they fucking studied for it. What do expect here?
The companies ask that candidates learn how to solve algorithm problems and the candidates do it.
I would call it “everyone playing a game they agreed to play by the rules they agreed to play with.”
And I don’t know what you mean by “it’s no longer about skill.” It still takes a lot of skill to be able to solve hard algorithm problems, even if you took a course on how to solve them and practiced solving them for 6 months.
When you audition for an orchestra they give you the sheet music ahead of time. It doesn’t mean you have no skill if you practice first instead of just showing up to sight read the music.
TFA: Coding interviews are a standard way of hiring.
Also you: “Too many people in the software development industry don’t know what they’re doing.”
The lack of cognitive dissonance is remarkable.
GP is of the opinion that coding interviews filter out people who can't code.
How can we even ban them using LLMs in their "ML-leetcode" problem when the problem is on LLM tokenization or something?
Like, if I lose a candidate because they're experts at using cursor + claude to do their coding (i.e. they solve it by writing 3 prompts and filling in 2 errors which are easily spotted in a total of 10 minutes), despite them having NeurIPS caliber publications and open source code, did I filter out a fake grifter, or did I filter a top tier candidate?
I just don't buy that leetcode style problems make any sense for anyone whose doing anything involving LLMs, and like it or not, increasingly large amounts of the cushy 500K/yr FAANG+ jobs will involve LLMs going forward.
outside the geography where the salaries are an order of magnitude less, we yet go through the pain because of companies copying each other's approach.
And frankly, if they're able to adapt a random leetcode problem to be able to solve the coding test that I give them, then that's exactly the kind of adaptability that I'm looking for in a prospective software engineer.
Sounds like an OK test to me. Great (senior) developers should be able to do that kind of thing. Categorizing yourself exclusively as "a Ruby developer" is a career trap.
And a lucrative one at that.
It's ridiculous how developers mindlessly accept that you should constantly be learning to keep yourself relevant, but keep it shallow by just jumping from one tool to another, instead of encouraging deeper knowledge of generalizable patterns that stay relevant across waves of technological disruption.
No engineer that makes 7 figures calls themselves a ruby developer with the exception of DHH.
Debugging old DSL vendor specific languages or code so old using, frameworks and standards long out of fashion and support, that they are half way to being a different language.
Adding support for some back ported features or patching security holes in an old client or legacy stacks.
Or at a big company we had some escrow code from a much smaller partner that we ended up becoming responsible for.
Often getting the environment setup for proper debugging is more work than anything.
But yes, it's a good test for a senior+.
Interviewer: Now reverse this array.
Me: OK, in Python that would be array.reverse(), or reversed(array). I bet JS has one of those, probably the .reverse method.
Interviewer: Great guess!
That was genuinely fun. I came out of it feeling like I'd learned a few things, and the other person got to see how I'd reason about a new problem.
It was a pair programming exercise and so with some help from the interviewer and the IDE I was able to fumble through to a working result. I agree it was fun and educational.
Know what I'd do if the interviewer asked me to debug PHP? Pretty much return the question:
"I've never used PHP. Are there logging macros/functions defined somewhere? Where do I see the output? syslog maybe? Is there a debugger of some sort I can use? How do I run each `piece` of code in isolation?"
(I am assuming the job listing did not explicitly mention PHP experience. If it did, both myself and the recruiter would absolutely deserve to fail me for this interview).
I have had to do this, on numerous occasions. Sometimes there is a system that no one who built it is left, the documentation doesn't exist or can't be found, and that's it. This is not a great example. Being able to debug a program, follow requests around and see what's happening is a good skill.
What this example is really bad at though is that you pigeonhole yourself into developers. If you have a legacy php application, you are weeding out the java, python, ruby, go etc developers that do have those skills but don't know php and will be slower at it. Despite them possibly being stronger developers given a couple months of hands on php.
> A “4-hour” assignment can quickly turn into 8, 10, or even more, just to ensure it’s in great shape.
This I agree with. Suggested time can easily be gamed, so someone who has more time can look like a better candidate.
Literally every day this week.
Documentation? The code was last updated in June, the documentation was last updated in January - of 2022.
The team? The half that didn't get laid off two years ago all quit for greener pastures. Sure, there's a team that owns this project - along with three others, all of which they've contributed maybe a dozen lines to over the past year, and 11 of those were dependency updates.
> This is like asking a Ruby developer to debug PHP as a test of flexibility
> If the job requires specific tech skills, test those skills
Sounds like someone failed a coding challenge.
In all seriousness, you need to understand that companies rarely implement their hiring practices for the sake of it. It is usually the best their collective minds could come up with. Telling them to get rid of it without attempting to understand what problem they're trying to solve and without offering alternatives, is, to say the least, not a productive use of anyone's time.
I find in software development we try to reinvent the wheel all too often instead of borrowing practices from other professions. Let's have a look at what civil engineers have to go through to get hired at half the salary of a software dev, and let's incorporate that into our practice. That will make everyone a happy trooper !
Unlikely. It's usually what the most senior person either read about or experienced in the past. In fact I'd say that it is highly unlikely that they did any introspection at all as to what they are actually trying to accomplish.
They just did it like everyone else because they believe exactly what you do -- that someone somewhere created the process with intention.
I feel like we've so lost the plot with tech hiring that we settled on what appears at best to be a local maxima.
I'm told this is still the common style of interview for mechanical engineers, which says something about what it's like to work in that industry too.
Based on the observations I've made of hiring managers I've worked with, what they're trying to accomplish is to provide the appearance to HR that they at least attempted an unbiased hiring process. The result often resembles a "literacy test" and I suspect one has to be very intentional about avoiding that to practically avoid it.
Deleted Comment