Assuming that a company does not look for candidates who are naturally good at ICPC-type of questions or geniuses who can come up with amazing algorithms in a matter of minutes, there is actually a different way to do coding interview: just give a high-level description of a sophisticated enough algorithm to a candidate and ask the candidate to turn that into code. I think it strikes a good balance between depth in CS and the coding abilities. This type of interview is similar to what engineers do at work too. We rarely invent new algorithms, but we do read white papers or blog entries or books to pick an algorithm to implement.
There are many variations in questions too: search a tree or graph with some customized criteria, using a double buffer to implement a tree-style prefix scan, implementing an integer multiplication with unlimited digits, some streaming algorithm, tree-walking to transform one tree to another, a simplified skip list, the options are unlimited. A good candidate tends to grasp the high-level concepts quickly (and they can ask questions), and is quick to convert intuition into working code. I find that there is a strong positive correlation between the performance in work and the performance in such coding interviews.
I still don't get why such questions are even asked as most jobs I've ever had not even remotely touched those and I've touched quite a few industries, technologies and types of companies.
To me, the value of a software engineer is to ask questions, make hypotheses and be able to iterate quickly. Balancing trees, leetcode and other algorithmic stuff on the spot sounds like bringing the dreadful education system structure to the real world.
Also if a senior person can't in 30/45 min of talking with someone figure out the general experience level then the problem is them, really.
> if a senior person can't in 30/45 min of talking with someone figure out the general experience level then the problem is them, really.
This is why I've always been so confused. Why is the software engineering interview wildly different from the traditional engineering interview (seniors sit down with candidates and discuss how to solve a relevant proxy problem the team is currently undergoing (has the side benefit of making the interview process potentially fruitful if you don't go with that candidate. This can be (and sometimes is) abused though)). I mean... we all speak the same language, and that isn't standard English... right?
> I still don't get why such questions are even asked
The thesis is not that these exercises are representative of work but rather predictive of performance.
Sales has a similar phenomenon with sports. While there is no athleticism involved in selling, many believe history in competitive sports to be a positive predictor of sales success.
---
You can reasonably argue whether leetcode accomplishes this well or poorly, but...
Always remember that purpose of an interview (for an employer) is to predict performance. So you are looking for the resume screening+interview process that most accurately assesses that.
> Also if a senior person can't in 30/45 min of talking with someone figure out the general experience level then the problem is them, really.
OK take it as read that your posturing has succeeded and we all agree that you're a brilliant interpersonal genius and the rest of us are all useless chumps. What then? The rest of us still need to interview and make hiring recommendations. Or are you suggesting that employers should fire anyone who lacks your magical talent?
Personally I think such questions have three values:
- Future proof. Unless I work for an outsourcing company, sooner or later I will want to push the envelope, or so I hope I do. And to push the envelop, one needs good CS fundamentals (maybe there are some exceptions in some specialized field). Think about React. It's a JS framework, yet to invent it one needs to understand at least compiler and graph.
- Geekiness/talent filter. The same reason that the nascent Google and Microsoft and any elite companies like Jane Street asked Putnam questions, Martin Gardner questions, ICPC questions, and clever probability puzzles. Whether it's a good idea is debatable, but at least those companies want to have such type of people and they were hugely successful. Note the word filter: people who pass the interview may not be good, but failing the interview means the candidate may not be smart or geeky enough. Again, I'm not endorsing the idea but just exploring the motivation of the interview policies.
- Information density. Assuming a company does want to time box a coding interview in an hour, it will be hard to come up with realistic question without spending too much time on the context. On the other hand, it's relatively easy to come up with a data structure/algorithm question that packs enough number of abstractions and inspection points to examine one's programming skill.
> Also if a senior person can't in 30/45 min of talking with someone figure out the general experience level then the problem is them, really.
Actually most of them, including the really inexperienced juniors have 'figured' you out in less that 15 minutes, or at least they have decided whether to hire you or not in 15 minutes. But they have to put on a charade of being fair.
Also a 'white' older male is the least preferred even if he is smarter compared to all females and the minorities that are being interviewed as long as they are not terrible. Biases galore.
>I still don't get why such questions are even asked as most jobs I've ever had not even remotely touched those and I've touched quite a few industries, technologies and types of companies.
I've had to work on tree traversal stuff multiple times in my life, anything low level GUI related will work with trees a ton.
I've also had to work with hash tables directly, and with memory caching layers.
I really should learn to write a proper parser, as I've had to write parsers multiple times now and they are always an ugly hack job.
At some of my past jobs and the current one, this kind of algorithmic knowledge was important to build features that were differentiators in the market. As much as people love to pretend, not every single possible solution is in a library. Sometimes you're the one building the library.
It doesn't have to be leetcode, but candidates should at least be able to produce some code that doesn't come from the README of their favourite framework.
Also, talking for 30/45 mins can be enough, but it produces false positives when you have people coaching candidates. I've had people completely acing interviews that it felt like the perfect candidate. Well, it was rehearsed. When I asked for a fizz-buzz type question they completely messed it up.
Unless you have data to show that your "different way" produces better results, your idea sounds exactly the same as every other idea, which is basically garbage.
Lots of people have lots of ideas on what makes a great interview question, but none of them are backed by data. LeetCode-style algo questions USED to be an indicator of intelligence as per Google's investigation, but now it's so heavily gamed that I doubt there's any signal behind it anymore.
Someone needs to spend time and effort and money and actually do random controlled tests to see if people who pass a particular type of test actually make good employees. But so far no one has done this, except Google as far as I've seen. But even now, I think there's no evidence that even algorithm questions are any indicator, given what I've seen.
Don't discard good anecdotal evidence because there's no proper randomly controlled trial. If you're experienced/senior in the industry you probably worked with dozens of developers closely, with hundreds less closely, you've seen many projects and designs, you've interviewed and was involved in the hiring of dozens of people.
This random controlled trial you're talking about is so hard to do because there are infinite confounders. How do you even measure success? After how long? How do you separate politics from technical ability?
In reality people above a certain bar are able to make some contribution to most projects. You can't and don't need to staff every project with superstars. The superstars are rare but are not hard to identify. They'll have the track record, reputation and the skills.
Another thing to consider is that the best outcome of interviews depends on the quality of the pipeline into those interviews. If you can't get good employees to apply you won't get good employees hired. Pay well and build a company that people want to work for, and then your hiring is going to be easier.
> Unless you have data to show that your "different way" produces better results, your idea sounds exactly the same as every other idea, which is basically garbage.
Do you have evidence that the standard coding interview works? (There's evidence that it doesn't)
I'm with you that the claim might be too strong to say "this is the way" but that's because I'm of the (very strong) opinion that interviewing is an extremely fuzzy process and there are no clear cut metrics to measure one's abilities in a significantly meaningful way. a̶l̶l̶ ̶m̶o̶d̶e̶l̶s̶ ̶a̶r̶e̶ ̶w̶r̶o̶n̶g̶ ̶b̶u̶t̶ ̶s̶o̶m̶e̶ ̶a̶r̶e̶ ̶u̶s̶e̶f̶u̶l̶ There are useful interviewing methods but certainly not "the best" method. Trying to just mark checkboxes only leads to mediocre results. The reason we're generally okay with this is because we more often than not don't need rockstars and it doesn't make sense to put a lot of time and energy into this process when we get sufficient results through lazy methods.
FWIW, a typical (non-software) engineering job really just involves high level discussions like the OP suggests but even without the implementation. It is generally about seeing how the person thinks/problem solves and looking at problems they solved in the past. It isn't resource intensive and good enough. Because truth is, you don't know what someone is like as an employee until they are an employee (and even then this is fuzzy)
> LeetCode-style algo questions USED to be an indicator of intelligence as per Google's investigation, but now it's so heavily gamed that I doubt there's any signal behind it anymore.
I have a different take on this. I ask a fairly simple coding question and have them write code. No fancy algorithms to memorize or competition style coding.
I want to see if they can think in code. If they can translate something simple to written code. There's always a chance to have a conversation as well to probe different aspects of their understanding.
A few of those by a few people and I think you get a pretty good read.
Mix in some questions about their work and/or some general knowledge. I've also given people code to read (real world code) and explain to me.
For Algorithms and Data Structures I can look at their grades if they're fresh out of school. But we all know most of our work is not that (someone's gonna show up on the thread and say differently - I'm sure ;) ). If you can think in code and have the right mental models implementing an algorithm from a book isn't hard if you really have to.
I used that approach too for a beginning frontend dev: see if they can get a solution to straight-forward problems, and ask them to explain it afterwards. They got all the time they needed, and a clean laptop with wifi. Two candidates came out well, two others didn't. That was a bit shocking, given the low bar.
But it won't cut it for a senior. Such people should be able to answer a wide range of questions, including some algorithm and data structure stuff. Not that they have to be able to code a min-max-heap from scratch, but they have to know what's out there, how to use it, and how to keep the overview of an entire project. That's not going to be evident from a coding interview.
And even for Google, leetcode has become noise because people simply cram the questions. When Microsoft started to use leetcode-style interviews, there were no interview site. So, people who aced the interviews were either naturally talented or were so geeky that they devoured math and puzzle books. Unfortunately, we have lost such signals nowadays.
A typical engineer doesn't begin their day thinking "I should implement a streaming algorithm" out of nowhere (and if they do, they can always seek reference). They analyze a (typically underspecified) problem, figure out the constraints, and then start thinking about a solution. Note that both processes, analysis and solution, can take hours, days, weeks or months!
Coding interviews have everything backwards by providing synthetic solutions to mostly out-of-context problems.
For example, interviewers could:
- Share some portion of a codebase with the candidate prior to the interview
- Go over the code with the candidate
That's it. You will know really fast who's senior and who's never wrote software. It can't be faked. And it can be adapted to mostly any positions (backend, frontend, devops, architecture, whatever).
My approach is to ask the candidate to show me some code they've written ( could be for class project, but one requirement for me to interview them is that they have a GitHub) and explain the choices they made or other anecdotes that surround the project.
Good idea. But the focus on algorithms is maybe overkill for at least 80% to 90% of companies. Perhaps lay out some subset of the business requirements or problems the company is working on and ask them to turn that into working code is more suitable
My attitude towards code interviews is to politely decline them and encourage people best of luck hiring a junior developer; because that's obviously what they are looking for. I'm closing in on 50, so not that junior anymore. If anyone has any doubts about my coding abilities after reading my CV, browsing my Github repos, and talking to me, then it's not going to work and we can both save ourselves some time. I've stopped responding to recruiters as I rarely see any evidence of them even being capable of doing these simple things.
I've hired and interviewed a lot of people over the years. Mostly without the help of recruiters. I like to think I'm pretty good at that, actually. I always look for people that are eager to learn. I care more about what they don't know than what they can regurgitate because they did lots of silly online prep. If people are young and fresh out of school, my assumption is they are going to have to learn a lot in a hurry. So I look for people that are curious, eager, and have an open mind. The best way to do that is to get them slightly out of their comfort zone and talking about how they would tackle things. The signs I look for here are enthusiasm, ability to surprise with good answers, etc.
This generally does not involve code interviews; though I might ask some targeted questions about how they would deal with certain things in languages, frameworks, etc. they list as being proficient in. I've of course worked for some people that insisted on me organizing code interviews and it's a chore and IMHO you learn absolutely nothing that isn't obvious if you just casually talk to people for 30 minutes or so. I usually argue against such things and prefer just talking to people directly.
I totally understand your point of view, but looking at it from the other side it’s not as simple. We had candidates with 10+ years of experience in their resume, talking to them it seemed they know what they’re doing, they showed some of the code they’ve supposedly written. Then they got hired and it turned out they can’t code - their PRs are below junior level, constant bugs, communication is abysmal, they overshoot their estimations by 3-4x, etc.
Thus happened to us twice, and after we introduced a simple live coding session to our interviews where you have to implement a simple real world web component (for a frontend position) suddenly the problem of bad hires disappeared almost entirely (you can’t judge someone’s character during a short interview but that’s another issue).
We recently interviewed a candidate, with 7 years of experience at a FAANG company, fail a really basic coding interview and we don't even make you write code.
I have no idea what these people expect. We ask coding questions, we have basic programming skills as a requirement, what do they think will happen if they get the job? Do they expect to just magically learn the required skills or hope that nobody will notice that they're not able to close their tasks?
The IT job market is a two sided lemon market, people may lie on their CV but companies can be very abusive as well. If you don't do what OP suggests, people will not respect you. I've had five developers join my interview call to see if I can implement a palindrome checker. This is proof that non of them bothered to read my CV and check out my github projects. This is not respectful, and you have to stand your ground against such nonsens. Let them now this is not what you expect from people working for you and and that you expect better from them in the future. If they don't like that you won't have lost anything.
same experience, same conclusion. I used to hire without coding interviews, i don't anymore.
Although i see it as a way for the candidate and i to talk about code in general, and assess his level. No as a simple barrier.
i hired a candidate that completely failed a code interview because he was super nervous, but just talking about the problem made me quite sure he was actually good.
Just by introducing coding interview, how would you know they will not overshoot estimations by 3-4x? It's not one person's job, normally estimates are done by the entire team
FWIW I always ask some really super simple coding questions in interviews, even for really senior people with apparently stellar CVs. Let them pick their language or use psuedocode or whatever.
It's surprising how many 'senior' engineers are actually BSers who will slow you down or derail you completely rather than speed you up and you need to spot them because they will excel at getting through the non-technical interview filtering!
Also, I'm interested in how you explore things and explain things. I'm not actually interested in acquiring an implementation of FizzBuzz or whatever. I just want you to show me that you 'get' it and then we can get on to the interesting stuff like 'tell me about your last project' etc.
So don't be too hasty to think the people doing technical interviews are idiots thinking devs are interchangeable cogs etc.
> Also, I'm interested in how you explore things and explain things. I'm not actually interested in acquiring an implementation of FizzBuzz or whatever. I just want you to show me that you 'get' it and then we can get on to the interesting stuff like 'tell me about your last project' etc.
Sounds like you'd more valuably/realistically get that from the discussion of a previous project though? How it works, or something interesting they had to figure out, etc.?
> don't be too hasty to think the people doing technical interviews are idiots
I don't think anyone has a problem with technical interviews? At least I agree that's not reasonable. That doesn't have to mean 'coding' though, you can ask about how they'd approach a particular problem, quiz on some fundamental knowledge in a fizzbuzz sort of way, etc.
You can more easily tune it to the kind of candidate you're looking for then too, for example I've asked how they'd tackle improving the performance of a particular SQL query that's been identified as too slow. There's a tonne of possible answers to that ranging from naïve/they don't really know, through pragmatic good fit responses, to way overkill I hope they understand we don't need them doing that here/not operating at that scale etc. - and it's fairly open-ended in what you can discuss driven by what they volunteer and know about. (Which is another good thing IMO, I don't like being on either side of interviewer quizzing and candidate umming and ahhing not really knowing! Both more comfortable and more beneficial to have a discussion about whatever is known IMO; to quickly move along the perimeter if you hit the edge of that.)
I think you're completely ignoring the reality of frauds. Jeff Atwood was writing about this over a decade ago with "FizzBuzz".
There are many people who spend more effort creating the illusion of competence on paper and on the job, getting harder to detect the higher they go.
We as a profession (software engineers) have continually resisted broad unified certification like other engineers which could be a replacement for code interviews to assess competence, but would have other drawbacks.
So we are stuck with code interviews to ferret out BSers. And even then it sometimes fails. But it appears to be the best tool we have, because there is nowhere to hide. Don't take ot so personally.
> There are many people who spend more effort creating the illusion of competence on paper and on the job, getting harder to detect the higher they go.
If you can navigate a software engineering position, purely undetected, by bullshitting it, I would say you can be a very good manager. You can probably handle high level concepts without knowing the implementation details.
> I think you're completely ignoring the reality of frauds.
Or maybe their strategy still catches all of the frauds and it has therefore never been a problem to them?
I have to agree with their take, and just asking a bunch of technical questions -even without any code- is good enough to filter out the obvious incompetents.
Hard disagree. On my teams, we're going to give you a problem with some slight ambiguities to see how you handle that. We're going to see what kinds of questions you ask and how you respond to feedback. We want to hear you walk through your thought process. The more senior the position, the more important all of this becomes. Getting the "correct" solution is, at best, 50% of the goal with the interview.
This is not the approach that the majority of coding interviews take. In my experience, it has been disinterested interviewers who are blatantly pretending to understand what they are asking. Any attempt to engage in the type of discussion you aim for is met with dead ends, because that's not what the Googled answer contains.
Your approach is a major outlier. You probably have many good candidates turning away, and for good reason, they have no idea that you are different to everyone else. Find a different way to do this, there are several other approaches.
There are very few options outside of coding interviews, and getting fewer by the day.
It's almost like everyone big and small is standardizing on this model. Feels like one of those mandatory courses you took in college: the teacher and all the students knew it was bullshit yet you needed to perform the parroting at some adequate level to pass.
I have 21 years of varied experience in software engineering, yet a recent "technical" phone interview was a kid asking me to "balance a B-tree"; I can tell his expectation was for me to start reciting some CS algorithm BS, probably that's what everyone else is doing. I politely declined and that was the end of the interviewing process with that company.
Classic mistake of overthinking it and failing to realize what interviewer really wants - which is to make sure the candidate can actually write code, like at all. The question itself doesn't really matter as much as it's just a pretext. I actually asked a variation of this question for many years at Google and it was clear within first 5 mins who has been writing code day-to-day and who's been mostly "brining key stakeholders into conversations at appropriate time".
Exactly this. In the interviews I give I care about whether the candidate can write code, yes, but also talk and think about code.
The conversation is the most important part of the interview, and the thinking (and communication) is the most important thing I'm trying to judge after basic skills.
Like you said, you can get a good sense within the first few lines of pseudocode if someone's at least competent at writing code. But that's just one motivation behind coding questions.
It's also very difficult to talk about code, algorithms, and solving problems without a concrete problem and concrete code in front of the candidate and interviewer. So both the question and the code the candidate writes are mainly context for the conversation where I try to see how the candidate thinks.
These kind of articles make me sad because I (and many other interviewers I've worked with) try to make it clear that this isn't a test - we don't care so much about being "right" or "wrong", and there shouldn't be any tricks of "a ha" moments.
We explain the goals and what we're looking for right up front. And I would hope most interviewers do the same, but I guess not. So there's this persistent myth among both interviewers and candidates that coding questions are about getting a right answer.
That's a shame because coding questions get such a bad rap, but I'm not really aware of a better options. Take-home problems and looking at GitHub are unfair to many people. A well-run technical interview should give lots of people a chance to succeed.
You are the exception I think. Most interviewers care about the correct answer. Get it and maybe get the job. Fail and definitely don't get the job.
If the interviewer said at the beginning, "I don't expect you to solve this problem in the 40 minute nor to have an optimal solution. I just want to watch you write some code and hear the problems you foresee and how you'd solve them" then maybe I could relax and do that. But, generally the pressure is on "get this right in 40 minutes or you're rejected"
Coding tests are an awful place to test someone’s conversational skills. I don’t talk while I code. You don’t either. Honestly I can’t even remember the last time I talked to anyone about the code itself outside of a PR. People talk about architecture and database migrations and why their containers aren’t behaving locally. Nobody ever tests for that stuff.
I, personally, cannot _think_ and _talk_ at the same time. It's just a stream of half-sentences, many of which my brain has already moved on from because what I originally thought won't work.
After writing this article it became very apparent to me that I'm complete garbage at interviews, but I'll outperform and exceed at the actual job function.
Adding another comment here, because this is part of the reason why I wrote this article.
> These kind of articles make me sad because I (and many other interviewers I've worked with) try to make it clear that this isn't a test - we don't care so much about being "right" or "wrong", and there shouldn't be any tricks of "a ha" moments.
> We explain the goals and what we're looking for right up front. And I would hope most interviewers do the same, but I guess not. So there's this persistent myth among both interviewers and candidates that coding questions are about getting a right answer.
I understand all of those things. I've written the same before[1]. However, as clear as your instructions are and as well meaning you may be, it may not help. I can logically understand every word you say, but as soon as that question rolls out, I will now be dealing with stress hormones and 30 years of learned behaviors from thousands of experiences, whether I choose to or not.
So while I applaud your methodology and wholeheartedly agree, just telling people that doesn't guarantee that it's not still an issue because humans are complex organisms.
This is pretty much how it worked at the robotics company I worked at.
We would give them a whiteboard problem, but:
a) it was a simple, stupid problem in C (C++ was our implementation language, so thinking at byte level was an important skill)
b) we were very generous about minor mistakes like missing semicolons, etc.
c) we were very generous about "points for effort"; if they didn't make it through the problem but we saw that they were on the right track we might pass them. Total frauds outed themselves very early; they would produce between jack and squat in terms of actual code (a lot of bloviation though).
But again, most companies aren't that company, or your company. For most screening coding exercises, a correct answer (and even something like optimal algorithm complexity) is a must to pass the candidate.
Meh. I’ve met dozens like you. People who swear they just want to see “how you think.”
Then, in the post interview roundup we talk about the candidates and you’re a bit disheartened that they didn’t complete the exercise, so you give a pass even when every other person in the room gives a thumbs up.
You are not alone! This is also how I run coding interviews - I have absolutely passed candidates who did not actually complete the described problem. I always inform my candidates that we want to have a conversation about the problem and its solution. I also deliberately pick questions that are going to need thought and some design, specifically to spark that conversation - talking about reversing a list gets boring pretty fast.
The issues people have with coding interviews are more about the interviewers than the questions, honestly.
I love the typo at the end. I have definitely worked with some stakeholders that I would have loved to stuff in a jar of pickling spices and leave on a shelf for several years to ferment.
>Classic mistake of overthinking it and failing to realize what interviewer really wants - which is to make sure the candidate can actually write code, like at all.
What about the lad who develops homebrew who got rejected from Google because he wasn't able to invert a binary tree on the spot? Many Googlers use his software internally and externally. If the purpose of the interview is to make sure he can code why did he fail?
You should really read your own link - the part where he admits he made that whole story up. Im gonna guess failing some basic coding is not what tanked him. He still thinks they owe him the job though because he’s kinda famous and did a popular project in a language that Google doesn’t use, so that’s cute.
I also dont see how this one anecdote (even if it was true) invalidates anything I said above. You’re gonna have false negatives in any system you chose unless you just wave everyone in
Why does "Many Googlers use his software internally and externally" mean he would be a good hire for Google?
From all his public complaining about failing an interview it seems Google did the right thing not hiring him, he has a massive ego and it's very possible that "writing homebrew" is less useful to google than "inverting a binary tree"
Not even Max Howell thinks Max Howell has a great attitude. He's often a dick in his own words. Maybe Google would have found a different job for him if he wasn't.
He said 90% of Google engineers used Homebrew. Google engineers said it wasn't true.[1]
He said Homebrew using OS libraries saved a lot of pain. He presented it as an example of why Google should have hired him. Actually it caused enough pain Homebrew stopped it.
My go-to first screening question, regardless of where I work, is to present some JSON from a public API relevant to the domain (nothing too crazy, maybe 2 or 3 levels of nesting max), then ask the candidate to do a filter plus a sum or max - then for bonus, analysis or refactoring of their choice.
Way too often, this results in a painful slog, lots of hints from me, and, nowadays, snippets of AI generated code copied in that actually solve part of the problem but that they don't read and therefore end up breaking.
If the interviewer want to know if someone can write code at all, then why he/she expects much more than that in an interview?
I believe there is no single declaration about 'what the interviewer' wants.
Especially that several times they do not know themselves, just try to mimic what is expected from them or seen in some random situation they came across. Sometimes, of course.
---
"brining key stakeholders into conversations at appropriate time"
This is some very good quality euphemism here, two thumbs up! : ))
The problem is that the candidate has been assured that they will be asked 'leet' code questions where solving the problem isn't enough, they will also be asked about O notation and how the code can be optimized and whether to use memoization or recursion. This is what the books will tell you, this is what YouTube will tell you, this is what 'helpful' recruiters will tell you.
And IME this is what most interviewers have been taught. They've got a list of sample questions, they've been told that if they give the knapsack problem and the interviewee doesn't immediately call out 'dynamic programming' than the interviewee is a pass.
If you only want to see working code than you are the exception rather than the rule.
I will ask all of those questions. But I don't expect perfect answers. You should at least know what big O is. I would really like it if you can tell an O(n^2) algorithm from a linear one. (That is often really important in real-world code). I would like you to consider different ways you can optimize the code.
I don't expect you to quickly crank out a novel optimal algorithm. But I like to see that you can think about how to solve problems programmatically. I would like to see you can identify differences in different algorithms and what tradeoffs there are. Considering different approaches that may be faster also shows that you didn't just memorize an algorithm from somewhere but that too took time to actual understand the problem and build a mental model in your head.
I have given people great reviews when they couldn't come up with a working algorthim because the clearly knew how to think like a programmer and considered all of the important things.
They only want to see somebody who can get working code and a glimpse of their thought process. But from 100s of mediocre examples, the better coder will have a "better thought process."
Same goes for dating. Of course people will swear up and down they "only consider personality." Turns out, they've met 10 other people with a better personality than you.
Just because they're "only looking for x" doesn't mean they'll accept anybody that clears the bar.
The ultimate read between the lines though is that "oh I'm only looking for xyz, nothing superhuman" in a process where you have 10,000 competitors and applicants will still require high performance on your part. It's just a nicety, a meaningless phrase.
Yes. I've filtered out countless candidates who can't code like this. It's amazing how people will fire off applications to jobs they can't do. You might doubt this because you wouldn't do it, but just wait until you get to sift through a fresh wave of applicants.
Has everyone I hired got "perfect marks" on the test? Of course not! It's not about that. It's about seeing how they react to a problem, watching them break it down, ask questions, and, ultimately, get on with it. If the job is too sweep floors you need to be able to hold a broom. It's as simple as that.
Yes, unless you’re in a niche area or only hiring through a network, you’re going to sift through mountains of unqualified candidates because those are mostly the ones on the market (see Market for Lemons[0])
I disagree, I think it's a failure of recruiting if you got someone into an interview who can't code "at all". That must mean nobody at the company is technical "at all" either then, if you are getting people that far along without having any clue if they even write code at all.
I don't have this problem - I can easily tell who is good and who is not good by looking at their stuff online, which repos they are contributing to and what their contribution is. I look at personal projects - I can easily tell what parts they wrote vs didn't because it's usually specific to the project.
I can tell from their blog posts and comments, especially GitHub comments - I can even see if they're pushing features at 11PM on a Friday if the obsession piece is crucial to the hire.
People who say all their work is hidden under NDA or they're new grads and haven't done anything yet - sorry, if there's nothing to view online I just wouldn't qualify you to interview.
Though I have given out 50+ code tests in my career because I had to, I would never choose to do this, if given the chance when hiring someone I would never give them a code test. I think it's an amateur move and wastes everyone's time. At its best (as in the case of CTCI interviews) it's an exclusivity filter for academics who memorized the optimal data structures for algorithms as taught in school, but the candidate might not have any of the skills needed to build app features, perform DevOps, etc. or even operate a Terminal - CTCI doesn't cover anything async, nothing about UIs, APIs, databases, services, git, design, file formats, etc. it's purely academic sport. And like I said, a good developer's work should be highly visible anyway - skip the random code test.
I would spend the recruiting effort finding specific developers using specific technologies that aligns with the role and making them excited about the opportunity rather than canvasing 1000 code tests out to anyone who applies.
> That must mean nobody at the company is technical "at all" either then, if you are getting people that far along without having any clue if they even write code at all.
Lots of people skate by in technical roles by barely doing anything technical. Lots of people also overinflate their achievements in resumes and conversations aka lying. How is non technical recruiter supposed to evaluate their coding chops?
> People who say all their work is hidden under NDA or they're new grads and haven't done anything yet - sorry, if there's nothing to view online I just wouldn't qualify you to interview.
Lol, great method! I have a better one - just hire acm winners. No need to test them
No way to know for sure, but I'd guess the number of employed developers writing blogs and having personal projects is in the single digits. For developers with more than 20 years experience, I'd be surprised if it's even 1%. I've got more experience than that, which means I have an extensive network of former colleagues at a similar experience level and I don't know one single person who blogs about tech or works on personal projects. We all have busy family lives and plenty of money so absolutely no need to spend any personal time doing work related stuff.
I find a simple realignment would solve a lot of this interview angst: Interviewers seem to interview on the premise that they need to find out what the candidate can't do. But this is not useful. I can tell you what they can't do. To a first approximation, the answer is "everything". I am an experienced and skillful developer, yadda yadda yadda, and I've never touched React, have no game development experience, have never written kernel code, haven't touched matrices since school, have only dabbled in embedded development as a hobbiest, etc. etc. etc. Make a list of all the things I can't do and I look like an idiot. The list of possible things is too long.
The goal of an interview is to find out what the candidate can do.
If you interview someone, and they "fail all the questions we asked", it is not the candidate who has failed... the interviewer has failed. You ran an entire interview and all you know is what the candidate can't do. You have learned virtually nothing. The questions must be adjusted to what the candidate can do. Only for the absolute worst candidates should you ever exit an interview saying they failed at everything. (Sadly, such candidates do exist. For instance, consider the recent stories about AI fraud. If your level of programming skill is "I have ChatGPT", I'm going to have a hard time scoping a question down to you.)
If I had asked this question in an interview, or a related one, I would have stepped the problem down until the candidate could pick it up. (Odds are I'd have started lower and worked my way up anyhow.) If the candidate then sort of finds their footing and once going can start folding in the other requirements as we go, great. Who sits down and designs a system for all twelve adjectives ("reliable", "concurrent", "redundant", "runs on a Z80") we want anyhow? One can not help but design a system one adjective at a time, with the others only kept in mind to make sure we don't back into a corner. There's no reason not to interview that way too. (And I tell the candidate this is what we're going to do, so hopefully they don't feel like I'm out to get them or experience requirements whiplash without knowing why I'm doing this.)
Yes this, and to further this, the "can't do" thing that you measured is in fact extremely partial and incomplete: you've measured that the developer can't write an algorithm on a whiteboard with a felt tip marker in front of a stranger they just met. You don't even know if they can't write the algorithm. Just that they can't do it there.
Is that important to you? Maybe it is for some people. As a person who has been in team lead and hiring capacity before, it is not for me.
I trained for interviewing at Google twice, but never really chose to give interviews despite that being a highly pressured part of the job because I could not philosophically vibe with that process. But what some people who have adopted this process are missing is that Google etc does this only because they are swamped with resumes and have their pick of bazillions of quality engineers. They don't care about false negatives, more false positives.
Startups and smaller companies should care about false negatives. It's hard to find and retain good people. Smaller companies need to aggressively find and cultivate good people to make good teams in order to be/stay competitive -- and that means accepting a diversity of ways of working.
I'm glad whiteboard interviews are dead now that everyone does "virtual" on-site interviews. When I interviewed at Google, I requested a laptop because writing on a whiteboard all day hurts my hand. Every interviewer complained about me using a laptop.
I just (barely) failed an automated interview by some company who posted on the last Hiring thread. Their approach was interesting but they definitely measure the wrong thing IMO. They were looking for senior Python devs yet assumed that every senior Python dev knew X, Y or Z Python API/feature by heart. I failed a single question, precisely one where I wasn't familiar with the particulars and couldn't "figure it out" live due to the nature of the platform.
I've worked on a Python interpreter (not CPython) for many years now, I just never used that particular module (because we do it differently in our implementation). I suppose businesses try their best, but aren't always measuring what they think they are...
Related to not knowing everything, there's also this mal-assumption that if the candidate doesn't know something simple (and obvious to the interviewer) that they will never be able to figure it out on their own.
My approach to interviews over the past decade has been as follows:
1. My preparation for an interview involves researching the company, not technical matters. I don't brush up on coding interview questions. I've never done leetcode.
2. If I find the interview questions to be ridiculously off-topic (such as silly algorithm questions), I end the interview. You're not the kind of company I want to work with.
3. If I find the questions to be valid, but I can't answer them, then I'm not the right candidate for the job (hopefully I'd already have found this out during the research phase, but we all make mistakes).
4. If we can get past all this gatekeeping to the actual important topic if what BUSINESS issues they're trying to solve, and how I can fit into this process, then we've got a real interview and I'm interested.
So far I haven't been out of work more than a few months.
I have done a few of these types of interviews. It's incredibly awkward. At one place, I was even issued a laptop, a badge and added to the Github organization. I mean, I guess? But it felt like I was starting work, not doing an interview. People there also didn't know if I was a new employee or who I was when I went to lunch and get snacks. I spent more time getting my local dev up and running than actually doing the task I was assigned, which they could have gathered the same information about me in a 30 minute interview.
If you can't determine from a few interviews you want to hire me or not, it's a no from me. It's a HUGE red flag if an employer can't seem to make up their mind so they have to resort to awkward "test drives" like this. If you have a job at the time already, it also makes it almost impossible to pull off without using PTO.
Add all these things together and you really don't want to work for a company who is very inexperienced in hiring such that they resort to tests like these. It shows they don't know how to interview, are not efficient with your time (and probably their time also), and a high chance they are non-technical.
The past decade is the golden age for software developers, so we had lots of options. I was wondering if we will still have so much freedom as the market has increasingly become the buyer's one.
I haven't been interviewing full-time, but these past couple of years I have had barely any success. Currently getting insta-refused even for jobs with lower skill requirements. I've heard similar from friends and colleagues.
There's a bootstrap issue as well. A developer with a brand name on their resume is much more desirable than one without, even for equal skill and experience.
I'm not op, but I have had to do this a few times because of the same reasons. You pause, take a breath and kindly say "thank you for the opportunity, but at this time I don't think this is the right fit" and leave it at that. No need to embellish, or add extraneous detail or think you're being awkward because they will do the same thing if they don't want to go further in the process with you. It's just business, treat it as such.
Sometimes there are valid reasons for why they look at very strong algorithm skills and then I simply admit it's not really my cup of tea nor my passion.
Sometimes they answer variations of "it's standard/it's how we do it" so then I propose whether we would like to code something more similar to what the daily job would be to which they generally play along, even happily.
But if some don't I then say that I don't see a fit.
Might be awkward but...who cares? I don't, they won't after 5 seconds the interview's over either.
We should be a union of people who won't take this kind of bs. I have the same exact procedure as I think interviews like that show a complete lack of understanding of what makes a developer great in a company, and let me tell you, solving CodeSignal/Leetcode unpaid for hours isn't that.
The coding interview looks different when you view it for what it would be called in other industries: a licensure examination. It looks particularly insane to relicense for every single job you apply to. It also looks supremely unfair to have proctors for this exam with varying expectations and training to actually correctly administer it.
- licensing ensures only a minimum level of quality
- people with licenses still do interviews, often just as grueling
- licensed careers with high performers (lawyers, doctors, ib, etc) have other forms of filtering which are much more painful, like years of low pay internships
Studying for a few weeks to solve fun puzzles to make 400k sounds like a deal to me.
Hearing from people that did internships in other fields for almost nothing, during which CS students make more than most workers really drove that point home for me.
> - licensing ensures only a minimum level of quality… people with licenses still do interviews, often just as grueling
Agreed, I never asserted against these two points. The point of licensure is to make the first round, which is fairly routine at this point, more equitable and less susceptible to probabilistic effects. It also frees labor from administering this exam round to every candidate.
You could view the trial-by-Leetcode that people undergo when they switch jobs every 4 years or so as a form of relicensing. One advantage that the current setup has over officially proctored examinations is that you get to try again repeatedly until you are successful.
Other industries do not require repeating the license exam. Many have continuing education or other professional development requirements. These can be expensive and time consuming but not difficult generally.
Real licenses do not expire without notice. Real license exams are more consistent, have clear pass criteria, and have higher pass rates.
The roughest outline for hiring as far as I can follow it is start with 100 resumes, filter it down to 10 by picking ones that are nicely formatted, then filter down to 1 by interviewing. Each level of filtering should be structured so that it biases towards technical competence.
It isn't really a licensure examination because, as you point out, the industry doesn't bother to put the resources in to make sure anything is systemically analysed. It is a process to balance supply and demand of jobs.
90% of the filter is resume formatting!!!? I'm open to hear your experience but that sounds like an arbitrary filter which limits your pool more than it selects for talent
Yeah, but the variation can be good. If I had to imagine what the hypothetical licensing process would be, I imagine it would be something easy that admits tons of mediocre devs. Plus, would it come in different language/subfield flavors to account for different roles? One job would ask me to implement a custom allocator, and I'd pass; another would ask me how to add Frondle to Artifactory, and I'd fail miserably.
> I imagine it would be something easy that admits tons of mediocre devs
Given that Medical boards and the Bar strike fear into the hearts of students and demand immense preparation, I think we could do pretty well. The test prep mentality isn’t altogether different from the “grinding leetcode” that happens today anyway. The difference is that it would at least be fair.
Actual engineering licenses in the US have kind of solved this.
There’s the easy exam that pretty much everyone passes eh e they get their degree (the FE), and then there’s the hard one that not even everyone attempts after a couple years of experience (the PE).
And within each level, you specify your discipline (civil, mechanical, etc) and then are required to have deeper knowledge of several subfields within that discipline.
I personally think an industry-wide license would be an easier filter than requiring a degree: Schools don't often teach what you need to know, and sitting for an exam is a lot easier for the "I've been in the industry XX years without a degree."
It would also be easier for the industry to settle on some filters in the exam to block people who just SPAM job postings. For example, open jobs for doctors are only posted on websites that are available to doctors. There is no way for the general public to SPAM job listings for doctors.
I've found that asking them to review some obviously bad code with glaring errors and problems is more informative than asking them to solve some random DSA problem.
Candidates who can code well can point out code that has obvious problems. Just ask if this is good or bad, and if it is bad, how they could improve it. This demonstrates competency and doesn't make the interview seem like a grind but instead more like a conversation.
Just got be careful with the “how to improve” part. In my experience as an interviewee sometimes it becomes a regular algorithmic interview. A couple times I found some N+1 queries or inner loops and was asked to “fix it”, which might just turn into leetcode.
The best code review interviews are the ones where there is a healthy amount of actual code, with a handful of functions and classes, some badly named variables, bad comments, some misleading code paths, couple bad patterns, etc… the worst ones are a non-optimal solution and you’re asked to make it optimal. That’s just leetcode disguised as “code review”.
Leave it open-ended and include code with multiple levels of bad so it's not just a quiz. I used real C code that research scientists had given me. If they look at it and say there's no reason this should be in C and in a dynlang instead, that is fine. If I hand them C code where the entire program is a 1000-line main function, with lots of repetition, hard coded file names, and fixed-sized string buffers, and all they tell me is the indentation is icky: that's a negative signal.
"The martini may be simple, but it is not easy to make an excellent one. It's a very solid test of a bartender's skill because, unlike many drinks, ingredients alone cannot carry the cocktail. A piña colada for example, is mostly about ingredients (are you using a good coconut cream? fresh pineapple?) For the martini the chilling and dilution need to be just right. This tests the bartender's most important skill: mixing. Proper mixing of the beverage is ultimately what makes a martini."
[..]
"martinis are shockingly easy to fuck up. and this conversation is exactly the reason why the martini is a good test of a bartender's capability. being a bartender is more than putting fixed quantities of ingredients in a glass. how do you know when your martini is properly diluted, either by shaking or stirring? a good bartender will know. a bad bartender will not. a terrible bartender won't even realize dilution is crucial."
I don't really drink much and never had a martini in my life, but I thought it was pretty interesting.
True enough... Even in software, I was pretty amazed at how much of a filter of, here's a CSV, use one of N languages to load the data, do a check and output the valid entries to one file and the invalid inputs to an error file. You can use any libraries you like, please create a github repo and share with $ACCOUNT for your solution.
I know not everyone works with CSV necessarily, but there are dozens of libraries for a lot of languages. Even if focused on N being those supported in a given company/org. It should be less than an hour of work. Bonus points for any tests/automation, etc.
I use the "launch ramp" technique. Ask a series of prompts (instead of a single prompt with a long answer). Explain the prompts that will get progressively more complex, aka the ramp will gradually get steeper and get very steep later. I can stop the interview quickly if the candidate can not find simple mistakes and how to remedy them. I can also jump ahead to complex issues to engage highly qualified candidates.
Last time I got hit with an interview question like that, my answer ultimately had to be "block the merge and counsel the person who wrote this about performance." I'm still not sure if that's the answer they were looking for (this was for a staff engineer position), but I'd stand by it 100% of the time.
> I've found that asking them to review some obviously bad code with glaring errors and problems is more informative than asking them to solve some random DSA problem.
I once had a coding interview like this, but the problem was that the code was so obviously bad, I couldn't even make sense of what the code was supposed to do if it were good. It felt like the interviewer had just come up with an example of bad code without any context of how the code would make sense if made "good". It was just totally artifical.
If someone had presented the bad code in some Stack Overflow question, I would have started by stepping back to ask, "What are your trying to do?" Except in this case, the interviewer wasn't actually trying to do anything except quiz me.
Identifying a bug in production code would be better, I think.
This is a good idea. This would also show someone's ability to read and contribute to existing code which is a large part of our day to day tasks. There are some that can only solve problems their way, which often means them trying to rewrite everything.
We used four basic exercises during the first remote interview. Simple things like, "there is a number on each line of this text file. Find the sum of them."
This was an effective method to screen out applicants who didn't have the basic coding skills to align with their stated resume experience. And there were a decent number of these.
We did further development project exercises later in the process that took about 15 hours. We paid the candidates for this time, even those that didn't pass. It was also an effective screening tool.
All our exercises were very "real world." In the candidates own development environment and having been given instructions on how to prepare. They also have access to whatever Internet resources they want while doing the exercises. If they can't do the exercises, they can't do the job.
I know there are mixed opinions on this and I feel for candidates who have to invest a ton of time in exercises like this. But I can't imagine trying to hire without visibility into how they execute relatively basic software development tasks.
I think employers can and should structure the process so the time investment is minimal upfront and only increases as both parties have gotten to know each other and want to proceed.
Truth is, I'd much rather hire someone who is interested in working for us for 2-4 weeks as a contractor instead of the normal interview process. But that usually doesn't work for the candidate.
For those wondering, we paid $50 an hour. It wasn't quite wage equivalent, but we felt it was reasonable to consider some of that "gap" as investment on the part of the applicant.
Readit News
There are many variations in questions too: search a tree or graph with some customized criteria, using a double buffer to implement a tree-style prefix scan, implementing an integer multiplication with unlimited digits, some streaming algorithm, tree-walking to transform one tree to another, a simplified skip list, the options are unlimited. A good candidate tends to grasp the high-level concepts quickly (and they can ask questions), and is quick to convert intuition into working code. I find that there is a strong positive correlation between the performance in work and the performance in such coding interviews.
To me, the value of a software engineer is to ask questions, make hypotheses and be able to iterate quickly. Balancing trees, leetcode and other algorithmic stuff on the spot sounds like bringing the dreadful education system structure to the real world.
Also if a senior person can't in 30/45 min of talking with someone figure out the general experience level then the problem is them, really.
This is why I've always been so confused. Why is the software engineering interview wildly different from the traditional engineering interview (seniors sit down with candidates and discuss how to solve a relevant proxy problem the team is currently undergoing (has the side benefit of making the interview process potentially fruitful if you don't go with that candidate. This can be (and sometimes is) abused though)). I mean... we all speak the same language, and that isn't standard English... right?
The thesis is not that these exercises are representative of work but rather predictive of performance.
Sales has a similar phenomenon with sports. While there is no athleticism involved in selling, many believe history in competitive sports to be a positive predictor of sales success.
---
You can reasonably argue whether leetcode accomplishes this well or poorly, but...
Always remember that purpose of an interview (for an employer) is to predict performance. So you are looking for the resume screening+interview process that most accurately assesses that.
OK take it as read that your posturing has succeeded and we all agree that you're a brilliant interpersonal genius and the rest of us are all useless chumps. What then? The rest of us still need to interview and make hiring recommendations. Or are you suggesting that employers should fire anyone who lacks your magical talent?
- Future proof. Unless I work for an outsourcing company, sooner or later I will want to push the envelope, or so I hope I do. And to push the envelop, one needs good CS fundamentals (maybe there are some exceptions in some specialized field). Think about React. It's a JS framework, yet to invent it one needs to understand at least compiler and graph.
- Geekiness/talent filter. The same reason that the nascent Google and Microsoft and any elite companies like Jane Street asked Putnam questions, Martin Gardner questions, ICPC questions, and clever probability puzzles. Whether it's a good idea is debatable, but at least those companies want to have such type of people and they were hugely successful. Note the word filter: people who pass the interview may not be good, but failing the interview means the candidate may not be smart or geeky enough. Again, I'm not endorsing the idea but just exploring the motivation of the interview policies.
- Information density. Assuming a company does want to time box a coding interview in an hour, it will be hard to come up with realistic question without spending too much time on the context. On the other hand, it's relatively easy to come up with a data structure/algorithm question that packs enough number of abstractions and inspection points to examine one's programming skill.
Actually most of them, including the really inexperienced juniors have 'figured' you out in less that 15 minutes, or at least they have decided whether to hire you or not in 15 minutes. But they have to put on a charade of being fair.
Also a 'white' older male is the least preferred even if he is smarter compared to all females and the minorities that are being interviewed as long as they are not terrible. Biases galore.
Or it's not their call, theirs is just to work with them after.
I've had to work on tree traversal stuff multiple times in my life, anything low level GUI related will work with trees a ton.
I've also had to work with hash tables directly, and with memory caching layers.
I really should learn to write a proper parser, as I've had to write parsers multiple times now and they are always an ugly hack job.
At some of my past jobs and the current one, this kind of algorithmic knowledge was important to build features that were differentiators in the market. As much as people love to pretend, not every single possible solution is in a library. Sometimes you're the one building the library.
It doesn't have to be leetcode, but candidates should at least be able to produce some code that doesn't come from the README of their favourite framework.
Also, talking for 30/45 mins can be enough, but it produces false positives when you have people coaching candidates. I've had people completely acing interviews that it felt like the perfect candidate. Well, it was rehearsed. When I asked for a fizz-buzz type question they completely messed it up.
Dead Comment
Lots of people have lots of ideas on what makes a great interview question, but none of them are backed by data. LeetCode-style algo questions USED to be an indicator of intelligence as per Google's investigation, but now it's so heavily gamed that I doubt there's any signal behind it anymore.
Someone needs to spend time and effort and money and actually do random controlled tests to see if people who pass a particular type of test actually make good employees. But so far no one has done this, except Google as far as I've seen. But even now, I think there's no evidence that even algorithm questions are any indicator, given what I've seen.
This random controlled trial you're talking about is so hard to do because there are infinite confounders. How do you even measure success? After how long? How do you separate politics from technical ability?
In reality people above a certain bar are able to make some contribution to most projects. You can't and don't need to staff every project with superstars. The superstars are rare but are not hard to identify. They'll have the track record, reputation and the skills.
Another thing to consider is that the best outcome of interviews depends on the quality of the pipeline into those interviews. If you can't get good employees to apply you won't get good employees hired. Pay well and build a company that people want to work for, and then your hiring is going to be easier.
Do you have evidence that the standard coding interview works? (There's evidence that it doesn't)
I'm with you that the claim might be too strong to say "this is the way" but that's because I'm of the (very strong) opinion that interviewing is an extremely fuzzy process and there are no clear cut metrics to measure one's abilities in a significantly meaningful way. a̶l̶l̶ ̶m̶o̶d̶e̶l̶s̶ ̶a̶r̶e̶ ̶w̶r̶o̶n̶g̶ ̶b̶u̶t̶ ̶s̶o̶m̶e̶ ̶a̶r̶e̶ ̶u̶s̶e̶f̶u̶l̶ There are useful interviewing methods but certainly not "the best" method. Trying to just mark checkboxes only leads to mediocre results. The reason we're generally okay with this is because we more often than not don't need rockstars and it doesn't make sense to put a lot of time and energy into this process when we get sufficient results through lazy methods.
FWIW, a typical (non-software) engineering job really just involves high level discussions like the OP suggests but even without the implementation. It is generally about seeing how the person thinks/problem solves and looking at problems they solved in the past. It isn't resource intensive and good enough. Because truth is, you don't know what someone is like as an employee until they are an employee (and even then this is fuzzy)
Where can I find the data on this?
I want to see if they can think in code. If they can translate something simple to written code. There's always a chance to have a conversation as well to probe different aspects of their understanding.
A few of those by a few people and I think you get a pretty good read.
Mix in some questions about their work and/or some general knowledge. I've also given people code to read (real world code) and explain to me.
For Algorithms and Data Structures I can look at their grades if they're fresh out of school. But we all know most of our work is not that (someone's gonna show up on the thread and say differently - I'm sure ;) ). If you can think in code and have the right mental models implementing an algorithm from a book isn't hard if you really have to.
But it won't cut it for a senior. Such people should be able to answer a wide range of questions, including some algorithm and data structure stuff. Not that they have to be able to code a min-max-heap from scratch, but they have to know what's out there, how to use it, and how to keep the overview of an entire project. That's not going to be evident from a coding interview.
So, horses for courses.
A typical engineer doesn't begin their day thinking "I should implement a streaming algorithm" out of nowhere (and if they do, they can always seek reference). They analyze a (typically underspecified) problem, figure out the constraints, and then start thinking about a solution. Note that both processes, analysis and solution, can take hours, days, weeks or months!
Coding interviews have everything backwards by providing synthetic solutions to mostly out-of-context problems.
For example, interviewers could:
- Share some portion of a codebase with the candidate prior to the interview
- Go over the code with the candidate
That's it. You will know really fast who's senior and who's never wrote software. It can't be faked. And it can be adapted to mostly any positions (backend, frontend, devops, architecture, whatever).
I've hired and interviewed a lot of people over the years. Mostly without the help of recruiters. I like to think I'm pretty good at that, actually. I always look for people that are eager to learn. I care more about what they don't know than what they can regurgitate because they did lots of silly online prep. If people are young and fresh out of school, my assumption is they are going to have to learn a lot in a hurry. So I look for people that are curious, eager, and have an open mind. The best way to do that is to get them slightly out of their comfort zone and talking about how they would tackle things. The signs I look for here are enthusiasm, ability to surprise with good answers, etc.
This generally does not involve code interviews; though I might ask some targeted questions about how they would deal with certain things in languages, frameworks, etc. they list as being proficient in. I've of course worked for some people that insisted on me organizing code interviews and it's a chore and IMHO you learn absolutely nothing that isn't obvious if you just casually talk to people for 30 minutes or so. I usually argue against such things and prefer just talking to people directly.
Thus happened to us twice, and after we introduced a simple live coding session to our interviews where you have to implement a simple real world web component (for a frontend position) suddenly the problem of bad hires disappeared almost entirely (you can’t judge someone’s character during a short interview but that’s another issue).
I have no idea what these people expect. We ask coding questions, we have basic programming skills as a requirement, what do they think will happen if they get the job? Do they expect to just magically learn the required skills or hope that nobody will notice that they're not able to close their tasks?
Although i see it as a way for the candidate and i to talk about code in general, and assess his level. No as a simple barrier.
i hired a candidate that completely failed a code interview because he was super nervous, but just talking about the problem made me quite sure he was actually good.
It's surprising how many 'senior' engineers are actually BSers who will slow you down or derail you completely rather than speed you up and you need to spot them because they will excel at getting through the non-technical interview filtering!
Also, I'm interested in how you explore things and explain things. I'm not actually interested in acquiring an implementation of FizzBuzz or whatever. I just want you to show me that you 'get' it and then we can get on to the interesting stuff like 'tell me about your last project' etc.
So don't be too hasty to think the people doing technical interviews are idiots thinking devs are interchangeable cogs etc.
Sounds like you'd more valuably/realistically get that from the discussion of a previous project though? How it works, or something interesting they had to figure out, etc.?
> don't be too hasty to think the people doing technical interviews are idiots
I don't think anyone has a problem with technical interviews? At least I agree that's not reasonable. That doesn't have to mean 'coding' though, you can ask about how they'd approach a particular problem, quiz on some fundamental knowledge in a fizzbuzz sort of way, etc.
You can more easily tune it to the kind of candidate you're looking for then too, for example I've asked how they'd tackle improving the performance of a particular SQL query that's been identified as too slow. There's a tonne of possible answers to that ranging from naïve/they don't really know, through pragmatic good fit responses, to way overkill I hope they understand we don't need them doing that here/not operating at that scale etc. - and it's fairly open-ended in what you can discuss driven by what they volunteer and know about. (Which is another good thing IMO, I don't like being on either side of interviewer quizzing and candidate umming and ahhing not really knowing! Both more comfortable and more beneficial to have a discussion about whatever is known IMO; to quickly move along the perimeter if you hit the edge of that.)
There are many people who spend more effort creating the illusion of competence on paper and on the job, getting harder to detect the higher they go.
We as a profession (software engineers) have continually resisted broad unified certification like other engineers which could be a replacement for code interviews to assess competence, but would have other drawbacks.
So we are stuck with code interviews to ferret out BSers. And even then it sometimes fails. But it appears to be the best tool we have, because there is nowhere to hide. Don't take ot so personally.
If you can navigate a software engineering position, purely undetected, by bullshitting it, I would say you can be a very good manager. You can probably handle high level concepts without knowing the implementation details.
Or maybe their strategy still catches all of the frauds and it has therefore never been a problem to them?
I have to agree with their take, and just asking a bunch of technical questions -even without any code- is good enough to filter out the obvious incompetents.
How can a code interview fail? Hidden earpiece?
Your approach is a major outlier. You probably have many good candidates turning away, and for good reason, they have no idea that you are different to everyone else. Find a different way to do this, there are several other approaches.
It's almost like everyone big and small is standardizing on this model. Feels like one of those mandatory courses you took in college: the teacher and all the students knew it was bullshit yet you needed to perform the parroting at some adequate level to pass.
I have 21 years of varied experience in software engineering, yet a recent "technical" phone interview was a kid asking me to "balance a B-tree"; I can tell his expectation was for me to start reciting some CS algorithm BS, probably that's what everyone else is doing. I politely declined and that was the end of the interviewing process with that company.
The conversation is the most important part of the interview, and the thinking (and communication) is the most important thing I'm trying to judge after basic skills.
Like you said, you can get a good sense within the first few lines of pseudocode if someone's at least competent at writing code. But that's just one motivation behind coding questions.
It's also very difficult to talk about code, algorithms, and solving problems without a concrete problem and concrete code in front of the candidate and interviewer. So both the question and the code the candidate writes are mainly context for the conversation where I try to see how the candidate thinks.
These kind of articles make me sad because I (and many other interviewers I've worked with) try to make it clear that this isn't a test - we don't care so much about being "right" or "wrong", and there shouldn't be any tricks of "a ha" moments.
We explain the goals and what we're looking for right up front. And I would hope most interviewers do the same, but I guess not. So there's this persistent myth among both interviewers and candidates that coding questions are about getting a right answer.
That's a shame because coding questions get such a bad rap, but I'm not really aware of a better options. Take-home problems and looking at GitHub are unfair to many people. A well-run technical interview should give lots of people a chance to succeed.
If the interviewer said at the beginning, "I don't expect you to solve this problem in the 40 minute nor to have an optimal solution. I just want to watch you write some code and hear the problems you foresee and how you'd solve them" then maybe I could relax and do that. But, generally the pressure is on "get this right in 40 minutes or you're rejected"
After writing this article it became very apparent to me that I'm complete garbage at interviews, but I'll outperform and exceed at the actual job function.
> These kind of articles make me sad because I (and many other interviewers I've worked with) try to make it clear that this isn't a test - we don't care so much about being "right" or "wrong", and there shouldn't be any tricks of "a ha" moments. > We explain the goals and what we're looking for right up front. And I would hope most interviewers do the same, but I guess not. So there's this persistent myth among both interviewers and candidates that coding questions are about getting a right answer.
I understand all of those things. I've written the same before[1]. However, as clear as your instructions are and as well meaning you may be, it may not help. I can logically understand every word you say, but as soon as that question rolls out, I will now be dealing with stress hormones and 30 years of learned behaviors from thousands of experiences, whether I choose to or not.
So while I applaud your methodology and wholeheartedly agree, just telling people that doesn't guarantee that it's not still an issue because humans are complex organisms.
[1]: https://darrenkopp.com/posts/2016/02/25/always-learn-somethi...
We would give them a whiteboard problem, but:
a) it was a simple, stupid problem in C (C++ was our implementation language, so thinking at byte level was an important skill)
b) we were very generous about minor mistakes like missing semicolons, etc.
c) we were very generous about "points for effort"; if they didn't make it through the problem but we saw that they were on the right track we might pass them. Total frauds outed themselves very early; they would produce between jack and squat in terms of actual code (a lot of bloviation though).
But again, most companies aren't that company, or your company. For most screening coding exercises, a correct answer (and even something like optimal algorithm complexity) is a must to pass the candidate.
Then, in the post interview roundup we talk about the candidates and you’re a bit disheartened that they didn’t complete the exercise, so you give a pass even when every other person in the room gives a thumbs up.
Nah. Re evaluate yourself and your biases.
The issues people have with coding interviews are more about the interviewers than the questions, honestly.
What about the lad who develops homebrew who got rejected from Google because he wasn't able to invert a binary tree on the spot? Many Googlers use his software internally and externally. If the purpose of the interview is to make sure he can code why did he fail?
https://www.quora.com/Whats-the-logic-behind-Google-rejectin...
He seems to have a great attitude and would fit right in but it's clear Google is optimising to keep people out rather than find great software Devs.
I also dont see how this one anecdote (even if it was true) invalidates anything I said above. You’re gonna have false negatives in any system you chose unless you just wave everyone in
From all his public complaining about failing an interview it seems Google did the right thing not hiring him, he has a massive ego and it's very possible that "writing homebrew" is less useful to google than "inverting a binary tree"
He said 90% of Google engineers used Homebrew. Google engineers said it wasn't true.[1]
He said Homebrew using OS libraries saved a lot of pain. He presented it as an example of why Google should have hired him. Actually it caused enough pain Homebrew stopped it.
[1] https://news.ycombinator.com/item?id=23844936
Way too often, this results in a painful slog, lots of hints from me, and, nowadays, snippets of AI generated code copied in that actually solve part of the problem but that they don't read and therefore end up breaking.
I believe there is no single declaration about 'what the interviewer' wants.
Especially that several times they do not know themselves, just try to mimic what is expected from them or seen in some random situation they came across. Sometimes, of course.
---
"brining key stakeholders into conversations at appropriate time"
This is some very good quality euphemism here, two thumbs up! : ))
Deleted Comment
And IME this is what most interviewers have been taught. They've got a list of sample questions, they've been told that if they give the knapsack problem and the interviewee doesn't immediately call out 'dynamic programming' than the interviewee is a pass.
If you only want to see working code than you are the exception rather than the rule.
I don't expect you to quickly crank out a novel optimal algorithm. But I like to see that you can think about how to solve problems programmatically. I would like to see you can identify differences in different algorithms and what tradeoffs there are. Considering different approaches that may be faster also shows that you didn't just memorize an algorithm from somewhere but that too took time to actual understand the problem and build a mental model in your head.
I have given people great reviews when they couldn't come up with a working algorthim because the clearly knew how to think like a programmer and considered all of the important things.
They only want to see somebody who can get working code and a glimpse of their thought process. But from 100s of mediocre examples, the better coder will have a "better thought process."
Same goes for dating. Of course people will swear up and down they "only consider personality." Turns out, they've met 10 other people with a better personality than you.
Just because they're "only looking for x" doesn't mean they'll accept anybody that clears the bar.
The ultimate read between the lines though is that "oh I'm only looking for xyz, nothing superhuman" in a process where you have 10,000 competitors and applicants will still require high performance on your part. It's just a nicety, a meaningless phrase.
Has everyone I hired got "perfect marks" on the test? Of course not! It's not about that. It's about seeing how they react to a problem, watching them break it down, ask questions, and, ultimately, get on with it. If the job is too sweep floors you need to be able to hold a broom. It's as simple as that.
[0] - https://en.m.wikipedia.org/wiki/The_Market_for_Lemons
I don't have this problem - I can easily tell who is good and who is not good by looking at their stuff online, which repos they are contributing to and what their contribution is. I look at personal projects - I can easily tell what parts they wrote vs didn't because it's usually specific to the project.
I can tell from their blog posts and comments, especially GitHub comments - I can even see if they're pushing features at 11PM on a Friday if the obsession piece is crucial to the hire.
People who say all their work is hidden under NDA or they're new grads and haven't done anything yet - sorry, if there's nothing to view online I just wouldn't qualify you to interview.
Though I have given out 50+ code tests in my career because I had to, I would never choose to do this, if given the chance when hiring someone I would never give them a code test. I think it's an amateur move and wastes everyone's time. At its best (as in the case of CTCI interviews) it's an exclusivity filter for academics who memorized the optimal data structures for algorithms as taught in school, but the candidate might not have any of the skills needed to build app features, perform DevOps, etc. or even operate a Terminal - CTCI doesn't cover anything async, nothing about UIs, APIs, databases, services, git, design, file formats, etc. it's purely academic sport. And like I said, a good developer's work should be highly visible anyway - skip the random code test.
I would spend the recruiting effort finding specific developers using specific technologies that aligns with the role and making them excited about the opportunity rather than canvasing 1000 code tests out to anyone who applies.
Lots of people skate by in technical roles by barely doing anything technical. Lots of people also overinflate their achievements in resumes and conversations aka lying. How is non technical recruiter supposed to evaluate their coding chops?
> People who say all their work is hidden under NDA or they're new grads and haven't done anything yet - sorry, if there's nothing to view online I just wouldn't qualify you to interview.
Lol, great method! I have a better one - just hire acm winners. No need to test them
The goal of an interview is to find out what the candidate can do.
If you interview someone, and they "fail all the questions we asked", it is not the candidate who has failed... the interviewer has failed. You ran an entire interview and all you know is what the candidate can't do. You have learned virtually nothing. The questions must be adjusted to what the candidate can do. Only for the absolute worst candidates should you ever exit an interview saying they failed at everything. (Sadly, such candidates do exist. For instance, consider the recent stories about AI fraud. If your level of programming skill is "I have ChatGPT", I'm going to have a hard time scoping a question down to you.)
If I had asked this question in an interview, or a related one, I would have stepped the problem down until the candidate could pick it up. (Odds are I'd have started lower and worked my way up anyhow.) If the candidate then sort of finds their footing and once going can start folding in the other requirements as we go, great. Who sits down and designs a system for all twelve adjectives ("reliable", "concurrent", "redundant", "runs on a Z80") we want anyhow? One can not help but design a system one adjective at a time, with the others only kept in mind to make sure we don't back into a corner. There's no reason not to interview that way too. (And I tell the candidate this is what we're going to do, so hopefully they don't feel like I'm out to get them or experience requirements whiplash without knowing why I'm doing this.)
Is that important to you? Maybe it is for some people. As a person who has been in team lead and hiring capacity before, it is not for me.
I trained for interviewing at Google twice, but never really chose to give interviews despite that being a highly pressured part of the job because I could not philosophically vibe with that process. But what some people who have adopted this process are missing is that Google etc does this only because they are swamped with resumes and have their pick of bazillions of quality engineers. They don't care about false negatives, more false positives.
Startups and smaller companies should care about false negatives. It's hard to find and retain good people. Smaller companies need to aggressively find and cultivate good people to make good teams in order to be/stay competitive -- and that means accepting a diversity of ways of working.
I've worked on a Python interpreter (not CPython) for many years now, I just never used that particular module (because we do it differently in our implementation). I suppose businesses try their best, but aren't always measuring what they think they are...
Wait, is Z80 seriously something that you want/design for?
https://tvtropes.org/pmwiki/pmwiki.php/Main/ArsonMurderAndJa...
1. My preparation for an interview involves researching the company, not technical matters. I don't brush up on coding interview questions. I've never done leetcode.
2. If I find the interview questions to be ridiculously off-topic (such as silly algorithm questions), I end the interview. You're not the kind of company I want to work with.
3. If I find the questions to be valid, but I can't answer them, then I'm not the right candidate for the job (hopefully I'd already have found this out during the research phase, but we all make mistakes).
4. If we can get past all this gatekeeping to the actual important topic if what BUSINESS issues they're trying to solve, and how I can fit into this process, then we've got a real interview and I'm interested.
So far I haven't been out of work more than a few months.
If you can't determine from a few interviews you want to hire me or not, it's a no from me. It's a HUGE red flag if an employer can't seem to make up their mind so they have to resort to awkward "test drives" like this. If you have a job at the time already, it also makes it almost impossible to pull off without using PTO.
Add all these things together and you really don't want to work for a company who is very inexperienced in hiring such that they resort to tests like these. It shows they don't know how to interview, are not efficient with your time (and probably their time also), and a high chance they are non-technical.
There's a bootstrap issue as well. A developer with a brand name on their resume is much more desirable than one without, even for equal skill and experience.
I generally ask the purpose of the question.
Sometimes there are valid reasons for why they look at very strong algorithm skills and then I simply admit it's not really my cup of tea nor my passion.
Sometimes they answer variations of "it's standard/it's how we do it" so then I propose whether we would like to code something more similar to what the daily job would be to which they generally play along, even happily.
But if some don't I then say that I don't see a fit.
Might be awkward but...who cares? I don't, they won't after 5 seconds the interview's over either.
- people with licenses still do interviews, often just as grueling
- licensed careers with high performers (lawyers, doctors, ib, etc) have other forms of filtering which are much more painful, like years of low pay internships
Studying for a few weeks to solve fun puzzles to make 400k sounds like a deal to me.
The number of people who "aced the interview" but are borderline worthless for real product development is not trivial in my experience.
Agreed, I never asserted against these two points. The point of licensure is to make the first round, which is fairly routine at this point, more equitable and less susceptible to probabilistic effects. It also frees labor from administering this exam round to every candidate.
The licensed engineers I know think software interviews are insane.
Real licenses do not expire without notice. Real license exams are more consistent, have clear pass criteria, and have higher pass rates.
It isn't really a licensure examination because, as you point out, the industry doesn't bother to put the resources in to make sure anything is systemically analysed. It is a process to balance supply and demand of jobs.
Those are your top two problems? When was the last time you saw an exam where the proctor was expected to hold a conversation with the examinees?
Given that Medical boards and the Bar strike fear into the hearts of students and demand immense preparation, I think we could do pretty well. The test prep mentality isn’t altogether different from the “grinding leetcode” that happens today anyway. The difference is that it would at least be fair.
There’s the easy exam that pretty much everyone passes eh e they get their degree (the FE), and then there’s the hard one that not even everyone attempts after a couple years of experience (the PE).
And within each level, you specify your discipline (civil, mechanical, etc) and then are required to have deeper knowledge of several subfields within that discipline.
It would also be easier for the industry to settle on some filters in the exam to block people who just SPAM job postings. For example, open jobs for doctors are only posted on websites that are available to doctors. There is no way for the general public to SPAM job listings for doctors.
Candidates who can code well can point out code that has obvious problems. Just ask if this is good or bad, and if it is bad, how they could improve it. This demonstrates competency and doesn't make the interview seem like a grind but instead more like a conversation.
The best code review interviews are the ones where there is a healthy amount of actual code, with a handful of functions and classes, some badly named variables, bad comments, some misleading code paths, couple bad patterns, etc… the worst ones are a non-optimal solution and you’re asked to make it optimal. That’s just leetcode disguised as “code review”.
"The martini may be simple, but it is not easy to make an excellent one. It's a very solid test of a bartender's skill because, unlike many drinks, ingredients alone cannot carry the cocktail. A piña colada for example, is mostly about ingredients (are you using a good coconut cream? fresh pineapple?) For the martini the chilling and dilution need to be just right. This tests the bartender's most important skill: mixing. Proper mixing of the beverage is ultimately what makes a martini."
[..]
"martinis are shockingly easy to fuck up. and this conversation is exactly the reason why the martini is a good test of a bartender's capability. being a bartender is more than putting fixed quantities of ingredients in a glass. how do you know when your martini is properly diluted, either by shaking or stirring? a good bartender will know. a bad bartender will not. a terrible bartender won't even realize dilution is crucial."
I don't really drink much and never had a martini in my life, but I thought it was pretty interesting.
I know not everyone works with CSV necessarily, but there are dozens of libraries for a lot of languages. Even if focused on N being those supported in a given company/org. It should be less than an hour of work. Bonus points for any tests/automation, etc.
And the patch:
- does fix something but not the described bug.
- could do the same thing in a third of the added line count.
- has typos or other errors.
Last time I got hit with an interview question like that, my answer ultimately had to be "block the merge and counsel the person who wrote this about performance." I'm still not sure if that's the answer they were looking for (this was for a staff engineer position), but I'd stand by it 100% of the time.
I once had a coding interview like this, but the problem was that the code was so obviously bad, I couldn't even make sense of what the code was supposed to do if it were good. It felt like the interviewer had just come up with an example of bad code without any context of how the code would make sense if made "good". It was just totally artifical.
If someone had presented the bad code in some Stack Overflow question, I would have started by stepping back to ask, "What are your trying to do?" Except in this case, the interviewer wasn't actually trying to do anything except quiz me.
Identifying a bug in production code would be better, I think.
This was an effective method to screen out applicants who didn't have the basic coding skills to align with their stated resume experience. And there were a decent number of these.
We did further development project exercises later in the process that took about 15 hours. We paid the candidates for this time, even those that didn't pass. It was also an effective screening tool.
All our exercises were very "real world." In the candidates own development environment and having been given instructions on how to prepare. They also have access to whatever Internet resources they want while doing the exercises. If they can't do the exercises, they can't do the job.
I know there are mixed opinions on this and I feel for candidates who have to invest a ton of time in exercises like this. But I can't imagine trying to hire without visibility into how they execute relatively basic software development tasks.
I think employers can and should structure the process so the time investment is minimal upfront and only increases as both parties have gotten to know each other and want to proceed.
Truth is, I'd much rather hire someone who is interested in working for us for 2-4 weeks as a contractor instead of the normal interview process. But that usually doesn't work for the candidate.