I generally don't mind whiteboard programming interviews but even so asking someone to implement merge sort is a bad question.
My sanity check for any question I ask is- Is the question I'm asking something that someone in the past published a paper about? If so then that's probably not a good fit for a 45 minute whiteboarding interview. This question falls in that category. Generally those algorithms are complex enough that it is unrealistic to expect someone to get them right from first principles so at that point you are just selecting for people who just happened to have dealt with them, memorized them or implemented them recently.
I have a co-worker who used to ask people to past to parse and evaluate infix notation expressions in interviews (and there are companies that ask that question). Always stuck me as weird as multiple papers have been written about that problem, are you really expecting someone to implement a solution to a problem that people spend weeks (months?) thinking about before publishing? Could even Djikstra have come up with the Shunting Yard Algorithm in a 45 minute interview (with implementation on whiteboard) if that was his first exposure to the problem of expression parsing?
My impression of the facebook interview experience was that it is specifically selecting people that were calm under pressure and well rehearsed. They are testing for people that put in the effort to crank out a bunch of algorithms practice because they want to ace an interview. FB and Amazon were also explicit that they were looking for people with a particular "get things done and promote myself" attitude. Whether or not those people make good employees, I don't know.
When I interviewed at FB, every round was something that I had drilled on leetcode or similar sites. Yes, the wording was changed and the problems were altered a bit, but they were the same algorithms. And there were also two "lightning" rounds of 15 minutes each to solve two simpler algorithms problems (maybe easy/medium difficulty on leetcode). This experience gave me the impression that facebook's goals in hiring were to select people who really wanted the job and could pass quizzes well.
One bit of dirty laundry to keep in mind with FAANG and friends is that the interview process is somewhat self-selecting. Often times, the very people that crammed leetcode in the hopes of landing FAANG roles are the ones interviewing candidates three months later. So there this sort of twisted feedback cycle where the interviewers think that leetcode solving ability is a sort of badge of honor or whatever.
Obviously not all interviewers are like that, and at more senior levels, interviewers do place more emphasis on other criteria, but for entry/L4/L5-ish roles, it's a pretty rampant thing.
I got asked to parse infix notation and figured it out in the 45 minutes despite having no prior experience with parsing anything more complicated than CSV. I am not bragging or anything, I just know these discussions are biased towards kvetching so someone has to weigh in for it being possible to accomplish. My field is not computer science and if anything I'm the perfect example of who the infix notation question should be biased against.
To be fair the interviewer described my performance as "muscling through it" (exact phrasing changed to protect the innocent) so take it as you will. :)
I am aware that there is no way to write this comment without sounding like a boast, but that's just more reason to write it because there are probably a large number of people with the same story who can't balance out the discussion by telling it.
This all started back in the 90´s with companies like Microsoft, but at that time the questions were somewhat reasonable, and just reading something like Programming Pearls was most of what you needed to know.
But I suspect what has happened, is that as those hired by this method got promoted, they started asking harder questions... and that has over time escalated into the current situation which has become somewhat ridiculous in terms of pre-study, etc.
> My impression of the facebook interview experience was that it is specifically selecting people that were calm under pressure and well rehearsed.
Yes, the more stressed / slow thinker you are, and the more work you need to put to prepare these interviews. It's not uncommon for people to practice for years.
I don't think these companies exactly want this type of profile, but when you interview 10000s of people, you need some consistent criteria so that all candidates are treated equally. With that constraint, I don't think you can find a better interviewing system.
If you can crank out some moderately complicated algorithms, even having rehearsed them, and get all the little details right, like all the cases that occur, that does show you kind of know what you're doing.
Someone without the proper CS/programming background who is just trying to rote memorize will screw up the details, and not be able to explain anything if asked about it: like what is the purpose of that line of code?
Only problem is, even if this interview question is good at testing that which it is testing, it might not be selecting people along the right dimensions in terms of being suited for the job. It's not probing all the attributes of a software engineer.
Five candidates, A, B, C, D and E have done equally well on the exercise, excellent in fact, and are hired.
Yet, in actual work, things go funny.
A doesn't seem to believe in testing. Deliveries are sloppy and break things all the time. A is proud of her skills and treats criticism of their code as an insult.
B has never heard the words "version" and "control" in one sentence. B works on an out-of-date version of the code to make a change, and then follows the Wiki instructions for getting a fresh checkout (which pulls the latest baseline) and then just copies their previously changed files into the new checkout. Their code submission therefore reverts numerous changes from other people (incompletely: just in those files), while asserting their own changes. On top of that, B makes gratuitous whitespace changes, doesn't follow any consistent coding standard (never mind the prescribed one), and randomly fixes things he sees wrong that do not relate to the topic of the change.
C cannot handle changes in requirements. Even after code base has undergone many changes, C keeps thinking in the same concepts from when the code was two months old and C had been mostly working alone on that code, knowing everything about it. C's ideas about making changes no longer fit the new code. E.g. C cannot remember that some logic is now distributed, running in several places at once, because that's not how it once was, and consistently forgets that aspect.
D cannot say no to any new task. Whenever someone talks to D, D gets distracted and excited and starts looking into what that person was talking about, and doing work on it. Whenever D gets stuck on something, D easily finds something else, and forgets about being stuck, or to report having been stuck. D leaves numerous tickets in an in-progress state, and that list just grows over time.
E is simply lost in any large project. E has done nothing but programming challenges under 1000 lines of code. E has no skills with regard to navigating a large code tree and understanding the structure of something made of tends of thousands of lines (or orders of magnitudes more). E gets paralyzed when faced with making a change to a large system that will involve making some small changes to eight or nine files. But have you seen E's Tetris in five lines of code?
Yep. I think it goes further than whether there's a published paper about. It boils down to "does the question have exactly one correct answer?"
One important thing you learn in interviewer workshops is to avoid asking questions with "aha" components, i.e. things that can only be solved if a very specific insight is known. This type of question is problematic because it falls victim to the fallacious assumption that a candidate that excels in one specific problem must necessarily excel at others that are in actuality, at best, only marginally related.
Instead, it is better to ask questions for which the solution can be arrived at with small increments in effort (e.g. maybe there's an obvious brute force solution and you start from there to discuss more optimal solutions, or maybe the candidate got stuck on some specific peculiarity and you can give a hint to unblock them and move on to another part of the problem to get more signal)
The "aha moment" is also known as the eureka effect. Psychologists call this "insight problem solving", which is terrible for interviewing since like you said since it isn't a problem that you can build up towards solving. As a teacher, I try to make sure my assignments/quizzes don't fall victim of this.
For anyone interested, there are a lot of interesting studies on the topic if you search on Google Scholar. You've probably already seen matchstick arithmetic problems before (i.e., move a single matchstick to make the equation valid).
I don't think whiteboard programming is really a good metric for measuring someone's skills for a normal software engineering job.
However, assuming that extensive knowledge of certain algorithms (like merge sort), data structures, or computational methods are necessary for a particular job, I think a better question to ask isn't "how do you implement merge sort" but instead to present the merge sort algorithm and then ask the interviewee how it works based on the code given. Obviously still not the best way to assess someone's skills, but I think it's better than just memorizing a bunch of algorithms without being able to explain how they work.
> Always stuck me as weird as multiple papers have been written about that problem, are you really expecting someone to implement a solution to a problem that people spend weeks (months?) thinking about before publishing
except that these things aren't research material anymore and are now taught at undergraduate level.
That's the point. No one is solving the problem, they're regurgitating knowledge.
If you care is someone has completed a Computer Science degree, ask to see their resume. If you care if they actually learned the material, ask to see their GPA.
> My sanity check for any question I ask is- Is the question I'm asking something that someone in the past published a paper about? If so then that's probably not a good fit for a 45 minute whiteboarding interview.
Also, is it a widely known algorithm? Because then I'm not measuring for programming ability; I'm measuring for rote memorization (there are people that will rote memorize all the sorting algos to be able to spit them out during interviews!).
Asking for a simply program like wc, in my opinion, gives a much better idea if someone can code (hint: there's not a single way of writing it and to define a word).
> I have a co-worker who used to ask people to past to parse and evaluate infix notation expressions in interviews
That's a cool problem to use. I wouldn't expect a perfect parser or even a working one but I would be curious to see how they approach the problem and what they end up writing.
Say that it's a binary merge sort over a singly linked list.
A binary merge sort, if written recursively, is straightforward divide-and-conquer. The input list is divided into two halves, which are separately sorted via recursion, and then the sorted halves are merged to produce the sorted output.
The algorithm pops out of the definition. If you know the above definition (or the interviewer has provided it, which they should!) the rest is just an exercise which demonstrates that you can:
- completely analyze a problem's cases, including figuring out the existence of cases that are not spelled out in the description (like how to terminate the recursion)
- manipulate pointers
- decompose a problem into small, simple functions that you can separately reason about: (merging two lists, chopping a list in half, and such).
It's not such a bad interview question. I would actually tell all the candidates at least a day ahead of time that they will be implementing merge sort in C using a singly linked list during the interview, so they can study it all they want and rehearse coding it.
It's complicated enough that it can't just be regurgitated from memory by someone who doesn't know what the are doing. (Say, someone who had someone else write the code, and then tried to rote memorize it.) The memory has to be accompanied by understanding if the candidate is to reconstruct it with few or no flaws.
Wirth's "Pascal User Manual and Report" has a listing of a program on pg 75 that does just that. Except it emits the expressions in postfix form rather than evaluating them. It's 31 lines of code, including declarations and `end` statements.
K+R "The C Programming Language" has a desk calculator that evaluates postfix form on pages 74-75 which is about 60 lines of code including lines with just a {, and handles +-*/= including user interface and error checking.
I was just watching a Jon Gjengset "Crust of Rust" stream on sorting, and when his quicksort implementation kept going sideways, I was like "haha how nostalgic".
> I was interviewing at various places, and I got an interview with the Apple Watch team
Funnily enough, I also interviewed with the Apple Watch team around the same time and had a similar negative experience.
I was struggling with a problem on the whiteboard and paused to ask a question. Instead, one of the interviewers (it was a pair) turned to the other and said “I wish the Calendar complication on my watch would show when this meeting ends”
All the other interviewers that day were great, but it just takes a few bad apples… Even though it’s been years and I’ve had a nice tenure at FAANG, I’ll never consider joining Apple after that experience.
A candidate asking questions and trying to unblock himself is an extremely positive signal. It means they are capable of communicating and not start panicking when they hit a blocker.
Ouch! I once interviewed at a local company that used the same hardware and OS as my then employer. When I told the guy where I worked he started laughing. He apologized but he was sympathetic in a you poor bastard kind of way.
I’ve had so many poor interview experiences and all of them at the big tech companies. I once went to google and only 2/5 interviewers bothered to show up.
Hell, in an interview I had this week I was given multiple incorrect answers to my clarification questions. Even when I said “I believe I answered it” the interviewer said “I don’t think so” until he finally looked up and was like “oh you got it”
I’m on zoom watching the interviewer text / look at other screens as I ask clarifying questions and they give dismissive answers “yea yea that sounds right, do that” while never looking up from their phone. Was a pretty “wtf” moment. I completed the interview, but I got the feeling the review will be subpar.
> Trick questions and esoteric puzzles give the interviewer no useful information about the interviewee — her skills, her strengths, her weaknesses, her potential for growth. Those are the things a great interviewer should be seeking out. That means you give hints when the candidate gets stuck and you don’t interrupt them with asinine comments. Whatever problem you give a candidate to solve, your goal as an interviewer should be to get them through as much of that problem as possible, to get as much signal as possible. The added benefit is that the further you get, the better and more confident the candidate feels, which also helps prepare them to do their best for the next session. Less experienced interviewees often need more help, and that’s ok.
This article lays out the problem very well, but then again so to very many others which have appeared on HN over the years.
Why does this keep happening? I mean this at the most fundamental level.
Why waste time with programming puzzlers to begin with? Does the candidate's degree (assuming no job experience) not obviate the need for this kind of thing? Do past programming gigs not absolve them of the need to perform in a way they'll never be asked to perform on the job? Are CS departments so full of bullshitters that unless you sic your 10x devs on them to sniff out whether they can program their way out of a paper bag, your organization runs the risk of filling up with "C Players?"
I used to feel like you and ask the same questions until recently when I read someone on HN saying these interviews are meant to see how a person handles stress. Then it clicked for me.
I've done pretty well at interviews despite being terrible at white board exercises because my personality is such that I treat these interviews casually. It's not that important in the larger scheme of things that I get this particular job, I'm looking at the interviewers as much as they are looking at me to see if I could work with them and I'll often joke around and keep it light hearted, even when failing a white board exercise.
The interviewers know the work culture and are looking for someone that can fit. In the case presented in this blog post, the writer could not handle a small joke about the Swift language. The interviewer probably knew the team like to joke around and tease, so this person was not a good fit. I think it's really that simple. I really doubt the interviewer thought the person was not technically qualified.
In one case, I was in a particularly light hearted mood and I joked throughout the entire interview. I believe to this day I was hired to lighten up the mood of the team, which turned out to be a very serious and gloomy bunch.
> Does the candidate's degree (assuming no job experience) not obviate the need for this kind of thing?
I've met people with degrees who can't program out of a wet paper bag. And also people without a degree who can program circles around anyone. It only seems to be an indicator of your ability to finish a long term project, unless it's a top school. Then it's a slightly more positive indicator.
> Do past programming gigs not absolve them of the need to perform in a way they'll never be asked to perform on the job?
When I was doing interviews for Netflix for Senior Engineers, I always asked FizzBuzz on the phone screen. These were all people who had at least five years of programming experience on their resume with titles like "Senior Engineer".
50% of them were unable to do the assignment in any language with unlimited time.
So it appears that a resume is not sufficient either.
I hate puzzlers and quizzers and never use them (other than FizzBuzz, which frankly anyone who can code should be able to do). But I don't know what exactly we can replace them with.
I personally just ask the person more general questions to understand their depth of knowledge and then explain what the work will entail and ask them if they think they can handle that. Then I hire them and give them work and let them go (or let them quit on their own) if they can't handle the work.
I don't think that programming puzzles are the best way to divine better information in this area. But yes, there's a ton of people looking for a programming job that both have a lot of experience on paper and can't code their way out of a wet paper bag. It seems you can have very long stretches in a large company where you do negative work.
Best way I've found is to talk to them about their experience, sort of post mortem style. Pretty much everyone can talk about their experience in some respects, even those working on classified projects; you just need to respect their boundaries.
I ask tech questions in interviews as a litmus test (for any given concept) and it always relates directly to the work being done so they can get a feel of the kind of code they would be writing and maintaining.
As an example -- in my line of work, one task we perform is processing files, some small, some medium, some very large; and I wanted to make sure a candidate understood the concept of Maps in order to perform O(n) to create a map and O(1) to look up items (generally speaking, I'm sure it's not exactly that in practice), which aide in efficient processing (the age old processor vs memory tradeoff). This particular candidate got everything I asked immediately, and the interview was very short. The person was made an offer soon after and has gone on to learn what we've done, adapt, and also teach us some things that have made the software better.
And the moment candidates grok the concepts I'm after, I confirm that's what I was after, and move on, trying not to waste too much time on any given item. And if they don't get the concepts after a short while, I try to lead them there and see how they react (is it a lightbulb moment or a confusion moment, etc). These are all signs for me to determine if someone is right for the position.
Interviewing is a 2-way street of respect. The interviewer often has all the perceived power, but in my experience, a good interviewer eschews that perception right from the get-go and lets the interviewee know that this is more of a discussion than any kind of trickery or exam, even when it contains coding questions.
This is why I will never interview for Google or any other company that has multi-stage interviews and also employs time-wasteful, and sometimes esoteric, methods such as "How many jelly beans fit in a school bus?"
I wholeheartedly agree with your idea of having a 2-way street of respect, and treating interviews less like interrogations and more like discussions. It certainly feels more natural that way, and puts less pressure on the interviewee.
Regarding Google specifically, I just interviewed there about 2 weeks ago, and although I didn't get the job, I will say I felt very respected, and with the exception of maybe one interview, thought everything felt like a discussion. There were no trick questions, everything was fairly straightforward. Basically a leetcode medium/hard for the 4 technicals and just discussing my experiences in various situations for the "Googleyness" interview.
I have heard similar stories about Google using these trick questions, but from what I understand these were mistakes made in the earlier days of the company and have been effectively phased out, although that is purely speculative.
>This is why I will never interview for Google or any other company that has multi-stage interviews and also employs time-wasteful, and sometimes esoteric, methods such as "How many jelly beans fit in a school bus?"
I said this once upon a time. then I realized I like money and I'm willing to dance a jig if thats what will get me beaucoup RSUs to fund my personal startup venture
I get paid well where I'm at right now. I live a pretty good life. I don't begrudge anyone who works for Google or any other tech giant, I simply set my own standards and I live by them. I find it is a waste of time for me to interview at places that try those kinds of gimmickry. And from what others are saying, it looks like Google doesn't do that anymore nor does it sound like it was widespread, so it may have been simply anecdotal accounts I remember reading.
> I will never interview for Google or any other company that
> asks questions like "How many jelly beans fit in a school bus?"
I conducted more than 100 interviews at Google and saw the questions the other interviewers on each panel asked. Google does not ask questions like that.
Good to hear. They have done that in the past, though, I've read a few first-hand accounts of it (and sadly I cannot find them now so take it with a grain of salt, of course). Perhaps it was just early-days.
> This is why I will never interview for Google or any other company that has multi-stage interviews and also employs time-wasteful, and sometimes esoteric, methods such as "How many jelly beans fit in a school bus?"
Good news: they don't do that any more.
Bad news: you still need to practice the fermi estimation techniques that question employs, but also need to know like, the details of every paper they've published on distributed systems, and bring your own heuristics about computers and email in order to determine how big a GFS cluster you need to launch gmail.
My worst tech interview was with me as the interviewer. An applicant was a friend of a coworker and forwarded the candidate's resume to HR. HR set up a round of interviews.
He went through the five interviews. After, we met for 20 minutes to express our thoughts. Pretty uniformly we agreed: the candidate was middling and we'd be better off waiting for someone better. We rejected him.
A day later the rejected candidate called my phone (how did he get it?) and said something like: "Hey, this is <candidate>. Why the hell wasn't I given an offer? The recruiter said that you (that is me, personally) had rejected me. Explain why." He insisted he had done great and we should reconsider.
The candidate hadn't reacted well to the rejection, and rather than being the firewall that shields the interviewers, the HR guy passed the buck and blamed me. We didn't reconsider, and I gave the HR guy hell about it.
Out of curiosity, what made him middling and not good?
Surely someone at your org could have determined that it's better to stop wasting each other's time before going through 5 interviews?
Sounds like there's more to the story than you're having us believe and the candidate had very good reasons to be upset (the exact wording and tone he used was not appropriate). If he insisted he had done great, I venture to say that he was led on during interviews and not given honest feedback.
What made him middling? It was 15 years ago so I don't recall the details. The usual reasons are things like: didn't answer some questions well, or maybe their resume oversold their accomplishments (eg, "Proficient in verilog, C, C++, pascal, fortran, javascript, java" "OK, describe one of the programs you have written in java." "Well, uh, this one time there was this program that was already written, and I had to add some better error messages when it received bad input.")
Why did he go through the full round of interviews? Because (1) he was referred by an existing employee so interviewer were inclined to give him the benefit of the doubt, and (2) he wasn't terrible, just nothing special; it sometimes happens that one person gets a low impression and the others think they are great and we hire them anyway.
You surmise I must be telling part of the story because his reaction doesn't fit my description of the process. Yes, that is exactly the point of telling the story -- his response wasn't appropriate. Had I felt he had a reason to react that way, I wouldn't be retelling the story.
Author states that the following threw him off: Interviewer: "you must be writing a lot of Swift" after a few minor syntax errors.
Maybe being put off by an ambiguous comment was why you didn't get the job? Did you consider maybe in his notes he might've said: "Smart guy but was put off by a single comment (a failed attempt at humour on my part). Probably not someone who would thrive in a lively atmosphere".
I had an interviewer ask a question and, not satisfied with the answer, get up and walk out. The other guy apologized and we had a good discussion after the other guy left.
> "Smart guy but was put off by a single comment (a failed attempt at humour on my part). Probably not someone who would thrive in a lively atmosphere"
This sounds like a cartoonish training example of the kind of subjective, possibly biased feedback that you absolutely shouldn’t be writing down as an interviewer.
If you’re making jokes like this in an interview, you shouldn’t be an interviewer. There is a clear power imbalance between interviewer and interviewee that needs to be understood clearly.
A somewhat comparable analogy is your manager joking about firing you and then evaluating you negatively when you don’t like the joke.
There is absolutely 0 signal there as to whether someone would "thrive in a lively atmosphere" when you're jumping through hoops for an interview that will determine your financial well being and a good portion of your live for the next n years, and its a fun example of how "engineer social hacks" continuously fail in the real world.
Ah, so you're saying you're financially and emotionally secure enough not to be completely discombobulated by this incredible display of disrespect?
Good for you I guess, but maybe don't apply your own mental blueprint to the rest of the world. Recommending candidates have thicker skin isn't really great a way to improve hiring in general.
An incredible display of disrespect? It might be rude, but that’s about it. Depending on the tone of voice, it could have been intended as a way to relieve the author’s tension.
It does sound a bit annoying, but if that is the author’s worst interviewing experience, it’s not that bad.
i'm not sure you can jump to that conclusion either. there takes a certain level of maturity both emotional and financial at certain skill levels. a psychologist once pointed out to me that actors are able to take their nervousness and translate it into energy to use in the moment.
at a point in my career i faced a similar dilemma but handled it completely differently than i had earlier. someone made a snide comment "but why would you use bash for this?" and i was able to take it in stride and the result of that comment got me the job. but at the start of my career i may have reacted negatively, i may have clammed up and stammered my way through the rest of the interview etc (i didn't get those jobs). could you imagine being in front of a C-level exec or a $300M customer and being discombobulated?
no one wants to hire an insecure pre-sales engineer etc
EDIT: i'm assuming engineering levels beyond senior. my past experiences have been around principal/architect roles which require a 'tough skin'. i would not expect rude remarks to a SE1 or SE2 and if that was the role, i would leave flustered with a bad taste in my mouth as well
Dial the rhetoric down, we can't really read the "context" in which this line was delivered by the interviewer. Glib? Sure. Incredible display of disrespect? Give me a break.
I've had a ton of bad interview experiences, but my worst was still Google. This was circa 2012 or so, was my 2nd session of an on-site day. I guess I didn't grow up around a lot of people with thick accents because I could literally not understand a single word this guy was saying, and he didn't seem to understand me. He wanted me to explain the solution before writing any code, but since we were having communication issues I asked if I could write pseudocode.
He slammed (SLAMMED) his fist onto the table and yelled, "NO CODE!"
-----
Other bad experiences included Palantir telling me my onsite went really well, and that they'd like me to come in again to meet the hiring manager. Expecting to talk business, I instead was presented with 2 more algorithm questions. Those apparently went OK, so the next discussion with the recruiter yielded another "informal" (his words) meeting with 2 engineers via VC. Both ended up being 2 more DS/Algo questions.
-----
Another bad one I remember is a request to do a take-home coding test from Docusign (also circa 2012). They wanted me to program a "fully tested" elevator simulation using Stylus, Coffeescript, and Jade. I had never used any of these technologies, so I asked if I could write it in what I was familiar with (jQuery, Backbone, SCSS, ES5), which was denied. It would have taken me a full week to get up to speed on all of those technologies enough to submit a solution worthy of review. Absolutely ridiculous to ask a candidate to learn new frameworks just to complete the interview.
Had an interview yesterday where the guy was seemingly more nervous than I was, and he seemed to mask that nervousness with anger/being rude. I would answer his questions, and he would act like I wasn't understanding the question. In reality, it turns out I wasn't answering his questions using the _exact_ buzzwords he was looking for. I realized this when I happened to mention a buzzword after minutes of tense back-and-forth and he immediately perked up and we could suddenly move on. Went into the next interview feeling like shit because of this.
This whole process is so broken. Great devs get passed up on for terrible devs who know how to grind Leetcode and the like. I hate having to grind leetcode for weeks just to end up getting asked obscure algorithm puzzle questions that I could never prepare for. And all for a 100% React job? Jesus.
Same problem with a poor English speaker at Google back in ~2019. It took several minutes just to start understanding the words he was saying. I have many years of working/growing up with ESL folks and have a good ear for it and still struggled with this guy.
Readit News
My sanity check for any question I ask is- Is the question I'm asking something that someone in the past published a paper about? If so then that's probably not a good fit for a 45 minute whiteboarding interview. This question falls in that category. Generally those algorithms are complex enough that it is unrealistic to expect someone to get them right from first principles so at that point you are just selecting for people who just happened to have dealt with them, memorized them or implemented them recently.
I have a co-worker who used to ask people to past to parse and evaluate infix notation expressions in interviews (and there are companies that ask that question). Always stuck me as weird as multiple papers have been written about that problem, are you really expecting someone to implement a solution to a problem that people spend weeks (months?) thinking about before publishing? Could even Djikstra have come up with the Shunting Yard Algorithm in a 45 minute interview (with implementation on whiteboard) if that was his first exposure to the problem of expression parsing?
When I interviewed at FB, every round was something that I had drilled on leetcode or similar sites. Yes, the wording was changed and the problems were altered a bit, but they were the same algorithms. And there were also two "lightning" rounds of 15 minutes each to solve two simpler algorithms problems (maybe easy/medium difficulty on leetcode). This experience gave me the impression that facebook's goals in hiring were to select people who really wanted the job and could pass quizzes well.
Obviously not all interviewers are like that, and at more senior levels, interviewers do place more emphasis on other criteria, but for entry/L4/L5-ish roles, it's a pretty rampant thing.
To be fair the interviewer described my performance as "muscling through it" (exact phrasing changed to protect the innocent) so take it as you will. :)
I am aware that there is no way to write this comment without sounding like a boast, but that's just more reason to write it because there are probably a large number of people with the same story who can't balance out the discussion by telling it.
But I suspect what has happened, is that as those hired by this method got promoted, they started asking harder questions... and that has over time escalated into the current situation which has become somewhat ridiculous in terms of pre-study, etc.
Yes, the more stressed / slow thinker you are, and the more work you need to put to prepare these interviews. It's not uncommon for people to practice for years.
I don't think these companies exactly want this type of profile, but when you interview 10000s of people, you need some consistent criteria so that all candidates are treated equally. With that constraint, I don't think you can find a better interviewing system.
Someone without the proper CS/programming background who is just trying to rote memorize will screw up the details, and not be able to explain anything if asked about it: like what is the purpose of that line of code?
Only problem is, even if this interview question is good at testing that which it is testing, it might not be selecting people along the right dimensions in terms of being suited for the job. It's not probing all the attributes of a software engineer.
Five candidates, A, B, C, D and E have done equally well on the exercise, excellent in fact, and are hired.
Yet, in actual work, things go funny.
A doesn't seem to believe in testing. Deliveries are sloppy and break things all the time. A is proud of her skills and treats criticism of their code as an insult.
B has never heard the words "version" and "control" in one sentence. B works on an out-of-date version of the code to make a change, and then follows the Wiki instructions for getting a fresh checkout (which pulls the latest baseline) and then just copies their previously changed files into the new checkout. Their code submission therefore reverts numerous changes from other people (incompletely: just in those files), while asserting their own changes. On top of that, B makes gratuitous whitespace changes, doesn't follow any consistent coding standard (never mind the prescribed one), and randomly fixes things he sees wrong that do not relate to the topic of the change.
C cannot handle changes in requirements. Even after code base has undergone many changes, C keeps thinking in the same concepts from when the code was two months old and C had been mostly working alone on that code, knowing everything about it. C's ideas about making changes no longer fit the new code. E.g. C cannot remember that some logic is now distributed, running in several places at once, because that's not how it once was, and consistently forgets that aspect.
D cannot say no to any new task. Whenever someone talks to D, D gets distracted and excited and starts looking into what that person was talking about, and doing work on it. Whenever D gets stuck on something, D easily finds something else, and forgets about being stuck, or to report having been stuck. D leaves numerous tickets in an in-progress state, and that list just grows over time.
E is simply lost in any large project. E has done nothing but programming challenges under 1000 lines of code. E has no skills with regard to navigating a large code tree and understanding the structure of something made of tends of thousands of lines (or orders of magnitudes more). E gets paralyzed when faced with making a change to a large system that will involve making some small changes to eight or nine files. But have you seen E's Tetris in five lines of code?
One important thing you learn in interviewer workshops is to avoid asking questions with "aha" components, i.e. things that can only be solved if a very specific insight is known. This type of question is problematic because it falls victim to the fallacious assumption that a candidate that excels in one specific problem must necessarily excel at others that are in actuality, at best, only marginally related.
Instead, it is better to ask questions for which the solution can be arrived at with small increments in effort (e.g. maybe there's an obvious brute force solution and you start from there to discuss more optimal solutions, or maybe the candidate got stuck on some specific peculiarity and you can give a hint to unblock them and move on to another part of the problem to get more signal)
For anyone interested, there are a lot of interesting studies on the topic if you search on Google Scholar. You've probably already seen matchstick arithmetic problems before (i.e., move a single matchstick to make the equation valid).
It's way too complicated to code on a whiteboard. I would never have asked somebody that question at Google.
However, assuming that extensive knowledge of certain algorithms (like merge sort), data structures, or computational methods are necessary for a particular job, I think a better question to ask isn't "how do you implement merge sort" but instead to present the merge sort algorithm and then ask the interviewee how it works based on the code given. Obviously still not the best way to assess someone's skills, but I think it's better than just memorizing a bunch of algorithms without being able to explain how they work.
except that these things aren't research material anymore and are now taught at undergraduate level.
If you care is someone has completed a Computer Science degree, ask to see their resume. If you care if they actually learned the material, ask to see their GPA.
Also, is it a widely known algorithm? Because then I'm not measuring for programming ability; I'm measuring for rote memorization (there are people that will rote memorize all the sorting algos to be able to spit them out during interviews!).
Asking for a simply program like wc, in my opinion, gives a much better idea if someone can code (hint: there's not a single way of writing it and to define a word).
> I have a co-worker who used to ask people to past to parse and evaluate infix notation expressions in interviews
That's a cool problem to use. I wouldn't expect a perfect parser or even a working one but I would be curious to see how they approach the problem and what they end up writing.
A binary merge sort, if written recursively, is straightforward divide-and-conquer. The input list is divided into two halves, which are separately sorted via recursion, and then the sorted halves are merged to produce the sorted output.
The algorithm pops out of the definition. If you know the above definition (or the interviewer has provided it, which they should!) the rest is just an exercise which demonstrates that you can:
- completely analyze a problem's cases, including figuring out the existence of cases that are not spelled out in the description (like how to terminate the recursion)
- manipulate pointers
- decompose a problem into small, simple functions that you can separately reason about: (merging two lists, chopping a list in half, and such).
It's not such a bad interview question. I would actually tell all the candidates at least a day ahead of time that they will be implementing merge sort in C using a singly linked list during the interview, so they can study it all they want and rehearse coding it.
It's complicated enough that it can't just be regurgitated from memory by someone who doesn't know what the are doing. (Say, someone who had someone else write the code, and then tried to rote memorize it.) The memory has to be accompanied by understanding if the candidate is to reconstruct it with few or no flaws.
Wirth's "Pascal User Manual and Report" has a listing of a program on pg 75 that does just that. Except it emits the expressions in postfix form rather than evaluating them. It's 31 lines of code, including declarations and `end` statements.
K+R "The C Programming Language" has a desk calculator that evaluates postfix form on pages 74-75 which is about 60 lines of code including lines with just a {, and handles +-*/= including user interface and error checking.
Funnily enough, I also interviewed with the Apple Watch team around the same time and had a similar negative experience.
I was struggling with a problem on the whiteboard and paused to ask a question. Instead, one of the interviewers (it was a pair) turned to the other and said “I wish the Calendar complication on my watch would show when this meeting ends”
All the other interviewers that day were great, but it just takes a few bad apples… Even though it’s been years and I’ve had a nice tenure at FAANG, I’ll never consider joining Apple after that experience.
A candidate asking questions and trying to unblock himself is an extremely positive signal. It means they are capable of communicating and not start panicking when they hit a blocker.
Hell, in an interview I had this week I was given multiple incorrect answers to my clarification questions. Even when I said “I believe I answered it” the interviewer said “I don’t think so” until he finally looked up and was like “oh you got it”
I’m on zoom watching the interviewer text / look at other screens as I ask clarifying questions and they give dismissive answers “yea yea that sounds right, do that” while never looking up from their phone. Was a pretty “wtf” moment. I completed the interview, but I got the feeling the review will be subpar.
"Sorry, it looks like I must have caught you at a bad time. If you have a fire to put out, I totally get it."
Deleted Comment
This article lays out the problem very well, but then again so to very many others which have appeared on HN over the years.
Why does this keep happening? I mean this at the most fundamental level.
Why waste time with programming puzzlers to begin with? Does the candidate's degree (assuming no job experience) not obviate the need for this kind of thing? Do past programming gigs not absolve them of the need to perform in a way they'll never be asked to perform on the job? Are CS departments so full of bullshitters that unless you sic your 10x devs on them to sniff out whether they can program their way out of a paper bag, your organization runs the risk of filling up with "C Players?"
I mean, what's really going on here?
I've done pretty well at interviews despite being terrible at white board exercises because my personality is such that I treat these interviews casually. It's not that important in the larger scheme of things that I get this particular job, I'm looking at the interviewers as much as they are looking at me to see if I could work with them and I'll often joke around and keep it light hearted, even when failing a white board exercise.
The interviewers know the work culture and are looking for someone that can fit. In the case presented in this blog post, the writer could not handle a small joke about the Swift language. The interviewer probably knew the team like to joke around and tease, so this person was not a good fit. I think it's really that simple. I really doubt the interviewer thought the person was not technically qualified.
In one case, I was in a particularly light hearted mood and I joked throughout the entire interview. I believe to this day I was hired to lighten up the mood of the team, which turned out to be a very serious and gloomy bunch.
I've met people with degrees who can't program out of a wet paper bag. And also people without a degree who can program circles around anyone. It only seems to be an indicator of your ability to finish a long term project, unless it's a top school. Then it's a slightly more positive indicator.
> Do past programming gigs not absolve them of the need to perform in a way they'll never be asked to perform on the job?
When I was doing interviews for Netflix for Senior Engineers, I always asked FizzBuzz on the phone screen. These were all people who had at least five years of programming experience on their resume with titles like "Senior Engineer".
50% of them were unable to do the assignment in any language with unlimited time.
So it appears that a resume is not sufficient either.
I hate puzzlers and quizzers and never use them (other than FizzBuzz, which frankly anyone who can code should be able to do). But I don't know what exactly we can replace them with.
I personally just ask the person more general questions to understand their depth of knowledge and then explain what the work will entail and ask them if they think they can handle that. Then I hire them and give them work and let them go (or let them quit on their own) if they can't handle the work.
Best way I've found is to talk to them about their experience, sort of post mortem style. Pretty much everyone can talk about their experience in some respects, even those working on classified projects; you just need to respect their boundaries.
As an example -- in my line of work, one task we perform is processing files, some small, some medium, some very large; and I wanted to make sure a candidate understood the concept of Maps in order to perform O(n) to create a map and O(1) to look up items (generally speaking, I'm sure it's not exactly that in practice), which aide in efficient processing (the age old processor vs memory tradeoff). This particular candidate got everything I asked immediately, and the interview was very short. The person was made an offer soon after and has gone on to learn what we've done, adapt, and also teach us some things that have made the software better.
And the moment candidates grok the concepts I'm after, I confirm that's what I was after, and move on, trying not to waste too much time on any given item. And if they don't get the concepts after a short while, I try to lead them there and see how they react (is it a lightbulb moment or a confusion moment, etc). These are all signs for me to determine if someone is right for the position.
Interviewing is a 2-way street of respect. The interviewer often has all the perceived power, but in my experience, a good interviewer eschews that perception right from the get-go and lets the interviewee know that this is more of a discussion than any kind of trickery or exam, even when it contains coding questions.
This is why I will never interview for Google or any other company that has multi-stage interviews and also employs time-wasteful, and sometimes esoteric, methods such as "How many jelly beans fit in a school bus?"
Regarding Google specifically, I just interviewed there about 2 weeks ago, and although I didn't get the job, I will say I felt very respected, and with the exception of maybe one interview, thought everything felt like a discussion. There were no trick questions, everything was fairly straightforward. Basically a leetcode medium/hard for the 4 technicals and just discussing my experiences in various situations for the "Googleyness" interview.
I have heard similar stories about Google using these trick questions, but from what I understand these were mistakes made in the earlier days of the company and have been effectively phased out, although that is purely speculative.
I said this once upon a time. then I realized I like money and I'm willing to dance a jig if thats what will get me beaucoup RSUs to fund my personal startup venture
Good news: they don't do that any more.
Bad news: you still need to practice the fermi estimation techniques that question employs, but also need to know like, the details of every paper they've published on distributed systems, and bring your own heuristics about computers and email in order to determine how big a GFS cluster you need to launch gmail.
Is that better? I'm not so sure.
He went through the five interviews. After, we met for 20 minutes to express our thoughts. Pretty uniformly we agreed: the candidate was middling and we'd be better off waiting for someone better. We rejected him.
A day later the rejected candidate called my phone (how did he get it?) and said something like: "Hey, this is <candidate>. Why the hell wasn't I given an offer? The recruiter said that you (that is me, personally) had rejected me. Explain why." He insisted he had done great and we should reconsider.
The candidate hadn't reacted well to the rejection, and rather than being the firewall that shields the interviewers, the HR guy passed the buck and blamed me. We didn't reconsider, and I gave the HR guy hell about it.
Surely someone at your org could have determined that it's better to stop wasting each other's time before going through 5 interviews?
Sounds like there's more to the story than you're having us believe and the candidate had very good reasons to be upset (the exact wording and tone he used was not appropriate). If he insisted he had done great, I venture to say that he was led on during interviews and not given honest feedback.
Why did he go through the full round of interviews? Because (1) he was referred by an existing employee so interviewer were inclined to give him the benefit of the doubt, and (2) he wasn't terrible, just nothing special; it sometimes happens that one person gets a low impression and the others think they are great and we hire them anyway.
You surmise I must be telling part of the story because his reaction doesn't fit my description of the process. Yes, that is exactly the point of telling the story -- his response wasn't appropriate. Had I felt he had a reason to react that way, I wouldn't be retelling the story.
Maybe being put off by an ambiguous comment was why you didn't get the job? Did you consider maybe in his notes he might've said: "Smart guy but was put off by a single comment (a failed attempt at humour on my part). Probably not someone who would thrive in a lively atmosphere".
I had an interviewer ask a question and, not satisfied with the answer, get up and walk out. The other guy apologized and we had a good discussion after the other guy left.
This sounds like a cartoonish training example of the kind of subjective, possibly biased feedback that you absolutely shouldn’t be writing down as an interviewer.
A somewhat comparable analogy is your manager joking about firing you and then evaluating you negatively when you don’t like the joke.
“Oh hey jumping between languages is hard, looks like you’re thinking in Swift right now, no worries about fumbling on syntax happens all the time”
Good for you I guess, but maybe don't apply your own mental blueprint to the rest of the world. Recommending candidates have thicker skin isn't really great a way to improve hiring in general.
It does sound a bit annoying, but if that is the author’s worst interviewing experience, it’s not that bad.
at a point in my career i faced a similar dilemma but handled it completely differently than i had earlier. someone made a snide comment "but why would you use bash for this?" and i was able to take it in stride and the result of that comment got me the job. but at the start of my career i may have reacted negatively, i may have clammed up and stammered my way through the rest of the interview etc (i didn't get those jobs). could you imagine being in front of a C-level exec or a $300M customer and being discombobulated?
no one wants to hire an insecure pre-sales engineer etc
EDIT: i'm assuming engineering levels beyond senior. my past experiences have been around principal/architect roles which require a 'tough skin'. i would not expect rude remarks to a SE1 or SE2 and if that was the role, i would leave flustered with a bad taste in my mouth as well
He slammed (SLAMMED) his fist onto the table and yelled, "NO CODE!"
-----
Other bad experiences included Palantir telling me my onsite went really well, and that they'd like me to come in again to meet the hiring manager. Expecting to talk business, I instead was presented with 2 more algorithm questions. Those apparently went OK, so the next discussion with the recruiter yielded another "informal" (his words) meeting with 2 engineers via VC. Both ended up being 2 more DS/Algo questions.
-----
Another bad one I remember is a request to do a take-home coding test from Docusign (also circa 2012). They wanted me to program a "fully tested" elevator simulation using Stylus, Coffeescript, and Jade. I had never used any of these technologies, so I asked if I could write it in what I was familiar with (jQuery, Backbone, SCSS, ES5), which was denied. It would have taken me a full week to get up to speed on all of those technologies enough to submit a solution worthy of review. Absolutely ridiculous to ask a candidate to learn new frameworks just to complete the interview.
This whole process is so broken. Great devs get passed up on for terrible devs who know how to grind Leetcode and the like. I hate having to grind leetcode for weeks just to end up getting asked obscure algorithm puzzle questions that I could never prepare for. And all for a 100% React job? Jesus.
It looks like these folks really bought into it.