Readit News logoReadit News
PragmaticPulp · 4 years ago
Link to actual study: http://chrisparnin.me/pdf/stress_FSE_20.pdf

The study participants were split into two groups. One group solved the problem alone, in silence. The other group was instructed to narrate their thought process out loud to an interviewer who stood over their shoulder (literally, see the photo on page 3) while they solved it.

The flaw in this study is that the two groups weren't really assigned the same task. Solving a problem isn't the same as verbally narrating your solution to a problem. The authors attributed the differences not to the extra work narration work assigned to the second group, but to anxiety levels.

The participants even explained that their difficulty came from having to talk while solving:

> Participants also had difficulty with performing tasks that involved multiple simultaneous actions. Participants felt stressed by having to “talk while trying to write” (P44), “talking while writing” (P25), and “think and talk and do code at the same time” (P39). P41 found it difficult to “constantly speak during solving” and “lost breath at a few places during the task”.

There are other concerns with the study format such as many of a significant number of participants simply giving up well before the clock ran out, in both the private and public interview groups. They were given 30 minutes to solve 3 problems, but some of the participants (including in the private group) were giving up around the 10 minute mark without solving anything at all. This suggests a very high variance of the underlying abilities of their candidates, which necessitates a much larger sample size to draw conclusions.

SomeCallMeTim · 4 years ago
It sounds like the study points to the idea that "talk through your reasoning" is a bad interview approach.

The last interview sequence I needed to do they put me in front of an online IDE and said "go code." They also said I could use my OWN IDE if I had it set up. One even said "you can finish it later (post-interview) if you want."

The candidates giving up: Some nontrivial percentage of CS students are just ... not capable developers. That's why we use programming tests to begin with.

Programming tests may be the worst possible way to assess candidates, but they also seem to be the only way to assess candidates that doesn't involve paying them for a trial period, which really only works for companies large enough to be able to waste a lot of money. "Democ­ra­cy is the worst form of gov­ern­ment, except for all the oth­ers."

ziggus · 4 years ago
Having candidates talk through the solution to a non-trivial programming problem is a critical tool for me. I need to find developers that can not only code, but work as a part of a team - something that requires significant communications abilities.

I can't afford to hire developers that can't effectively communicate complex ideas to other developers - something they're often required to do on-the-fly.

arebop · 4 years ago
Probationary hiring also doesn't work that well for many candidates. Despite at-will employment, there's an understanding about W2 white collar employment that gives new hires the confidence to buy and sell houses, move across the country or around the world, terminate talks with other prospective employers and pause interviewing, etc. This is a cultural thing that could change over time, but it would be a significant change.
mprovost · 4 years ago
This is a great point about CS grads. Maybe in the past, you could just take a degree at face value: "oh you graduated from Stanford with an A average, you must know how to code, here's a job". But nobody trusts any university to produce capable developers, so you have to test everyone with crazy interviews. So then what's the point of going to a top CS school and why can they keep charging so much tuition?
chris11 · 4 years ago
> It sounds like the study points to the idea that "talk through your reasoning" is a bad interview approach.

That idea definitely makes sense to me. And I think some devs do definitely underperform in interviews.

Though I'm not sure if there's great alternatives. As an interviewer I feel that seeing someone work through a problem and debug issues gives useful information. I don't know if there's a better, less anxiety-producing way of testing those things.

I do think take homes can be really useful, but those have their own problems. Maybe leaving someone alone onsite to complete the problem would be an option. But I think leaving the discussion to the end would give less info about the candidate's communication skills.

omarhaneef · 4 years ago
That would be a flaw in the sense that anxiety may not be the right word, but it does capture the parallel with white board interviews and underscore the problem that you may be turning away people who can perform the job well but not code while explaining.
kerblang · 4 years ago
Am I reading this incorrectly or did literally all the female candidates finish the test in private while none of them finished in public?

(disclaimer: Am male but still cannot talk and hack at the same time)

Jensson · 4 years ago
Also the effect basically disappears if you take only male candidates, so the study is a bit suspect. Men and women typically aren't that different in studies.

Deleted Comment

Manuel_D · 4 years ago
Bear in mind there's a sample size of four women.
sharps_xp · 4 years ago
> Participants also had difficulty with performing tasks that involved multiple simultaneous actions. Participants felt stressed by having to “talk while trying to write” (P44), “talking while writing” (P25), and “think and talk and do code at the same time” (P39). P41 found it difficult to “constantly speak during solving” and “lost breath at a few places during the task”.

This is why I built myself a tool to practice talking and narrating my thoughts! https://enumerable.co

IshKebab · 4 years ago
> The flaw in this study is that the two groups weren't really assigned the same task.

The real flaw is that the title doesn't follow from the research. "Tech sector job interviews assess anxiety, not software skills." That's saying that you're really measuring level of anxiety, not skill. So skill is irrelevant and you're just measuring how anxious people are.

But the study didn't show that at all. It just showed that on average, people perform better when they're not anxious. Not exactly a surprising result!

Of course people are going to be more anxious in an interview. But if it affects everyone equally who cares? Just make your questions easier.

tonyedgecombe · 4 years ago
I would assume it doesn’t affect people equally.

Then the question is whether or not the anxiety will become a problem in the workplace. I can see arguments for both sides.

Deleted Comment

b20000 · 4 years ago
maybe you should get in touch with the professor and tell him? curious what the answer will be.

i think it’s reasonable to think that this study is relevant and that there is truth in it.

there is also the issue that every interviewer i encountered assumes lying and incompetence and will conduct the interview in such a way that the result is what they want to see - the candidate failing to write the code within a few minutes. they are not capable of letting a candidate do the interview without biasing the interview and results.

this also comes from the automatic assumption that because they work in a FAANG they are a fucking genius and are the only ones who can write software. i can see this happening as they are so brainwashed and lost all touch with reality and how the rest of the industry works.

meanwhile there are some marketing execs at that FAANG with a compensation package far above what they will ever get paid.

mrfusion · 4 years ago
Does it matter if it’s anxiety or not?
tonyedgecombe · 4 years ago
I think so, unless you expect your workplace to generate the same levels of anxiety during day to day work.
im3w1l · 4 years ago
Yes. If it's anxiety you can do things known to reduce anxiety. E.g. taking a beta blocker such as propranolol.
kodah · 4 years ago
As someone who didn't go to school, I struggle with doing algorithms and implementing data structures in front of someone. Outside of timeboxed interviews I negotiate them fairly well, but I'm almost always seeing something for the first time. Then again, for as long as I've been in software it's always been tailored for academics. This seems to be just fine and even championed as a good thing.

My reaction is to plan my exit from the industry whenever this job comes to an end. I'll end up going back to school in my thirties, despite being qualified, because of self-imposed constraints by this industry that have little if anything to do with the tasks I'll be given or planning day to day.

gameswithgo · 4 years ago
I think this can’t really be the reason for your going back to school. A few weeks of drilling of common data structures and algorithms would put you at parity with college grads. Could it be you just don’t like the industry and project that onto some kind of academic bias or something? Nothing wrong with not liking software development it is often awful.
kodah · 4 years ago
It is, I plan on going back to school to be an EE when that times comes.

> A few weeks of drilling of common data structures and algorithms would put you at parity with college grads.

I don't know where one gets this kind of idea. I also stated fairly clearly that I have no trouble negotiating algorithms and data structures, the problem is when they're timeboxed and involve an interviewers participation.

I think my main objection is to even doing these kinds of interviews. When, in the world, would one need to write out binary search and would have only 30 minutes to do so? Knowing the properties of different algorithms and data structures is important, imo, but implementing them in a timeboxed manner with abstract problems seems like an odd activity to demonstrate qualification.

> Could it be you just don’t like the industry and project that onto some kind of academic bias or something?

Academic Bias may play less of a role than my perception makes it out to be, but I'd at least say it's significant in this industry. The way I think it gets in is subtle, catering questions and problems that one might find in an academic setting is one. My company also heavily hires interns, which again facilitates an academic bias in hiring, but there is no program for people who don't come from a prestigious university. The lingo used usually points to an academic background as well; I've learned "top talent" usually refers to prestigious companies (that primarily hire from academia) or prestigious universities.

taurath · 4 years ago
As someone who's also self taught and currently going through the drilling process... its pretty tough to actually pull in and retain on an encyclopedic level all the information one needs, especially on concepts used less frequently in any sort of day to day - especially when in a timeboxed environment with an interviewer watching and evaluating my every move. Think also on the amount of people who are years out of college who struggle with these problems, given that they haven't actually used them in many years.

An example of an extremely common question: “Write a function to check if a binary tree is a binary search tree.”

Yes you can memorize an answer here, and the implementation of a binary tree - I think most people just do that. Knowing the inns and outs of a binary tree is probably useful if you're writing a database, or a compression format. If they're hiring someone with the capability to write a database, or compression format, this question makes plenty of sense. It also makes sense if your criteria for hiring is "did they study and understand an algorithms course?". It doesn't make much sense if they're looking for someone who can do the actual job, in my opinion.

bb88 · 4 years ago
> I think this can’t really be the reason for your going back to school.

What precisely makes you're more expert at the OP's experience then the OP?

vincentmarle · 4 years ago
As a fellow college dropout who had the same insecurities and actually went back to college in his thirties (part-time) to study exactly this... all that knowledge still doesn't prepare you for time-boxed stressful leet code interviews.

The people you are competing with study leet code for 3 months and get really good at solving coding puzzles within 45 minutes. Study the basic data structures (nothing fancy needed beyond binary trees) and solve leet code an hour a day for 3 months. That's all you need to do.

unanswered · 4 years ago
> The people you are competing with study leet code for 3 months

You can keep competing with the B and C players if you want but don't give that advice to others.

politician · 4 years ago
That's a shame. Most software development jobs don't actually require a degree and getting a degree is no guarantee that you'll be able to solve the whiteboard questions.
nickff · 4 years ago
There are many employers (such as mine,) which do not use these theoretical questions, and instead focus questions on prior project work. That said, many of the high-prestige employers (and their former employees) do use the tests you're talking about, largely because they see themselves as exclusive. If you're aiming for one of those prestige employers, you should be aware that they're also exclusive about which schools they recruit from.
irq · 4 years ago
Or you can ignore employers who interview like that and keep searching until you find one that doesn’t. A lot of people do this for the same reason you outlined.
annoyingnoob · 4 years ago
I gave up on software interviews. Everyone wants to follow what FB and Google do to 'find the best people'. In my experience, 'keep searching until you find one' is bad advice.

Can't tell you how many companies I interviewed with that simply take their entire process from others. We interview like X, we group teams/squads like Y, we do our agile process like Z. I don't want to work for your wannabe something else organization, get a clue.

jonfw · 4 years ago
A degree to learn data structures and algorithms would be a massive waste. I think I had 2 classes that focused on them
raydev · 4 years ago
But didn't you have classes where some subset popped up again and again?

That's why I still encourage people to pursue a Bachelors degree over bootcamp if they have minimal programming experience prior to going to school. If I'd taken only the "intro to ds" and only "intro to algos" courses to get my degree I would've been poorly equipped to start my first job, since that was the first time I'd seen those concepts.

oriolid · 4 years ago
I think the idea of the algorithm interview is to see if you were awake in class while getting the degree and whether you can think on your feet. For other classes it's more difficult to come up with questions that aim for the same.
lunch · 4 years ago
Art school graduate here, and my only advice is to broaden your sights and keep trying (if you want to). I've interviewed with FAANG too many times to count and every time it's crashed and burned on the comp-sci quizzes. Fuck em, keep looking. You'll eventually find a team that doesn't demand solving riddles that aren't applicable to your day-to-day CRUD. Take these jobs and you'll quite possibly learn the skillsets, or meet enough people, to get into the Big Co's without the riddles. It worked for me.

I feel like I have PTSD from Google and Amazon interviews early in my career.

Deleted Comment

lifeplusplus · 4 years ago
My enter CS degree had one course on data structures and it didn't make me expert at white boarding ... Doing 40 mock interviews helped me
throwaway81523 · 4 years ago
Disclaimer: I haven't read TFA yet (will look at it later). But I've interviewed software people, detected anxiety sometimes, and made a mental note to not hold it against them. They're not interviewing for sales jobs, they are nerds, and it's fine for nerds to be shy introverts, especially around strangers.

If you're interviewing someone like that, try to give them a little breathing space, and maybe even tell them that you are noticing the nervousness and that it is ok. If you're being interviewed, I think it is ok to say you are a little bit nervous, take a few deep breaths to calm yourself down, etc.

me_im_counting · 4 years ago
I never had test anxiety growing up and didn't really empathize with why someone would (sympathized obviously). But solving a problem in FRONT of someone as they are evaluating me (rather than a normal collaboration) triggers a panic. The only thing that helped reduce the panic is repeated drills, which take a lot of time.
WalterSear · 4 years ago
IMHO, you can't really drill for technical interviews. The landscape is way too large for one to be fresh on whatever they ask you to do.
oneepic · 4 years ago
I disagree because:

1) you don't have to know everything, only the topics they happen to ask. You are rolling the dice, but part of the time you'll be lucky. Besides, after reviewing for a few months (also did 4 yr degree in CS) I felt really strong in the algo topics. That said, I started half of my recent Google interviews thinking "how the fuck do I solve this? is this where I fail?"

2) Aside from the topics, you can absolutely drill the process. Drill a basic flow like understand/clarify the question, do examples, mention a brute-force, etc. when you do LC problems.

mywittyname · 4 years ago
Sure you can:

1. HackerRank (or similar) challenges are pretty close to what you'd find in a lot of technical interviews.

2. Searching online for "<technology> interview questions" for a few of the technologies that you're likely going to be asked about. Make sure you have good answers for the questions that pop up a lot. This helps a lot with remembering Stuff You Should Know that kind of slipped your mind (ie., angular digest cycle) because you maybe haven't seen it in a bit.

3. Write up a summary of your previous accomplishments and be sure you can call them out on the spot.

A lot of what the interviewer is looking for is confidence. And preparation begets confidence.

wombat-man · 4 years ago
learning some tips and tricks has made it all a lot less spooky. I hate that it's like this but it is kind of about that nowadays, for certain companies anyway. I guess they assume you have unlimited time to study, so if know enough it shouldn't be too hard. But if you are just walking in there cold as an experienced applicant, you can end up having a really bad time.
me_im_counting · 4 years ago
It's not drilling on the technical side. It's drilling the experience that produces anxiety (performing it to an interviewer).
cloverich · 4 years ago
You can. I've worked at a few startups and participated in interviews, witnessing the variety of questions asked. For most companies its a limited pool of questions and well trodden ground.
jstx1 · 4 years ago
Personal anecdote - several months ago I got an interview at a prestigious big tech company. I couldn't sleep properly for at least a week before the interview and I failed. I also track my bodyweight and I can see a spike up just after the interview and then another one after I got rejected - I gained over 10kg (22lbs) in the following weeks while before that I was successfully losing weight for a while.
yellow_lead · 4 years ago
It's hard to not be nervous for these interviews especially when coming from a non-FANG / huge tech company. The salary difference and opportunity can very much change your life. I don't have advice for this, but what you said resonates with me.
raydev · 4 years ago
Personally, I did the phone screen for FB when it would've been a lifechanging 6x compensation boost. And after a couple years trying out at lower-tier-but-still-Bay-Area companies I landed a job that was "only" 4x.

I made it to the onsite the second time and failed that, but it mattered less since the money, while still significant, was no longer life-changing.

You've probably heard it before, but.... yeah. Practice. All the failures you accumulate do eventually add up to knowledge and some successes.

Unfortunately, the failures seem to come at great cost to my physical and mental health. But the money is good.

joelbluminator · 4 years ago
Are you anxious in general? Or do you have some issue with tests? I hate algorithm interviews like the next guy but it sounds like you have it worse than most.
jstx1 · 4 years ago
In general I'm not anxious person at all, quite the opposite. I found it very surprising myself. I actually prefer algorithm interviews, and I had never really been anxious for an interview before. This was also remote - I suspect that it would have been easier in person (I know for a lot of people it would be the opposite though).
ssully · 4 years ago
I don't have it this bad, but I certainly have major anxiety during interviews. I was interviewing for a position at my current company and I blanked on what wget did. Literally a utility I used frequently and I completely blanked on it. So I bombed that interview, but I got a job with another group at the same company. I still get secondhand embarrassment whenever I think about it.
curiousgal · 4 years ago
As someone struggling to gain weight, alright!
jstx1 · 4 years ago
It wasn't the kind of weight you want to be gaining.
rsyring · 4 years ago
Our process is very evidence based. We use skills tests that are as close to real world as possible, given time limitations. We can't remove all anxiety, but we make an effort to mitigate it and limit its impact.

Details of our interview and assessment process is at the bottom of each job listing: https://www.level12.io/careers/

PragmaticPulp · 4 years ago
> Details of our interview and assessment process is at the bottom of each job listing: https://www.level12.io/careers/

I clicked the first job listing and didn't find anything about the interview or assessment.

They do have a weird note about the company's "Biblical principles" though:

> Level 12 was founded on biblical principles and has biblically informed Values.

Can you please write out your process here instead of making us search for it on your website?

alpaca128 · 4 years ago
Perhaps we have to look it up in the Bible?

> We exist to serve our customers, our employees, our partners, and our community in a way that brings them genuine benefit, honors Jesus Christ, and advances the Kingdom of God.

Well, that's not what I expected from a software development website. And I can't speak for others, but this alone would be a reason I wouldn't even apply.

devoutsalsa · 4 years ago
I assume it’s doing what we all do… pushing the code, praying it works, and hoping to avoid being crucified.
keithnz · 4 years ago
yeah, I saw that, that whole thing would be illegal in this country.
rsyring · 4 years ago
Expand "Next Step - Ready to apply". And it's at the very bottom. Yes, the UX is not ideal. The alternative is that the entire post is really long and yet, people really want this information.

Pasted verbatim without taking time to format:

The Rest of the Process

Our application process is outlined roughly below.

But, before you get there, we want to apologize in advance if this process seems...imposing. We have put considerable thought and refinement into each one of these steps in an effort to make sure our hiring process is as well crafted as our software. And just like software, hiring is a lot more complex than it might seem on the surface. Our hiring process is far from perfect (like our software), we are still learning and tweaking, but we want to assure you that each of these steps gives us crucial information regarding you and your development abilities that is essential for helping us to determine if this is a good match.

Consider this: our entire process is less than a week's worth of effort to make sure that where you spend the next 1-5 years of your life is a good fit. Isn't that worth it? Keep in mind that we have deliberately structured our process so that the earlier stages require less effort. Our hope is that if you make it to the later stages of our process, where the time commitment increases, you will have had a chance to get to know us a bit better so you can decide if the time investment on your part is worth it. We care about your time (and ours) and do our best not to waste it! Application Steps

    Evaluate resume and initial email correspondence
    Technical skills questionnaire
    Skills evaluation: 60-90 minute work simulation exercise
    Zoom interview(s): 45-90 minutes in one or two interviews to get to know you & your technical abilities
    Skills tests - phase I: three real-world programming challenges, no trick questions here (paid)
    Skills tests - phase II: project-based skills test: we give you a small project description and you build the best app you can (paid)
    Skills tests review interview: 2-3 hours on a Zoom meeting with our dev leadership team to get to know you and review your skills test results
    Collaborative work day:
        As close to a typical work day as we can get. We just want to see what it's like to work with each other.
        We'll assign you work based on a previous real-world project we performed This is a sample project, we're not using candidates for free or cheap labor.
        We will be available via Slack or Zoom throughout the day to talk through the work and assist you as needed.
If at any step we don't feel like it's a good match, we'll let you know promptly. We ask that you do the same for us.

Dead Comment

WalterSear · 4 years ago
> We use skills tests that are as close to real world as possible, given time limitations.

If you are timing limiting candidates, your process is not evidence based and I question how serious you are about making efforts to mitigate or limit the effects of anxiety on applicants.

rsyring · 4 years ago
Why do you believe time constraints and evidence based processes are antithetical?

You can't give someone an unlimited amount of time to take an assessment and still expect to meaningfully be able to compare results. One person does a good job in 1 hour another person does a good job but it takes them 10 hours. Time limitations are a fact of life. Interviewees are only willing to give so much time to the application/hiring process and we have similar constraints. The time constraints are reasonable for the tasks given.

We are working to mitigate anxiety in our process, I don't think we can ever completely eliminate it.

tharne · 4 years ago
What's going on with "anxiety" lately? This went from something you heard about here and there 20-30 years ago, to something that everyone and their brother is feeling all the time.

What changed?

handrous · 4 years ago
Shitloads of adults were on anti-anxiety meds (and anti-depressants, for that matter) 20 years ago. Plenty more self-medicated for those symptoms with alcohol or illegal drugs. One of the big shocks of my growing up was learning that, more or less, "everyone and their brother" in fact relied on mood- & perception-altering drugs, legal or otherwise, to get through the day.

And they all still do (except that the weed and some of the psychedelics may be legal now). It may just be zeitgeisty, now.

Then again, I'm not sure coverage of it is even greater now than it was in past decades. Future Shock was published in, what, the 70s? Or Affluenza? Bowling Alone's not new, though that treats of more than just anxiety. Direct associations between first The City and anxiety were so common they got really tropey in the first part of the 20th century, and later (starting in the 1940s and '50s) the suburban middle class and their (alienating, unfulfilling, and sometimes nearly or entirely useless, as in Graeber's Bullshit Jobs, the core observations of which were made by others all the way back at the start of the modern postwar economy) office jobs got similar treatment, which has continued ever since.

If there's a difference, I expect it's because people are using anxiety to cover more states of mind than it used to. It does seem to have become a blanket term for "mentally ill-at-ease, but maybe not full-on mentally ill" in some usage.

Or maybe it's just that Future Shock was right, in which case we'd expect that kind of thing to get worse over time, until something about the pace of change itself changes.

Deleted Comment

Apocryphon · 4 years ago
Postwar America was huge into self-medication. You watch a film like Invasion of the Body Snatchers and characters are casually popping pills and swigging scotch every other scene. Before that, you had patent medicine shows and snake oil remedies and puritanical types inventing Graham crackers and corn flakes to promote temperance. The U.S. has always had a self-medicating, consumerist approach to health, for whatever reason.
satellite2 · 4 years ago
What's changed is basically an entire pan of medicine, the one dedicated to sleep emerged. Also, we now have a much deeper understanding of stress, or anxiety, its effect on sleep, on hormones, on the reproductive system, on rational decision making. We also have a better understanding of cortisol, the role it plays in increasing inflammation and all the problems associated with it. We also better understand how low level and punctual source of stress can also be beneficial for performances and focus, and how it is dose and duration dependant. Basically, we don't simply discard emotions as unrelated to the body anymore. We have a better and more integrated view of body and the mind and all that entails.
erellsworth · 4 years ago
The fact that you didn't hear about it much 20-30 years ago doesn't mean it wasn't just as common. Admitting to mental health problems is much more socially acceptable today than it was 30 years ago.
iab · 4 years ago
Economic uncertainty in a fundamentally unfair economic system on a planet that is literally degrading in front of our eyes, if I was to hazard a guess
taurath · 4 years ago
The pandemic has made most people a bit more aware, at least. The distractions people enjoyed were suddenly taken away, for at least over a year. That has heightened awareness, along with the need for self care.

Even the idea of anxiety was extremely stigmatized, especially in the workplace, and still is but is now less so. What you're seeing is a growth of people talking about it as the stigma lessens.

mikkelam · 4 years ago
Honestly I think we just started paying more attention to the subject. Anxiety is a completely natural survival mechanism: “was that a tiger in those bushes!?”. It keeps us alive.
danaris · 4 years ago
5,000 years ago it kept us alive.

Now it just adds unnecessary stress and impedes our functioning 99.8% of the time. We no longer live in a situation where a tiger lurking in the bushes is likely, but our brains haven't evolved to compensate for that.

GDC7 · 4 years ago
It's the internet.

The Internet is amazing!*

*As long as there is a ban on discussing people , once you start discussing (and most importantly) advertising people on the internet , it's over.

Endless comparison ensues and anxiety skyrockets for everybody.

Humans were never made to be aware of being 1 unit in an 8B sample. The person sitting for an interview has high anxiety because they know they can be replaced easily

Apocryphon · 4 years ago
Things could be better (less concern about items lower on Maslow's hierarchy allows one to worry about self-actualization and other items at the top), and it could also be worse (concern about the basic lower needs leads to anxiety). For engineers who are missing out on the boom times caused by the bubble, it might be a mix of both.
jayd16 · 4 years ago
You're comparing, what, the booming 90s to now?

That said, I don't think anyone ever thought interviews were anxiety free.

Animats · 4 years ago
"For this study, researchers conducted technical interviews of 48 computer science undergraduates and graduate students."

Ah, the convenience sample problem.

tehjoker · 4 years ago
For what it's worth, that sample would be semi-accurate for people trying to get their first job, though not mid-late career.