Peeking at the source, it's just a zero-width div, which is not accomodating of people with disabilities. This might open you up to litigation if you disqualify a blind person on the grounds he gave the wrong answer 'using AI', when he might have just been answering the question his screen reader read out.
Instead of the zero-width div, you could set up an event listener for the copy event (using addEventListener() method) which calls .clipboardData.setData() on the ClipboardEvent to change it to your modified code.
That should avoid messing things up for people with screen readers while still trapping the copy+pasters.
There's an easier fix. Have the candidates state if they have a vision disability first and then send them down a different pathway for validation. There aren't that many, so it's not going to be costly or anything.
I'm pretty sure the intent is to weed people out well before they get to a point where you could share a screen with them. He mentioned a few people "resubmitted the application", so sure this is probably an initial step.
What most interviews get wrong is that there are usually just a few "bullet points" that if you see them, you instantly know that the the candidate at least has the technical chops.
Instead of creating a test that specifically aims for those bullet points, many technical assessments end up with convoluted scaffolding when actually, only those key bullet points really matter.
Like the OP, I can usually tell if a candidate has the technical chops in just a handful of really straightforward questions for a number of technical domains.
If you have a test that can identify a good candidate quickly then you have honestly struck gold and can genuinely use that to start your own company. I mean this with absolute sincerity.
One of the absolute hardest part of my business is really hiring qualified candidates, and it's really demoralizing and time consuming and unbelievably expensive. The best that I've managed to do is the same that pretty much every other business owner says... which is that I can usually (not always) filter out the bad candidates (along with some false negatives), and have some degree of luck in hiring good candidates (with some false positives).
Good candidates are not universally good, they are good for you after hire.
One of the best business analysts I worked with (also a profession, mind you) was almost fired when working under an old, grumpy and clearly underskilled one.
I was hired once without interview into a unicorn, was loved by colleagues but hated the work, the business and the industry, then left rather quickly.
See? There are mismatches and unknown unknowns, not just bad or good developers.
Yeah sourcing developers / collaborating with developers is a huge barrier of entry. More than others factors such as deciding what software to produce for the market
But along that thought, I’ve always held that a human conversation is the best filter. I’ll ask you what do you work on recently, what did you learn, what did you mess up, what did you hate about the tool / language / framework.
I strongly believe your ability to articulate your situation corresponds with your ability to do the job.
Take your current technical assessment and think about the types of responses or code submissions that really impressed you. What was special about them? What did you see in the response that drew a positive reaction from your team?
Can you re-frame your process or the prompt to only elicit those specific responses?
So instead of a whole exercise of building a React app or a whole backend API, for example, what would really "wow" you if you saw the candidate do it in the submission for a project? Could you re-frame your whole process so you only target those specific responses and elicit specific outputs?
Now that you've taken what was previously a 2 hour coding exercise (for example) and distilled down to 3-4 key questions, you can seek the same outputs in 15-30 minutes instead.
There are several advantages to this:
1) Many times, candidates know the answer but they actually can't figure out what you're looking for when there's a lot of cruft around the question/problem being solved. You can end up losing candidates that know how to solve the problem the way you want, but because of the way the question was posed, the objective is opaque.
2) It saves a lot of time for both sides. Interviewer doesn't have to review a big submission, candidate doesn't have to waste time doing a long project.
3) By condensing the cycle, you can evaluate more candidates and you can hopefully select a top candidate before they get another opportunity. You shorten the cycle time because the candidate doesn't have to find free time to do a project or sit down for long interviews, you don't need to have people review the code submissions, etc.
There are quite a few people who can code but have 0 social skills and can’t talk well or at all. In that sense I have to disagree with you, but I still wouldn’t hire them because they drag teams down.
For anyone who missed the (poorly-explained) trick, the website uses a CSS trick to insert an equals sign, thus showing different code if read or if copied/pasted. That's how the author knows whether you solved it in your head or pasted it somewhere.
The thing I found particularly fascinating, because the article was talking about discarding AI applicants, is that if you take a screenshot and ask ChatGPT, then it works fine (of course, it cannot really see the extra equals sign)
So this is not really foolproof, and also makes me think that feeding screenshots to AI is probably better than copy-pasting
Sure, it's not foolproof, but a large percentage of folks would just copy&paste rather than taking the screenshot. Now that may start to change.
There was story like this on NPR recently where a professor used this method to weed out students who were using AI to answer an essay question about a book the class was assigned to read. The book mentioned nothing about Marxism, but the prof inserted unseeable text into the question such that when it was copy&pasted into an AI chat it added an extra instruction to make sure to talk about Marxism in relation to this book (which wasn't at all related to Marxism). When he got answers that talked extensively about the book in Marxist terms he knew that they had used AI.
Thanks, I was wondering how in the hell that many would get the answer wrong and what is this hidden equal sign he was talking about.
Maybe the question could be flipped on its head to filter further with "50% of applicants get this question wrong -- why?" to where someone more inquisitive like you might inspect it, but that's probably more of a frontend question.
People are focusing on the > vs the >=, but for me the key point is being able to hold logic and variables in your mind over several iterations of a loop.
I’ve used similar tests in interviews before (a function that behaves like atoi and candidates have to figure out what it’s doing) and the good candidates are able to go over the code and hold values in their head across multiple iterations of a loop.
Other way around.. if you use reader mode you generate the wrong answer ("tricked" like an LLM). At least, it's wrong according to the author/a former CTO.
It also excludes users of Lynx, cURL, possibly people using accessibility tools, those with custom/overriding style sheets.
exactly the reason why you NEVER should copy-paste code from a website into your terminal, even if that has paste protection (https://lwn.net/Articles/749992/)
Was that the trick? When copying the text, it is also >=, which is why an online search or AI tools probably give the wrong answer as the article asserts. If you correct the code then at least Claude gives the right answer.
Which is literally the point of the post. They have the = in >= at an opacity of 0 and a font-size of 1, which means it doesn't appear if styles are used properly. And their point is that candidates that copy/paste such trivial code into an AI/interpreter will get -11 because it just sees the >=.
Though a gap in their process is that, as you mentioned, various reading modes also remove styles and likewise might see it. Though a reader should filter out opacity:0 content.
Not just an "AI solver", a Python interpreter will also do it "wrong". The idea is that it's so simple that anyone qualified should be able to do it in their head, so they get the answer without the equals sign (but IMO a qualified applicant might also know it takes 5 seconds to run it in the repl and it'd be better to be correct than to use the fewest tools, or might be using a screen reader).
I think it’s important to test these systems. Let some % of candidates who get this wrong through to the next stage and see what happens. Does failing this test actually correlate with being a bad fit later?
If you want to ineffectivly filter out most candidates just auto-reject everything that doesn’t arrive on a timestamp ending in 1.
It depends why they didn't get it "correct" (asked ChatGPT bad, used Python REPL not so bad, used screen reader very not bad) and what "correct" even means for this problem.
There's a bizarro version of this guy who rejects people who do it in their head because they weren't told to not use an interpreter and he values them using the tools available to solve a problem. In his mind, the = is definitely part of the code, you should have double checked.
> Let some % of candidates who get this wrong through to the next stage and see what happens.
This isn't a good methodology. To do your validation correctly, you'd want to hire some percentage of candidates who get it wrong and see what happens.
Your way, you're validating whether the test is informative as to passing rate in the next stage of your hiring process, not whether it's informative as to performance on the job.
(Related: the 'stage' model of hiring is a bad idea.)
It's not for getting an interview with the CTO, but a very early filter to weed out poor candidates. But yes, if that's the only question then it's not going to discover talent.
Readit News
That should avoid messing things up for people with screen readers while still trapping the copy+pasters.
If someone is visually impaired, it's short enough you can just read the problem text to them.
I'm pretty sure the intent is to weed people out well before they get to a point where you could share a screen with them. He mentioned a few people "resubmitted the application", so sure this is probably an initial step.
Deleted Comment
Instead of creating a test that specifically aims for those bullet points, many technical assessments end up with convoluted scaffolding when actually, only those key bullet points really matter.
Like the OP, I can usually tell if a candidate has the technical chops in just a handful of really straightforward questions for a number of technical domains.
One of the absolute hardest part of my business is really hiring qualified candidates, and it's really demoralizing and time consuming and unbelievably expensive. The best that I've managed to do is the same that pretty much every other business owner says... which is that I can usually (not always) filter out the bad candidates (along with some false negatives), and have some degree of luck in hiring good candidates (with some false positives).
One of the best business analysts I worked with (also a profession, mind you) was almost fired when working under an old, grumpy and clearly underskilled one.
I was hired once without interview into a unicorn, was loved by colleagues but hated the work, the business and the industry, then left rather quickly.
See? There are mismatches and unknown unknowns, not just bad or good developers.
But along that thought, I’ve always held that a human conversation is the best filter. I’ll ask you what do you work on recently, what did you learn, what did you mess up, what did you hate about the tool / language / framework.
I strongly believe your ability to articulate your situation corresponds with your ability to do the job.
Can you re-frame your process or the prompt to only elicit those specific responses?
So instead of a whole exercise of building a React app or a whole backend API, for example, what would really "wow" you if you saw the candidate do it in the submission for a project? Could you re-frame your whole process so you only target those specific responses and elicit specific outputs?
Now that you've taken what was previously a 2 hour coding exercise (for example) and distilled down to 3-4 key questions, you can seek the same outputs in 15-30 minutes instead.
There are several advantages to this:
1) Many times, candidates know the answer but they actually can't figure out what you're looking for when there's a lot of cruft around the question/problem being solved. You can end up losing candidates that know how to solve the problem the way you want, but because of the way the question was posed, the objective is opaque.
2) It saves a lot of time for both sides. Interviewer doesn't have to review a big submission, candidate doesn't have to waste time doing a long project.
3) By condensing the cycle, you can evaluate more candidates and you can hopefully select a top candidate before they get another opportunity. You shorten the cycle time because the candidate doesn't have to find free time to do a project or sit down for long interviews, you don't need to have people review the code submissions, etc.
So this is not really foolproof, and also makes me think that feeding screenshots to AI is probably better than copy-pasting
There was story like this on NPR recently where a professor used this method to weed out students who were using AI to answer an essay question about a book the class was assigned to read. The book mentioned nothing about Marxism, but the prof inserted unseeable text into the question such that when it was copy&pasted into an AI chat it added an extra instruction to make sure to talk about Marxism in relation to this book (which wasn't at all related to Marxism). When he got answers that talked extensively about the book in Marxist terms he knew that they had used AI.
Maybe the question could be flipped on its head to filter further with "50% of applicants get this question wrong -- why?" to where someone more inquisitive like you might inspect it, but that's probably more of a frontend question.
I’ve used similar tests in interviews before (a function that behaves like atoi and candidates have to figure out what it’s doing) and the good candidates are able to go over the code and hold values in their head across multiple iterations of a loop.
There are many candidates who can’t do this.
better yet, listen to a 1h meeting and compare notes/action points
But then the question is, how do you reach people who filter out the job ads?
It also excludes users of Lynx, cURL, possibly people using accessibility tools, those with custom/overriding style sheets.
Thus, if you get the wrong answer, you "cheated" (or used reader mode)
Though a gap in their process is that, as you mentioned, various reading modes also remove styles and likewise might see it. Though a reader should filter out opacity:0 content.
Not just an "AI solver", a Python interpreter will also do it "wrong". The idea is that it's so simple that anyone qualified should be able to do it in their head, so they get the answer without the equals sign (but IMO a qualified applicant might also know it takes 5 seconds to run it in the repl and it'd be better to be correct than to use the fewest tools, or might be using a screen reader).
Deleted Comment
Dead Comment
If you want to ineffectivly filter out most candidates just auto-reject everything that doesn’t arrive on a timestamp ending in 1.
Really, the better test would be to not discriminate on it before you know it's useful, but store their answer to compare later.
There's a bizarro version of this guy who rejects people who do it in their head because they weren't told to not use an interpreter and he values them using the tools available to solve a problem. In his mind, the = is definitely part of the code, you should have double checked.
This isn't a good methodology. To do your validation correctly, you'd want to hire some percentage of candidates who get it wrong and see what happens.
Your way, you're validating whether the test is informative as to passing rate in the next stage of your hiring process, not whether it's informative as to performance on the job.
(Related: the 'stage' model of hiring is a bad idea.)