I think there are some flaws in this analysis that can be confirmed when the code is released.
I think the reason Scala seems surprisingly popular is that is was simple string matching so a post that said "We are looking for someone who can build scalable infrastructure" would count as one mention of "scala".
I worked for a company, which is still in business, 80% of whose work involves "cleaning/normalizing" data, basically everything that isn't sales and client services.
Market research is all about getting different kinds of data and massaging it into queryable and productizable form (reports).
Probably the same for rust, looking at the October 2018 data set with https://kennytilton.github.io/whoishiring/ rust gives 50 results, trust gives 21, \brust gives 26. Similar results for scala.
Yeah, similarly with "Excel". Are there really that many posts looking for people with deep spreadsheet experience? Seems more likely that it's a lot of false positives for "excellent" and "excel at".
Yup. JavaScript, Node.js, Node, npm, React, Angular, Vue, Express and a few more words mean pretty the same thing and are fishing for the same pool of developers (Yes, I know there's a difference between Angular and React.)
I've noticed that in the hiring threads, a particular company is always hiring, and the email contact username is "austin". It would be easy to pick this up as a location even though it's a person's name.
This analysis has the Austin as one of the top 10 locations, and I'm curious whether or not that is skewing the numbers. There are a steady stream of jobs actually in Austin, so either seems possible.
Same with product manager. Many descriptions list working with product management as a requirement. There have not been 425 product management jobs listed this year. Trust me, I've applied to most of them.
Agreed. It looks like the author may be looking for something like a /.net/ regexp? I don't think .net is popular at all in HN job postings. A better regex would be pretty much what you suggest:
It appears that the same problem occurs with locations. For example it would appear that the small town of Visby (Sweden) which is 1/100 of the size of Stockholm has 1/10 the jobs of Stockholm. Or more likely the location matches the company name Visby of startup visby.io.
There are numerous ways to fix the glaring flaws in this analysis but a very simple approach would be using regex word boundaries, that way it'll work not only for the space but for commas, periods, end-of-line, etc.
OP here. As some people were commenting, Scala and excel I guess have some false positives which reflect the unusual popularity. I did try to limit the search to exact word, but think that I overlooked that part. I will try fix the code and update the skills chart soon. Thanks for pointing it out.
Edit: I fixed the false positives using a word boundary on regex "\\bword\\b". The data now seems little sane, will keep looking for any false positives. updated the data and regenerated the chart using new data.
I'd also expect some tough to catch exceptions for remote, as that's something I always look at and despite the posting guidelines, posters can't stop themselves from adding things like "sorry, no remote available at this time" or "part-time remote might be an option for the right person in some roles".
I get that you're trying to be complete, but if it's not a truly remote position, it's not remote. put nothing about remote in the position or at the very most "REMOTE: NO". Working from home 10% of the time but needing to be in a SF office the rest of the time is not a remote position; it's the flexibility I'd expect from most jobs in this industry.
Thanks. I added a note at the skills section about the false positives. Really should have paid attention, but i guess i was just excited to show it off, first :(. I am on it fixing the regex.
There seem to be errors in geo-locations as well. No results for Zurich while it's the biggest hub in Switzerland. A suspiciously high number in Venice, Italy (I think it's safe to assume California here).
I completely relied on the python package "GeoText" for locations.I see Geneva and Lausanne from Switzerland, i will have to double check if the package missed it or there were not many posts on the hiring thread. Thanks for pointing it out.
OP, thanks for the analysis. Really appreciate it. Quick question - Does San Francisco include abbreviations like "SF" or "San Fran" ? Does it also include jobs in the Valley towns like Palo Alto and San Mateo ?
Take that all you people crying "Scala is dead" after Java 8+ comes out. /s
Anyway, I've personally seen (and interviewed for) jobs that were advertising Scala (or plans to move to Scala, or that Scala experience is a plus) but actually meant there was one proof-of-concept project in Scala, 18 months ago, nobody ever touches it nowadays, the person who wrote it is long gone and by the way, our legacy JDK5 application is on fire, do you mind fixing that for the rest of your life?
Oh for sure; I'd take any job listing with something cool with a huge grain of salt; hardly any proper software development employer actually uses hip technology, but they will use hip terms to try and draw in developers.
Hype-driven development is a problem, and hype-driven recruitment another one. Just be a good java developer. Maybe push for Kotlin slowly. Don't pursue Scala for the language's sake.
See also: blockchain, IoT, etc. I've seen a job advert that basically bolted those onto the not well hidden "we're just looking for decent java developers" job description.
Cool to see that other folks find this data interesting. I've been analyzing languages/frameworks/skills in the "Who is Hiring" posts for a few years (https://www.hntrends.com/) as well.
I know it's just one part of the analysis, but the skills list appears to be be pretty far off of what I've been seeing. React gets over 200 a month by itself. Are you capturing all pages of the postings in each month's thread? Here's the data (counts) I have through November, broken down by month and term - https://www.hntrends.com/data/data-20181101.js
The fact that Rust and Python are quite high in the rankings, makes me hopeful about the future of programming again.
The fact that remote work is not growing is a bit concerning, but if people from the valley want to overpay - it is none of my business. I hope eventually the market will correct it.
Not OC, but for me it's just a real wonder why it's not better embraced at this stage of the game. We have all the tools to accommodate collaboration within remote teams and in (most) places the broadband to handle it. Add to this the continued funneling of companies into these metro areas where COL is high (NYC, SF, Seattle) and thus people may find themselves being forced into higher commute times just to attain a better COL situation.
I personally am commuting close to 2 hours each way, so 4 hours total, because the job market is much stronger in NYC then my immediate (30-45 minute) area. If a job is open around here the salaries are almost 30-40% lower than NYC despite our COL still being high.
Not OP but due to a lack of investment in public transportation in NA over the previous decades, many people spend a few hours a day commuting and in many cases this is done in private vehicles that pollute. For many in our profession there is no need to go to an office to work.
Not OP, but I'd like the world to be moving toward a paradigm where physical location is not a significant factor in career / pay / advancement. It makes things more "meritocratic," and it puts pressure on some of these big tech hubs to keep their costs of living competitive.
But I don't want to see it if the market doesn't justify it. I think today there is something beneficial to having people working together in an office, but I'd prefer to be proven wrong by some new remote-work management style (or something). And software development is one of the more ideal use cases for remote work, so if it's not expanding for us, it's less likely to expand in other industries.
Python is deeply troubling. It is a regression from FORTRAN and COBOL. Long ago, we invented compile-time type checking. The benefits for software quality have been enormous. There isn't really a downside here, as there would be with the performance loss of garbage collection or bounds checking. Shaking out lots of bugs before even attempting to test the software is a wonderful advance that we made half a century ago. Python's incompatibility with compile-time optimization is also horrifying. The situation is so extreme that you can't even make a decent-performing JIT.
You're awfully unfamiliar with the history of computer science if you truly believe that dynamic typing is some sort of recent invention or that static typing is some panacea that solves all of your problems.
Both systems of type checking have existed as long as... Programming languages have existed, essentially and, sadly, so have the endless comparisons and flamewars.
Of everyone who asked for work or applied for work via these posts - how many secured a position?
I've long given up on these posts trying to find freelance work or other permanent work. There's to much competition (here) to stand out from the flood of replies.
Anecdotally, the one time I posted a job, I had about 80 replies for a remote work junior engineering job where I specified the salary at $60k/yr in the post (this was in 2015). Of those 80, about 10 were promising and I probably would have made an offer to at least one or two if we hadn't shifted priorities and stopped the hiring process.
It probably understates the SF market, TBH. As some of the largest employers in SV who hire en masse rarely post jobs in HN. Such as Google, Facebook, Apple, Salesforce, Uber etc. Also, Google, FB, Uber, Airbnb and others have a boot camp system, where job postings do not correlate to hiring since one posting is used to hire an entire bootcamp class
Readit News
I think the reason Scala seems surprisingly popular is that is was simple string matching so a post that said "We are looking for someone who can build scalable infrastructure" would count as one mention of "scala".
Could be wrong though.
Also see: https://news.ycombinator.com/reply?id=18725655&goto=item%3Fi...
Honestly, it is an insult to call good data preprocessing skills grunt work.
It may not be as attractive, but requires a lot of understanding of the nature of data and way in which patterns present themselves.
Market research is all about getting different kinds of data and massaging it into queryable and productizable form (reports).
I would be pretty surprised to see these two so much more commonly requested than Javascript.
When can you start?
This analysis has the Austin as one of the top 10 locations, and I'm curious whether or not that is skewing the numbers. There are a steady stream of jobs actually in Austin, so either seems possible.
https://hnprofile.com/compare?search=scala,python,rust
Not the only issue I see, for instance my company posted for “machine learning engineers”, yet that isn’t in the results at all.
I think they are doing regex search in the strings, as opposed to identifying the words and doing comparisons
Because there are a multitude of possibilities.
Edit: I fixed the false positives using a word boundary on regex "\\bword\\b". The data now seems little sane, will keep looking for any false positives. updated the data and regenerated the chart using new data.
I get that you're trying to be complete, but if it's not a truly remote position, it's not remote. put nothing about remote in the position or at the very most "REMOTE: NO". Working from home 10% of the time but needing to be in a SF office the rest of the time is not a remote position; it's the flexibility I'd expect from most jobs in this industry.
"Sorry, no couples!"
https://en.wikipedia.org/wiki/Null_Island
And same would go for visa. "We do not sponsor VISA"
I really think you're going to have to do some sort of sentiment analysis, either via human or machine.
Deleted Comment
Anyway, I've personally seen (and interviewed for) jobs that were advertising Scala (or plans to move to Scala, or that Scala experience is a plus) but actually meant there was one proof-of-concept project in Scala, 18 months ago, nobody ever touches it nowadays, the person who wrote it is long gone and by the way, our legacy JDK5 application is on fire, do you mind fixing that for the rest of your life?
Hype-driven development is a problem, and hype-driven recruitment another one. Just be a good java developer. Maybe push for Kotlin slowly. Don't pursue Scala for the language's sake.
See also: blockchain, IoT, etc. I've seen a job advert that basically bolted those onto the not well hidden "we're just looking for decent java developers" job description.
https://www.hntrends.com/2018/nov-react-still-top-containers...
I know it's just one part of the analysis, but the skills list appears to be be pretty far off of what I've been seeing. React gets over 200 a month by itself. Are you capturing all pages of the postings in each month's thread? Here's the data (counts) I have through November, broken down by month and term - https://www.hntrends.com/data/data-20181101.js
I personally am commuting close to 2 hours each way, so 4 hours total, because the job market is much stronger in NYC then my immediate (30-45 minute) area. If a job is open around here the salaries are almost 30-40% lower than NYC despite our COL still being high.
But I don't want to see it if the market doesn't justify it. I think today there is something beneficial to having people working together in an office, but I'd prefer to be proven wrong by some new remote-work management style (or something). And software development is one of the more ideal use cases for remote work, so if it's not expanding for us, it's less likely to expand in other industries.
Python is deeply troubling. It is a regression from FORTRAN and COBOL. Long ago, we invented compile-time type checking. The benefits for software quality have been enormous. There isn't really a downside here, as there would be with the performance loss of garbage collection or bounds checking. Shaking out lots of bugs before even attempting to test the software is a wonderful advance that we made half a century ago. Python's incompatibility with compile-time optimization is also horrifying. The situation is so extreme that you can't even make a decent-performing JIT.
Both systems of type checking have existed as long as... Programming languages have existed, essentially and, sadly, so have the endless comparisons and flamewars.
Of everyone who asked for work or applied for work via these posts - how many secured a position?
I've long given up on these posts trying to find freelance work or other permanent work. There's to much competition (here) to stand out from the flood of replies.
My advice: skip this site and weworkremotely.com. Both are complete wastes of time based on the responses.
If you’re in IT, your best bet is LinkedIn (for a referral) or Careerbuilder/Dice/Indeed. At least those will result in face to face interviews.
Also find companies you like and apply directly on their website.
* SF is center of the tech world. Next biggest is just around half of its size. (not withstanding HN bias).
* Approx 25% jobs allow visa sponsorship
* HTML, Python, .Net overwhelmingly dominates everything else.
* Reduced interest in databases, SQL, Obj-C, Java
* No TensorFlow or PyTorch in demand
* 90% jobs in development/software engineering, 10% in management and misc.
The article lists SF as having 2922 jobs and New York 1746 jobs