It is similar to an article I read a while back saying we need "data engineers" not "data scientists". I think its generally true. Why my company, while it can do data science consulting, is choosing to focus on data engineering consulting first.
Readit Newsyachtman commented on Everyone wants to do the model work, not the data work [pdf] storage.googleapis.com/pu... · Posted by u/Anon84
Sorry to hear that. If you could share your approximate location, then I'll gladly investigate.
Sure thing. I won't here as that would be violating TOS for site
This comment breaks the Show HN guidelines: https://news.ycombinator.com/showhn.html and arguably the site guidelines also: https://news.ycombinator.com/newsguidelines.html
It's great to give helpful feedback about bugs, but please edit out swipes like "Major fail" when doing so. They're especially noxious when someone's presenting their own work. We want people do to that freely on HN, and get helpful feedback, not internet nastiness.
Whoa, sorry I think you added more emotional personal attack meaning to major fail than I intended. I major fail all the time, it's part of life and getting better.
Dead Comment
yachtman commented on How much your computer can do in a second (2015) computers-are-fast.github... · Posted by u/MindGods
Yes, it's definitely time for some slightly enhanced armchair activism!
Convert all active whatsapp contacts to signal? 80%? 50%?
Same for Facebook messenger!
It needs to be uncomfortable enough - if they want to communicate with us and are minimally tech savvy ("click link to install app") they have to switch to signal.
That's it.
I'm starting to send out messages over my morning coffee.
Let's do this together:
https://support.signal.org/hc/en-us/articles/360007060592-In...
The only way to do this is to build an integration that makes the switch DEAD SIMPLE. If it involves someone being highly motivated to switch over themselves without any help its not going to gain steam.
yachtman commented on Moving from TypeScript to Rust / WebAssembly nicolodavis.com/blog/type... · Posted by u/nicolodavis
yachtman commented on Moving from TypeScript to Rust / WebAssembly nicolodavis.com/blog/type... · Posted by u/nicolodavis
Whenever I write Rust, I have a lot of fun, but I'm still not sold on it for most web dev.
The analogy that comes to mind is that Rust is a really nice sports car with a great engine. It handles like a dream and you can feel the powerful engine humming while driving it. With the right open road, it's a great time. You can really optimize code, make great abstractions and work with data easily.
Unfortunately, web dev generally isn't a wide open road. It's a lot of little turns and alleys and special cases. Driving Rust in web dev is a lot of accelerating only to screech to a halt. I wrote a backend API for a side project in Rust. And true to what I said, Rust handled like a dream. But I didn't need the power. I spent most of my time screeching to a halt worrying about the plumbing between the various immature Rust libraries. And that's on the backend, which is way more mature compared to the frontend Rust libraries.
Judging by this post, OP managed to find a great open road in web dev to use Rust. I only wish I could find one as worthwhile.
You might enjoy this blog post I wrote: http://kyleprifogle.com/churn-based-programming/
There is a common belief that SparkSQL is better than Hive because SparkSQL uses in-memory computing while Hive is disk-based. Another common belief is that Presto is better than Hive because it is based on MPP design and was invented for the very purpose of overcoming the slow speed of Hive by the very company (Facebook) that invented Hive in early 2010s.
The reality is that nowadays both SparkSQL and Presto are way behind Hive, in terms of both speed and maturity. Hive made tremendous progress since 2015 (with the introduction of LLAP), while SparkSQL still has the issue of stability of fault tolerance and shuffling. (Presto does not support fault tolerance.) So, IMO, SparkSQL is nowhere near ready to replace Hive.
If you are curious about the performance of these systems, see [1] and [2] which compare Hive, SparkSQL, and Presto. Disclaimer: We are developing MR3 mentioned in the articles. However, we tried to make a fair comparison in the performance evalaution.
[1] https://mr3.postech.ac.kr/blog/2019/11/07/sparksql2.3.2-0.10... [2] https://mr3.postech.ac.kr/blog/2019/08/22/comparison-presto3...
Was SparkSQL ever intended to replace hive? My impression was that it was supposed to supplement spark for times it was convenient. I kind of suspected at one point they got caught up in the SQL hadoop race, but I always felt like it was best to do SQL elsewhere, and save spark for things that couldn't be easily expressed in SQL.
A company I worked at embraced nix. Beforehand I was a big docker fan and had few issues with it (yes there were occasionally caching issues and damn it docker figure out the issue with hyperkit on mac os, but largely its a productive tool). From an outside perspective nix just felt like alot more work, and nearly no one (except the people that set it up) could ever get it to work. So basically the entire build process for the core of the system was something that literally no one in the company wanted to touch. I'm sure if I was an insider on the tool I would appreciate the added stability you get on the backend, but at face value it seemed like it just made builds unapproachable on the front end. I'm happily back to using docker and haven't looked back.