Readit NewsThe point of mathematics is not just to know whether some proposition is true, it's to understand the mathematical objects under study. A beautiful proof sheds light, it explains why.
Otherwise, Spark is relatively new, so they might have some older infra/jobs in Hadoop.
Storm and Spark streaming work a little differently (real-time streaming vs "micro-batching) and apparently have different use cases, but I'm not totally sure what the practical difference are here either..
https://www.amazon.com/Concepts-Techniques-Models-Computer-P...
Language & library recs:
Java is actually a pretty shitty language to learn concurrency on, because the concurrency primitives built into the language & stdlib are stuck in the 1970s. There've been some more recent attempts to bolt more modern concurrency patterns on as libraries (java.concurrent is one; Akka is another; Quasar is a third), but you're still very limited by the language definition. Some other languages to study:
Erlang, for message-passing & distributed system design patterns. Go has a similar concurrency model, but not as pure.
Haskell for STM.
Python3.5/ES2017/C#, for async/await & promises. Actually, for a more pure implementation of promises, check out E or the Cap'n Proto RPC framework.
Rust, for mutable borrowing. Rust's concurrency story is fairly unique; they try to prove that data races can't exist by ensuring that only one reference is mutable at once.
JoCaml for the join calculus. Indeed, learning formal models like CSP, the pi-calculus, or the join-calculus can really help improve your intuitions about concurrency.
Hadoop for MapReduce-style concurrency. In particular, learning how you might represent, say, graph algorithms on a MapReduce system is a great teacher. Also look at real-time generalizations of MapReduce paradigms like Storm or Spark.
Paxos & Raft for the thorny problems in distributed consensus.
Vector clocks, operational transforms and CRDTs. One approach to concurrency is to make it not matter by designing your algorithms so that each stage can be applied in arbitrary order (or can compensate for other operations that have occurred in the meantime). That's the idea behind this, and it perhaps has the most long-term promise.
Project & job recs:
The best way to really learn concurrency is to take a job at a company that has to operate at significant scale. Google or Facebook are the most prominent, but any of the recent fast-growers (AirBnB, Uber, Dropbox, probably even Instacart or Zenefits) will have a lot of problems of this type.
Failing that, I've found that implementing a crawler is one giant rabbithole in learning new concurrency techniques. The interesting thing about crawlers is that you can implement a really simple, sequential one in about 15 minutes using a language's standard library, but then each step brings a new problem that you need to solve with a new concurrency technique. For example:
You don't want to wait on the network I/O, so you create multiple threads to crawl multiple sites at once.
You quickly end up exhausting your memory, because the number of URLs found on pages grows exponentially, and so you transition to a bounded thread pool.
You add support for robots.txt and sitemaps. Now you have immutable data that must be shared across threads.
You discover some URLs are duplicates; now you need shared mutable state between your fetch threads.
You start getting 429 and 403 request codes from the sites, telling you to back off and stop crawling them so quickly. Now you need a feedback mechanism from the crawl threads to the crawl scheduler, probably best implemented by message queues.
You want to process the results of the crawl. Now you need to associate the results of multiple fetches together to run analyses on it; this is what MapReduce is for.
You need to write out the results to disk. This is another source of I/O, but with different latency & locking characteristics. You either need another thread pool, or you want to start looking into promises.
You want to run this continuously and update a data store. Now you need to think about transactions.
It is driven by the same competition, but with article-level tracking rather than newspaper sales, the problem got worse. I don't know how to fix it either, but I agree with the author that its a huge problem. News and entertainment are basically the same thing now.
. Make a list of companies that I would apply to and sort them from most interesting to no-way-in-hell-i-am-working-here order
. spend a weak reviewing typical algo/data structure questions
. For the companies that I absolutely want to work for, I review every single glassdoor review and write down the interview questions. Remember, most companies have question banks and most interviewers have favorite questions which results in same questions being asked over an over again. You want to exploit that
. Then to get over my interviewing jitters, I interview at a few companies where I would absolutely not work at. This results in no pressure interview practise and you can literally laugh at their asinine interview questions and walk out
. Finally, for the companies i actually want to work at, I try my best to get rid of phone screen. This is usually accomplished by dazzling them with my decent size github profile, contributing some fixes to their OSS project or finding someone who already works there that is in my alumni network
. Then when you finally arrive for the interview, you have real world interview practise, they are already impressed with your github profile/references and biased toward you versus some random joe off the street and you have made sure you have a pretty high probability of getting a question that you have already seen or is similar to a question you already know.
This technique has helped me get Jobs at top 5 employers in the valley along with a few startups. The reason I am posting this here is to demonstrate how broken, unfair and easy to game this whole process is
The law says racism is illegal in certain situations, and society says racism is undesirable in most situations.
The law and society aren't claiming that racism is statistically non-optimal -- in fact, there are lots of things more optimal than status quo that many people would find totally horrifying.
If are widely replacing human systems with AI systems, I think this is a legitimate concern.
I really depart from the article in two areas:
1. The AI will inherit the biases of its creators. This is possible but far from guaranteed. And relatedly, inclusivity of the development team guarantees nothing regarding the goals of the system.
2. Criticising the people who are warning of the problem and trying to do something about it. This is related to the AI control problem. There is no switch that can be flipped that will prevent AI systems from Bad Ideas. It's not that we just aren't flipping it to preserve our chokehold on captialism. Implementing morality in AI systems is a genuinely monumental problem. And the people who are doing something about it are behaving very altruistically.
And with AI, we must make these values explicit, which is very difficult to do - i agree this is a very important problem to solve, and the people doing it should absolutely be rewarded.
Honestly, reading the article again after your summary i found it very reasonable :p I think the headline just ticked me off.
No one is stopping other people from getting in on the debate, they absolutely should (and I'm sure there are roadblocks in their way, and people who really are racist). It just feels wrong to implicitly all the "bad white people" for that. Blame those who cause the problem. Otherwise we are back to stereotyping.
I think you meant black people are more likely to be convicted of crime. The problems with crime 'statistics' is that on the surface, it all seems coldly scientific, yet they are generated and derived via very biased, very human, very unscientific processes - there is a lot of bad data. The ACLU did research that showed that there is no statistically significant difference in the possession of weed between white and black people, yet more black people are convicted[1] for possession.
Here's a mind experiment: after watching this YouTube video[2], how skewed do you think the statistics for white female criminals (bike thieves) vs black criminals would be?
1. https://www.aclu.org/files/assets/aclu-thewaronmarijuana-rel...
I just think we need to be able to TALK about these issues, so that when a real expert looks at those statistics they can get to the truth of the matter, and say that truth whether it is or isn't politically correct.
Also, trust me i can talk about this because i'm not white...
fact - people prefer people like them. Not even consciously, this is basic shit hardwired into us. I don't blame white men for being subconsciously biased to hiring white men, literally any other group would do the same. sure we can try to fight that bias, but its not at all evil or wrong to have that bias, only natural.
fact - taking an "agnostic" approach the way science does, of course the algorithms will reflect "biases". if men are statistically more likely to be programmers, or black people are more likely to commit crimes (STATISTICALLY), then the algorithm will pick that up. They are biases sure, but also statistical realities.
Now we can debate whether we should actively engineer algorithms to fight these "biases" on a case-by-case basis (for example, focusing more on women might be a win if you can find talent no one else can), but there's no reason to start pointing fingers at the "evil white guys" on top who planned this from the very beginning... it's just more stereotyping.
hypothesis - she wrote this crap to gain publicity.
In that case, the Ethereum hard fork doesn't sound very controversial - even if the devs decided to hard fork for bad reasons later on, no one would be forced to adopt it. They would be "voting with their feet" by sticking to the old chain.