We recently developed Sienna (http://sienna.gistnoesis.net/) as an experiment for easter (which went rather unnoticed by HN) to distribute computation. It allows to ML researchers to build cluster over peerjs rapidly by asking their friends to visit a webpage, so they can share their unused resources. We are not yet to 220,000 cores but we can keep dreaming :)
Cool! If you wanted you could accept donations and run on Preemptible to supplement spare desktops ;). I do love the SETI@Home vibe though. Especially since nothing beats free.
Can someone explain Google Preamble Instances vs Amazon Spot Instances?
The author says Sutherland was dissuaded by having to specify a spot price upfront on AWS but I don't see how that is any different than what Google is doing.
Can't you just bid for 20% of the original price and get the same behavior?
In AWS you simply tell how much you're willing to pay to keep the instance uninterrupted. If there's someone who is willing to pay more, they will get the instance from you and yours will be shut down.
We run specific components of our application in spot instances across different regions and from what I have seen in the last 3 months is that the price is generally much less than 80%. I'd say is less than 20% most of the time, but some times there is a peak and it goes beyond 150% for 1 to 3 hours, the frequency of these events may vary but I'd say once a week.
I think AWS might be cheaper in this regard but less predictable, so it is a tradeoff.
Preemptibles are at a fixed discount to the normal rate. Spot instances fluctuate according to demand.
That said, there is still a "market" here: your work vs Google's work. It just shows up as a private floating probability instead of a public floating price.
Most of the time you get the full 24 maximum window.
Just a small correction: it's not about "your work vs Google's", it's about regular VMs versus preemptible. There's no special code to give Googlers (or Alphabet Characters) special treatment with regards to preemption. The chrome clusterfuzz folks are standing side by side with you.
Disclosure: I work on Google Cloud (and launched Preemptible VMs).
"The LMFDB makes visible the connections predicted by the Langlands program. "
The Langlands program is mathematicians version of "string theory". And what this guy is doing with google cloud is comparable to the LHC: searching for new particles (of mathematics) in order to find the deeper unity therein.
I think it has to do with public key cryptography - in particular, elliptic curve cryptography[0], which was mentioned by Schneider to be one of the key topics on NSA's agenda[1].
There is always the existential "Does anything really matter, everybody dies in the end?". This could have direct application, or as with many math concepts, in 50-100 years we could find an application for it. Despite all of that it doesn't have to matter, the process of exploration can have value in and of itself.
Possibly a dumb question, but if it's this embarrassingly parallel, wouldn't this be a workload more suited for calculation on a GPU? I'm assuming there's a good reason he's not using one, so could someone who understands this a little better explain it?
> if it's this embarrassingly parallel, wouldn't this be a workload more suited for calculation on a GPU?
Maybe, but one does not necessarily follow from the other. Consider the task of compiling 1 million separate C++ projects. That is obviously embarrassingly parallel, but not well suited for a GPU. It's trivial to do many compilations at once, but compiling itself is not easy to parallelize.
That example is obviously contrived, but I think it demonstrates the principle that it's the computational profile of the core problem that will determine if you can use a GPU. If the core problem requires 10s of GB of RAM, or it's excessively branchy code, it may not be well suited for a GPU.
A great paper which delves into different approaches for parallel computing is "Three layer cake for shared-memory programming" [0]. They characterize parallel programming into three broad strategies:
1. SIMD (parallel lines)
2. Fork-Join (a directed acyclic graph of operations)
3. Message-Passing (a graph of operations)
GPUs are great at SIMD, but bad at the other sorts of parallelism.
That's not a dumb question. As alluded to below, for most people the main challenge is their software stack. Beyond that, 200k+ vcpus is still a ton so you'd need hundreds to thousands of GPUs to just match the flops (Note: I don't know if Drew's code is vectorised or not on CPUs).
Put together that by using Preemptible VMs (and yes, apologies we still don't offer GPUs as preemptible) it's economically rational to use spare CPUs.
Wow I'd be interested to see the code for this. Is it just a tiny amount of C running on a huge number of machines, or is it a big stack of mathematical libraries?
I guess since it's discrete math it probably doesn't use Fortran/BLAS?
The fuel bill per hour for the 737 going over my house right now is easily double $2200. I can visualise what is going on with a plane burning through fuel/money but with 220000 CPU cores it is all imaginary.
How big a house would be needed for such an array of 'cores'? How much electricity is required, and again can that be compared to aeroplanes, Teslas or even toasters?
Readit News
The author says Sutherland was dissuaded by having to specify a spot price upfront on AWS but I don't see how that is any different than what Google is doing.
Google Spot Instances (preemptibles) are 80% off and that's it. It's simple.
In AWS you simply tell how much you're willing to pay to keep the instance uninterrupted. If there's someone who is willing to pay more, they will get the instance from you and yours will be shut down.
I think AWS might be cheaper in this regard but less predictable, so it is a tradeoff.
Edit: I haven't looked yet into "Spot Fleets".
That said, there is still a "market" here: your work vs Google's work. It just shows up as a private floating probability instead of a public floating price.
Most of the time you get the full 24 maximum window.
edit: wording
Disclosure: I work on Google Cloud (and launched Preemptible VMs).
https://www.quora.com/What-are-the-key-differences-between-A...
Deleted Comment
http://www.lmfdb.org/Genus2Curve/Q/11664/a/11664/1
The Langlands program is mathematicians version of "string theory". And what this guy is doing with google cloud is comparable to the LHC: searching for new particles (of mathematics) in order to find the deeper unity therein.
[0] https://arstechnica.com/security/2013/10/a-relatively-easy-t... [1] https://www.schneier.com/blog/archives/2015/10/why_is_the_ns...
Maybe, but one does not necessarily follow from the other. Consider the task of compiling 1 million separate C++ projects. That is obviously embarrassingly parallel, but not well suited for a GPU. It's trivial to do many compilations at once, but compiling itself is not easy to parallelize.
That example is obviously contrived, but I think it demonstrates the principle that it's the computational profile of the core problem that will determine if you can use a GPU. If the core problem requires 10s of GB of RAM, or it's excessively branchy code, it may not be well suited for a GPU.
1. SIMD (parallel lines)
2. Fork-Join (a directed acyclic graph of operations)
3. Message-Passing (a graph of operations)
GPUs are great at SIMD, but bad at the other sorts of parallelism.
[0] https://www.researchgate.net/publication/228683178_Three_lay...
Put together that by using Preemptible VMs (and yes, apologies we still don't offer GPUs as preemptible) it's economically rational to use spare CPUs.
Disclosure: I work on Google Cloud.
Deleted Comment
Dead Comment
I guess since it's discrete math it probably doesn't use Fortran/BLAS?
Disclosure: I work on Google Cloud, launched Preemptible (and approved Drew's quota requests!)
How big a house would be needed for such an array of 'cores'? How much electricity is required, and again can that be compared to aeroplanes, Teslas or even toasters?