Listening to Jim Keller talk about engineering makes me feel the same way I used to feel when I listened to Jeff Beck playing guitar. I didn't know if I was inspired or if I should quit.
The first Graviton machines (which were worse than the current 64 core RISC-V SG2042) were installed in November 2018. Four and a half years later are now up to 20% or 25% of total AWS capacity. Even without any acceleration of adoption, it looks like they could be the majority of AWS within 10 years from first deployment.
There seems to be every reason RISC-V could do it faster. Graviton 1 level chips are available right now, Graviton 3 or better coming from multiple companies (including Keller's) in 2024-2025. The software is more critical, but once x86-only things have been made portable to Arm (which can be hard work), further porting them to RISC-V is much much easier.
None of those require user or OS instruction set compatibility for legacy apps that are hard or impossible to recompile.
And most of these applications don't really require gonzo superscalar performance. Add more cores, support more data streams.
If you can eliminate licensing costs for that portion of your fleet, then you only need to expand the ISA compatible portion of your fleet as demanded by paying customers.
As an example, suppose all of a cloud provider's services can migrate to RISC-V. As organic demand for x86 Cloud among customers grows, services can shift incrementally to the cheaper home grown platforms. And since the freed up machines are at least partially depreciated, the cost of these servers is much less than what a customer would pay for new servers on prem. (depreciated Cap-ex, far better Op-Ex).
The interesting question is the transition rate of end customer apps to the new ISA vs the growth rate of locked ISA apps.
Eventually the locked ISA apps portion becomes a lot like the current IBM mainframe business. Very valuable to a very small number of customers.
The only counter for this is if x86 can crank performance per $TCO so far that the non-x86 branch can't compete in business terms, which has historically been the issue with ARM.
… and fully managed cloud services (serverless databases, API gateways, messaging components, supplementary services etc etc). Which is where the hardware replacement is likely most frantic but impossible to glean into without having the insider knowledge. And since the fully managed services are charged on the usage basis, it is possible to utilise the hardware more efficiently under the hood and completely transparently to the end user (if properly architected) as opposed to spinning up cloud VM's.
> The only counter for this is if x86 can crank performance per $TCO so far that the non-x86 branch can't compete in business terms, which has historically been the issue with ARM.
If we take AWS for example, isn't the performance per TCO better of an Arm-based Graviton instance better than x86? I don't think the historical issue you cite represents the future.
This man knows exactly what he is talking about. He was responsible for designing the original AMD Athlon 64. He worked at apple to transition from Generic Samsung ARM SoC to their own Apple silicon which is the base for modern M1 Apple silicon. He worked for Intel (we'll see his work in Lunar Lake, Jim Keller's Royal Core Project). And most importantly he worked again at AMD and gave us Zen architecture.
My optimistic take: “All the data centers” are the Cloud hyperscalers, who are increasingly delivering value through PaaS/SaaS vs. raw VMs and IaaS.
They’re choosing the CPUs they like best, can turn over quickly if it’s worthwhile, and if the performance/economics of RISC-V are suitably appealing will do so.
I wonder how much of, say, S3’s infrastructure is running on Graviton?
I hope RISC-V servers will come with open BIOS, given Ron Minnich stance on the proprietary BIOS issue during his time at google I think hyperscalers would like that too.
Recently had basic BIOS/BMC bugs it's annoying as hell.
Data centers are one of the best demographics for adopting new architectures because more of the software can be custom-built towards a narrow application: Get a Linux stack to build, add some network I/O, add some virtualization, and you can do all sorts of things.
Client apps have a much harder time making that jump because the environment is more holistic, the hardware more varied, and the need for "must-have" proprietary apps more imperative.
> Data centers are one of the best demographics […]
Hardly. Data centers are a dying breed, and their number has been rapidly dwindling in recent years. DC (and the mythical «on-prem» by extension) has effectively become a dirty word in contemporary times. The most brutal lift-and-shift approach (without discussing the merits of doing so) is most common: create a backup, spin up a cloud «VM», restore from the backup and turn the server in the DC off forever. No-one is going to even remotely consider a new hardware architecture, not even in the cloud.
Moreover, since servers do not exist in a vacuum and either run business apps or do something at least remotely useful, that entails the software migration to adopt the new platform. And the adoption has to be force-pushed onto the app developers otherwise they won't bother, and for them to convert/migrate the app onto a new architecture, they need desktops/laptops that run on the new ISA, and no viable server and desktop hardware exists in June 2023 – it will come along later with «later» not having a clear definition. Talking open source is a moot point as most businesses out there run commercially procured business apps.
Considering modern processors spend 4-5 years in development before public release, someone would have to be building the game changing RISC-V CPU right now.
Maybe he meant that development on RISC-V CPUs would start in earnest in the next 5-10 years?
>Considering modern processors spend 4-5 years in development before public release, someone would have to be building the game changing RISC-V CPU right now.
And they are.
Tenstorrent is working on Ascalon. Wei-han Lien (lead architect of M1 at Apple) is the lead architect. Ascalon is a RISC-V microarchitecture expected to be released in 2024, with similar performance to projected AMD Zen5 (also 2024), but lower power consumption.
Ventana Veyron is due late 2023. A very high performance server chip AIUI implementing RVA22+V.
Rivos has been working on something RISC-V, with a very strong team, for several years now.
SiFive's next iteration of high performance CPUs is expected to be strong.
Alibaba group has something in the works, too.
And this is all just the tip of the iceberg. There's way more known projects ongoing, and even more that we do not know of.
The reason that processors take so long to develop, might have something to do with the complexity of the ISA, most of CPU development effort is spent on verification, if you have a simpler ISA, I would imagine that it makes verification easier.
My interpretation of 'take over' would be a majority of new server installs would be RISC-V based. There is a lead time for development, orders etc plus customers have to be content to switch to a new architecture. Amazon's Arm program started what 6ish years ago and they are at 20% installs (from my recollection).
Still awaiting the retail release of that 75w PCIe card with out-of-box PyTorch support. If it can run, say StableDiffusion / LAAMA and family, many more use cases shall emerge.
Obviously retail price will have to be competitive with Gaming Cards.
We're having great luck running Yolo 8 models on cheap intel n5105 boxes. 8gb ram, 256gb SSD, USB3 for ~$150... Ubuntu running ONNX gets us 600ms per frame. Not great, but good enough for now.
I watched the video, apparently they are targeting $1000-$2000, and said even $500 sounds interesting.
Maybe, just maybe a highly cut down version, 15 watt model could bring it down to RaspberryPi level price. $45. UsbC interface, if not x4 PCIe. Like, people buy it first, and decide what to do with it latter on. Guess I am going to far down the fantasy lane. ;)
Almost certainly not. Raspberry pi economics only work since they used a cheap of the shelf CPU. For the first few gens, I highly doubt anything below $200 will make any sense given the unit profit margins they will need.
I liked the part in which he pointed out that on the Linux kernel, he could get a fix within an hour, while on NT he had to wait a year. His general emphasis on open source is very encouraging. I didn't know he came around like that.
Yes, "source code for chips / custom hardware" as a product is mostly referred as IPs. This includes HDL/RTL, netlists and GDS layout for a specific process, in increasing order of closeness to physical reality.
Tangentially related, it is interesting to note that layouts and masks for manufacturing custom chips are not really copyrightable, so there is a thing named "mask rights" and (M) instead of (C) for them.
Yeah they're talking about hardware designs. Chip designers can buy "IP" from vendors for stuff they don't want to implement themselves, e.g. PCIe interfaces, embedded CPUs, etc. It's delivered as SystemVerilog code, probably obfuscated and maybe encrypted (yeah the encryption is theoretically worthless but I guess it is a sort of honesty / deniability thing).
I have a list of some I'm keeping tabs on, I'll share it here. I do it out of a combination of personal obsession plus because I want to source angel investments.
Note though that I'm very biased toward AI companies...
Most established, clear product-market fit:
- OpenAI
- Midjourney
- Character ai
- Runway ML
Ones that are interesting:
- Adept AI
- Modal, Banana.dev
- new.computer
- Magic.dev
- Modular (Mojo)
- tiny corp
- Galileo
- Hippo ML
- Tenstorrent
- contextual.ai
- Chroma
- e2b.dev
- Steamship
- Patterns.app
- GGML
Ones that I want to learn more about before deciding:
- Inflection AI
- GetLindy
- Embra
- Jam.dev
- Vocode.dev
That's about 50% of my list. Happy to clean up the rest and write a post if there's interest
Sally Foxton-Ward (EETimes) and Ian Cuttress (TechTechPotato) started a podcast (only 2 episodes so far I think) listing through companies and what they do. I agree it is very hard to follow with companies coming and going.
Readit News
Isn't this essentially impossible?
There seems to be every reason RISC-V could do it faster. Graviton 1 level chips are available right now, Graviton 3 or better coming from multiple companies (including Keller's) in 2024-2025. The software is more critical, but once x86-only things have been made portable to Arm (which can be hard work), further porting them to RISC-V is much much easier.
Cluster management, file systems, disk / storage systems, network management, database systems.
None of those require user or OS instruction set compatibility for legacy apps that are hard or impossible to recompile.
And most of these applications don't really require gonzo superscalar performance. Add more cores, support more data streams.
If you can eliminate licensing costs for that portion of your fleet, then you only need to expand the ISA compatible portion of your fleet as demanded by paying customers.
As an example, suppose all of a cloud provider's services can migrate to RISC-V. As organic demand for x86 Cloud among customers grows, services can shift incrementally to the cheaper home grown platforms. And since the freed up machines are at least partially depreciated, the cost of these servers is much less than what a customer would pay for new servers on prem. (depreciated Cap-ex, far better Op-Ex).
The interesting question is the transition rate of end customer apps to the new ISA vs the growth rate of locked ISA apps.
Eventually the locked ISA apps portion becomes a lot like the current IBM mainframe business. Very valuable to a very small number of customers.
The only counter for this is if x86 can crank performance per $TCO so far that the non-x86 branch can't compete in business terms, which has historically been the issue with ARM.
… and fully managed cloud services (serverless databases, API gateways, messaging components, supplementary services etc etc). Which is where the hardware replacement is likely most frantic but impossible to glean into without having the insider knowledge. And since the fully managed services are charged on the usage basis, it is possible to utilise the hardware more efficiently under the hood and completely transparently to the end user (if properly architected) as opposed to spinning up cloud VM's.
If we take AWS for example, isn't the performance per TCO better of an Arm-based Graviton instance better than x86? I don't think the historical issue you cite represents the future.
But this is Jim Keller and I’m a random internet guy.
Here is another interview with Jim Keller where he explains why instruction sets doesn't matter that much:
https://www.anandtech.com/show/16762/an-anandtech-interview-...
They’re choosing the CPUs they like best, can turn over quickly if it’s worthwhile, and if the performance/economics of RISC-V are suitably appealing will do so.
I wonder how much of, say, S3’s infrastructure is running on Graviton?
Recently had basic BIOS/BMC bugs it's annoying as hell.
Client apps have a much harder time making that jump because the environment is more holistic, the hardware more varied, and the need for "must-have" proprietary apps more imperative.
Hardly. Data centers are a dying breed, and their number has been rapidly dwindling in recent years. DC (and the mythical «on-prem» by extension) has effectively become a dirty word in contemporary times. The most brutal lift-and-shift approach (without discussing the merits of doing so) is most common: create a backup, spin up a cloud «VM», restore from the backup and turn the server in the DC off forever. No-one is going to even remotely consider a new hardware architecture, not even in the cloud.
Moreover, since servers do not exist in a vacuum and either run business apps or do something at least remotely useful, that entails the software migration to adopt the new platform. And the adoption has to be force-pushed onto the app developers otherwise they won't bother, and for them to convert/migrate the app onto a new architecture, they need desktops/laptops that run on the new ISA, and no viable server and desktop hardware exists in June 2023 – it will come along later with «later» not having a clear definition. Talking open source is a moot point as most businesses out there run commercially procured business apps.
Maybe he meant that development on RISC-V CPUs would start in earnest in the next 5-10 years?
And they are.
Tenstorrent is working on Ascalon. Wei-han Lien (lead architect of M1 at Apple) is the lead architect. Ascalon is a RISC-V microarchitecture expected to be released in 2024, with similar performance to projected AMD Zen5 (also 2024), but lower power consumption.
Ventana Veyron is due late 2023. A very high performance server chip AIUI implementing RVA22+V.
Rivos has been working on something RISC-V, with a very strong team, for several years now.
SiFive's next iteration of high performance CPUs is expected to be strong.
Alibaba group has something in the works, too.
And this is all just the tip of the iceberg. There's way more known projects ongoing, and even more that we do not know of.
Deleted Comment
Important changes always take much longer than we expect, but have much greater impact than we imagine.
RISC-V is inevitable.
Obviously retail price will have to be competitive with Gaming Cards.
My 2 cents.
When I talked with TensTorrent people they claimed it was a $1000 A100 competitor with 16gb RAM. Still waiting on it.
Maybe, just maybe a highly cut down version, 15 watt model could bring it down to RaspberryPi level price. $45. UsbC interface, if not x4 PCIe. Like, people buy it first, and decide what to do with it latter on. Guess I am going to far down the fantasy lane. ;)
Tangentially related, it is interesting to note that layouts and masks for manufacturing custom chips are not really copyrightable, so there is a thing named "mask rights" and (M) instead of (C) for them.
https://youtu.be/32CRYenTcdw?t=1560
They do this at the chip level apparently? Pretty nifty!
Note though that I'm very biased toward AI companies...
Most established, clear product-market fit:
- OpenAI
- Midjourney
- Character ai
- Runway ML
Ones that are interesting:
- Adept AI
- Modal, Banana.dev
- new.computer
- Magic.dev
- Modular (Mojo)
- tiny corp
- Galileo
- Hippo ML
- Tenstorrent
- contextual.ai
- Chroma
- e2b.dev
- Steamship
- Patterns.app
- GGML
Ones that I want to learn more about before deciding:
- Inflection AI
- GetLindy
- Embra
- Jam.dev
- Vocode.dev
That's about 50% of my list. Happy to clean up the rest and write a post if there's interest
They're developing a near-memory inference accelerator that looks pretty interesting.
It's a good source for trends in hardware.
Not sure if linking is allowed: https://www.youtube.com/watch?v=pQzf_1AkNXw
Keep an eye on notable people! They are the one driving the trends, industry direction, and narratives.
You can also create Google Alerts. I'm sure there are other alternates as well. https://www.google.com/alerts
And, in general - read news!
Kind of a minor point, but it's weird they omitted Intel.