It's unfortunate that no math is ever done in these stories.
If you take the "350,000" H100s that Facebook wants by EOY, each of those can do 700W, which gives you almost 250 MW for just the GPUs. That sounds like a lot, until you realize that a single large power plant is measured in Gigawatts. All of Google's data centers combined are O(10 GW) which are matched with renewable power offsets [1].
Importantly, the world installed >500 Gigawatts of renewable energy in 2023 [2], mostly driven by PV Solar in China. The amount of potential solar and wind and other renewable-ish (hydro) outstrips even a 10x'ing of a lot of these numbers. But even for a single site, dams like Three Gorges are >20 GW.
There are real efficiency and scale challenges in doing AI in a single, large site. But existing power generation systems deliver plenty of power.
The big point of course is that there is massive asymmetry between training and inference and that even inference at scale is going to require massive amounts of energy and that likely OpenAI's business model isn't viable at scale. It works right now because they have capital to burn but when the music stops it may well turn out that their model isn't sustainable at all.
Efficiency gains should come first, long before they start looking at alternative energy sources.
Facebook is not alone, and there is growth. Also cooling is to be taken account of.
And third: renewables need to be associated with its backup like hydro/step or ... batteries which cost a lot. Gas can't be taken in as it's not CO2-free. All that unless training and inference happen when there's the corresponding wind and sun shining. And I'm not seeing that happening right now.
Energy production capacity in the US is relatively flat for the last couple of decades. The renewable installation is offsetting the decommission of coal. The capacity installed in China is not really accessible to Open AI due to recent security competition (both export restrictions on AI and a desire to import less energy). The capital costs of power are also quite high so I think he is pretty accurate considering the expectations of a startup to hockey stick.
> This report also analyzes prospective generation capacity in four categories — under construction, permitted, application pending, and proposed. More than 466,000 MW of new generation capacity is under development in the United States — a 13% increase over 2022. Sixty-one percent of capacity most likely to come online, permitted plants and plants that are under construction, are in solar.
China's growth in power capacity is non-trivially due to increasing demand. If the US or Europe or wherever suddenly wanted to build XXX GW per year, they could (modulo bureaucracy, which is very real).
This is my concern with AI in general. Cost, both real and monetary. Right now Microsoft and VCs are dumping money into AI operation to help with growth and adoption. What happens when AI's business focus moves from cost to grow to cost to serve? Will all these business who integrated in AI suddenly be saddled with huge bills? What if your product depends on AI, and suddenly is not profitable to operate? Anecdotally I have already seen people pull back AI features, because it turned out to be too expensive to serve in the long run.
I already pay for a GPT subscription, and its reliability is one of the worst of any product I pay for. The novelty keeps me paying, but I can't imagine building a business on it.
Ah we've now reached the "We're ready to go except for the magical development in another domain that's already resisted decades of concerted research and is absolutely necessary for what we do to actually make sense" stage. Good to know.
Record time compared to the crypto space, "all we need is infinite data storage with instant recall. Then we can store everything on the chain, and verify its authenticity!"
The size some of the large language models have is just insane. To put it into context GPT3 is a Large Language Model (LLM) but GPT4 has like (handwaveing) 5+ times as many parameters. This means the energy cost is also at least that much larger for inference.
And if we look at training instead of inference it's a quite a bit more complicated (each iteration is more costly, more parameter can also require more iterations, but at the same time you might need to spend less time in an area of increasingly diminishing returns per iteration), but if we go with GPT3 vs. GPT4 the 5x+ increase of parameters lead to a 10x+ increase in training cost (of which a huge part is energy cost, through also amortized hardware cost; ~10M$ to >100M$).
Additionally there are various analysis steps you might do when creating new models which can be hugely (energy) costly.
And that is with GPT4 and with OpenAI any major increase in version seem to come with a major increase in parameter size so with that trend we are looking at energy bills in the (potential many) hundreds of million US dollar for training alone.
Another thing wrt. inference cost is that with my limited understanding currently the best way to reach AGI and also a lot of other tasks is to run multiple models in tandem. Through this model might be domain adoptions of the same model, so not twice the training cost.
---
Domain Adaption == take a trained model and then train it a bit more to "specialize" it on a specific task, potentially adding additional knowledge etc. While you can try to do so just with prompt engineering there a larger prompt comes at a larger cost and a higher chance for unexpected failure, so by in a certain way "burning in" some additional behavior, knowledge etc. you can get nice benefit.
He's probably trying to build out a vertically integrated AI conglomerate spanning power, chip fabs, software, etc. And he's great at marketing in the meantime
I think both the OpenAI and the Sam Altman brand sustained some damage. He may be great at marketing but now that the gun is on the table and the cloth is withdrawn we'll see how much success he has in the future with trying to wrap people around his fingers.
Well, if the AI is good enough then ask it to design one.
Joking aside: the options are to turn it off or to charge what it should cost and given that no matter what it costs and no matter what the source if it uses too much energy the better solution would be to make it use less energy rather than to look for a breakthrough energy source because that also reduces the climate impact.
Well actually: "In a twist, Microsoft is experimenting with generative artificial intelligence to see if AI could help streamline the [nuclear power] approval process, according to Microsoft executives." [0]
One should be suspicious of ulterior motives when the CEO of an AI company makes a claim like this.
On one hand, LLMs do require significant amounts of compute to train. But the other hand, if you amortize training costs across all user sessions, is it really that big a deal? And that’s not even factoring in Moore’s law and incremental improvements to model training efficiency.
yes, at least as long as you constantly develop new AI models
and you still need to run the models, and e.g. for GPT4 that is alone already non trivial (energy cost/compute wise)
through for small LLMs if they are not run too much it might be not that bad
---
Generally I would always look for ulterior motives for any "relevant" public statement Sam Altman makes. As history has shown there often seems to be some (through in that usage "ulterior" has a bit too much of a "bad"/"evil" undertone).
To cut it short he seem to be invested in some Nuclear Fusion company, which is one of the potential ways to "solve" that problem. Another potential way is to use smaller LLMs but smaller LLMs can also be potentially a way how OpenAI loses their dominant position, as there is a much smaller barrier for training them.
Data centers cost 2% of electricity (Statistics from a few years ago).
AI inference is so costly at scale, one can easily see data centers start using 4% of total electricity, and in the next decade 8%. That will start to have severe effects on the power grid, basically require planning many years in advance to setup new power plants and such.
"Moore's law and incremental improvements' are irrelevant in the face of scaling laws. Since we aren't at AGI yet, every extra bit of compute will be dumped back into scaling the models and improving performance.
Is this the new PR playbook? Claim to be building something big and dangerous and ready to ruin everything if everyone doesn't drop everything and pay attention to me?
The AI winter is coming, and it's going to be bad. So many promises have been made, to the extent that the average person thinks that AGI is just around the corner.
Readit News
If you take the "350,000" H100s that Facebook wants by EOY, each of those can do 700W, which gives you almost 250 MW for just the GPUs. That sounds like a lot, until you realize that a single large power plant is measured in Gigawatts. All of Google's data centers combined are O(10 GW) which are matched with renewable power offsets [1].
Importantly, the world installed >500 Gigawatts of renewable energy in 2023 [2], mostly driven by PV Solar in China. The amount of potential solar and wind and other renewable-ish (hydro) outstrips even a 10x'ing of a lot of these numbers. But even for a single site, dams like Three Gorges are >20 GW.
There are real efficiency and scale challenges in doing AI in a single, large site. But existing power generation systems deliver plenty of power.
[1] https://www.gstatic.com/gumdrop/sustainability/google-2023-e...
[2] https://www.iea.org/reports/renewables-2023/executive-summar...
Efficiency gains should come first, long before they start looking at alternative energy sources.
Deleted Comment
And third: renewables need to be associated with its backup like hydro/step or ... batteries which cost a lot. Gas can't be taken in as it's not CO2-free. All that unless training and inference happen when there's the corresponding wind and sun shining. And I'm not seeing that happening right now.
> If you take the "350,000" H100s that Facebook wants by EOY, each of those can do 700W
Plus ~500W for cooling
> This report also analyzes prospective generation capacity in four categories — under construction, permitted, application pending, and proposed. More than 466,000 MW of new generation capacity is under development in the United States — a 13% increase over 2022. Sixty-one percent of capacity most likely to come online, permitted plants and plants that are under construction, are in solar.
China's growth in power capacity is non-trivially due to increasing demand. If the US or Europe or wherever suddenly wanted to build XXX GW per year, they could (modulo bureaucracy, which is very real).
I already pay for a GPT subscription, and its reliability is one of the worst of any product I pay for. The novelty keeps me paying, but I can't imagine building a business on it.
Employers will save health, logistics, HR, etc.
Governments will have to pay for unemployment
Just the same as always - privatize the gains
The size some of the large language models have is just insane. To put it into context GPT3 is a Large Language Model (LLM) but GPT4 has like (handwaveing) 5+ times as many parameters. This means the energy cost is also at least that much larger for inference.
And if we look at training instead of inference it's a quite a bit more complicated (each iteration is more costly, more parameter can also require more iterations, but at the same time you might need to spend less time in an area of increasingly diminishing returns per iteration), but if we go with GPT3 vs. GPT4 the 5x+ increase of parameters lead to a 10x+ increase in training cost (of which a huge part is energy cost, through also amortized hardware cost; ~10M$ to >100M$).
Additionally there are various analysis steps you might do when creating new models which can be hugely (energy) costly.
And that is with GPT4 and with OpenAI any major increase in version seem to come with a major increase in parameter size so with that trend we are looking at energy bills in the (potential many) hundreds of million US dollar for training alone.
Another thing wrt. inference cost is that with my limited understanding currently the best way to reach AGI and also a lot of other tasks is to run multiple models in tandem. Through this model might be domain adoptions of the same model, so not twice the training cost.
---
Domain Adaption == take a trained model and then train it a bit more to "specialize" it on a specific task, potentially adding additional knowledge etc. While you can try to do so just with prompt engineering there a larger prompt comes at a larger cost and a higher chance for unexpected failure, so by in a certain way "burning in" some additional behavior, knowledge etc. you can get nice benefit.
Joking aside: the options are to turn it off or to charge what it should cost and given that no matter what it costs and no matter what the source if it uses too much energy the better solution would be to make it use less energy rather than to look for a breakthrough energy source because that also reduces the climate impact.
[0] https://www.wsj.com/tech/ai/microsoft-targets-nuclear-to-pow...
On one hand, LLMs do require significant amounts of compute to train. But the other hand, if you amortize training costs across all user sessions, is it really that big a deal? And that’s not even factoring in Moore’s law and incremental improvements to model training efficiency.
yes, at least as long as you constantly develop new AI models
and you still need to run the models, and e.g. for GPT4 that is alone already non trivial (energy cost/compute wise)
through for small LLMs if they are not run too much it might be not that bad
---
Generally I would always look for ulterior motives for any "relevant" public statement Sam Altman makes. As history has shown there often seems to be some (through in that usage "ulterior" has a bit too much of a "bad"/"evil" undertone).
To cut it short he seem to be invested in some Nuclear Fusion company, which is one of the potential ways to "solve" that problem. Another potential way is to use smaller LLMs but smaller LLMs can also be potentially a way how OpenAI loses their dominant position, as there is a much smaller barrier for training them.
AI inference is so costly at scale, one can easily see data centers start using 4% of total electricity, and in the next decade 8%. That will start to have severe effects on the power grid, basically require planning many years in advance to setup new power plants and such.
"Moore's law and incremental improvements' are irrelevant in the face of scaling laws. Since we aren't at AGI yet, every extra bit of compute will be dumped back into scaling the models and improving performance.