> Although the GPU performs inference faster than the FPGA,
one of the primary bottlenecks of deep learning inference is
memory bandwidth and the availability of on-chip memory
(Balasubramanian et al., 2021). A RTX 3090 has 24GB
VRAM running at 1219 MHz with a base core clock of 1395
MHz (TechPowerUp, 2024). In comparison, a VU9P FPGA
has 345.9 MB of combined on-chip BRAM and URAM,
running at a much slower clock speed of around 200-300
MHz depending on the module; however, with much lower
clock speeds, the FPGA is able to achieve better efficiency
on power and energy consumption, as shown below.
So as far as I can understand, the biggest "bottleneck"/limiting factor with using FPGAs for LLMs is the available memory -- with current large models exceeding 40 GiB in parameter size, GPUs and TPUs with DRAM look like the only way to go forward for the months to come ... Thoughts?
Yeah, I think DRAM is almost certainly the future, just in terms of being able to afford the memory capacity to fit large models. Even Cerebras using a full wafer only gets up to 44 GB of SRAM on a chip (at a cost over $2M).
An interesting twist is that this DRAM might not need to be a central pool where bandwidth must be shared globally -- e.g. the Tensortorrent strategy seems to be aiming for using smaller chips that each have their own memory. Splitting up memory should yield very high aggregate bandwidth even with slower DRAM, which is great as long as they can figure out the cross-chip data flow to avoid networking bottlenecks
That being said, the techniques discussed here are not totally irrelevant (yet). There still exists some hardware with fast instructions for float/int conversion, but lacking rsqrt, sqrt, pow, log instructions, which can all be approximated with this nice trick.