LlamaF: An Efficient Llama2 Architecture Accelerator on Embedded FPGAs

Re:LlamaF: An Efficient Llama2 Architecture Accelerator on E... by jsheard on Friday September 27, 2024 @11:17AM (#41671526)

Is there any particular reason you'd want to use an FPGA for this? Unless your problem space is highly dynamic (e.g. prototyping) or you're making products in vanishing low quantities for a price insensitive market (e.g. military) an ASIC is always going to be better.

There doesn't seem to be much flux in the low level architectures used for inferencing at this point, so may as well commit to an ASIC, as is already happening with Apple, Qualcomm, etc building NPUs into their SOCs.

[ Parent | on HN ]

Re:LlamaF: An Efficient Llama2 Architecture Accelerator on E... by wongarsu on Friday September 27, 2024 @12:11PM (#41672325)

This specific project looks like a case of "we have this platform for automotive and industrial use, running Llama on the dual-core ARM CPU is slow but there's an FPGA right next to it". That's all the justification you really need for a university project.

Not sure how useful this is for anyone who isn't already locked into this specific architecture. But it might be a useful benchmark or jumping-off-point for more useful FPGA-based accelerators, like ones optimized for 1 bit or 1.58 bit LLMs

[ Parent | on HN ]

Re:LlamaF: An Efficient Llama2 Architecture Accelerator on E... by PaulHoule on Friday September 27, 2024 @11:34AM (#41671782)

(1) Academics could make an FPGA but not an ASIC, (2) FPGA is a first step to make an ASIC

[ Parent | on HN ]

Re:LlamaF: An Efficient Llama2 Architecture Accelerator on E... by someguydave on Friday September 27, 2024 @11:24AM (#41671612)

gotta prototype the thing somewhere. If it turns out that the LLM algos become pretty mature I suspect accelerators of all kinds will be baked into silicon, especially for inference.

[ Parent | on HN ]

Re:LlamaF: An Efficient Llama2 Architecture Accelerator on E... by jsheard on Friday September 27, 2024 @11:32AM (#41671742)

That's the thing though, we're already there. Every new consumer ARM and x86 ASIC is shipping with some kind of NPU, the time for tentatively testing the waters with FPGAs was a few years ago before this stuff came to market.

[ Parent | on HN ]

Re:LlamaF: An Efficient Llama2 Architecture Accelerator on E... by PaulHoule on Friday September 27, 2024 @11:35AM (#41671787)

But the NPU might be poorly designed for your model or workload or just poorly designed.

[ Parent | on HN ]

Re:LlamaF: An Efficient Llama2 Architecture Accelerator on E... by mistrial9 on Friday September 27, 2024 @12:14PM (#41672371)

like this? https://www.d-matrix.ai/product/

[ Parent | on HN ]

Re:LlamaF: An Efficient Llama2 Architecture Accelerator on E... by israrkhan on Friday September 27, 2024 @12:31PM (#41672573)

You can open-source your FPGA designs for wider collaboration with the community? wider collaboration. Also, FPGA is the starting step to make any modern digital chip.

[ Parent | on HN ]

Re:LlamaF: An Efficient Llama2 Architecture Accelerator on E... by danielmarkbruce on Friday September 27, 2024 @02:12PM (#41673854)

Model architecture changes fast. Maybe it will slow down.

[ Parent | on HN ]