Serving 70B-Scale LLMs Efficiently on Low-Resource Edge Devices [pdf]

Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D... by freehorse on Thursday October 03, 2024 @11:26AM (#41731651)

From what I get skimming through the article the main cost is speed of token generation (token latency). You can always run a large model by reading directly from the disk and not care much about ram; but it is very slow. They try to improve that aspect doing some optimisations, but it is still definitely slower than using ram or vram.

[ Parent | on HN ]

Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D... by refulgentis on Thursday October 03, 2024 @11:55AM (#41731881)

Table 3 directly refutes this* and claims 0 tradeoffs.**

Below that, they indicate that a key part of the implementation is loading weights from disk before they're needed using a separate thread.***

* maybe I'm missing something though, someone please triple check :)

** ttft (time to first token) and s/token (seconds per token) are both lower than any alternative in all cases.

*** "daemon thread asynchronously preloads the weights"

[ Parent | on HN ]

Re:Serving 70B-Scale LLMs Efficiently on Low-Resource Edge D... by sgc on Thursday October 03, 2024 @12:26PM (#41732207)

I want to add that their chart shows s/token per device (edit: as per the heading on table 1 - it could also be confusing grammar), so it sounds like you are getting 4x the listed s/t on their 4 laptop cluster . Their laptops are not even hardwired - they are connecting over wifi.

This comes at a very interesting time for me. I have an ancient dual xeon workstation with 64gb memory that I was researching how to convert to run an llm. I can just run that with 4 instances on the same machine and see how it goes, without purchasing a better GPU, to start. It sounds like this will allow you to run very large models with minimal quants, on craigslist quality devices.

If it does what they say it does (and it seems to do), it will be an absolute game changer for most users.

[ Parent | on HN ]