Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by takeiteasyy on Wednesday May 31, 2023 @08:37PM
(#36146344)
This will require a cpp port first to run on e.g. apple silicon?
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by rovr138 on Wednesday May 31, 2023 @09:23PM
(#36146617)
This is using huggingface. It's python. There's no reason it wouldn't run on arm.
https://huggingface.co/tiiuae/falcon-40b
I'll also add that the fact something is in C++ doesn't mean it will run on arm or that it can be compiled in it.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by wsgeorge on Wednesday May 31, 2023 @09:57PM
(#36146779)
No, but to benefit from the ggml ecosystem, yes. Someone's taking a stab at it: https://github.com/nikisalli/falcon.cpp
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by bongobingo1 on Wednesday May 31, 2023 @08:59PM
(#36146467)
The hardware requirements on these models is basically at a fixed floor, and the democratisation will come from cheaper, possibly specialised, hardware, not reduced requirements, right?
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by BonoboIO on Wednesday May 31, 2023 @10:01PM
(#36146797)
Will be interesting to see if someone comes up with an ASIC or FPGA.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by groby_b on Thursday June 01, 2023 @12:21PM
(#36153422)
Why would you think there's an ASIC/FPGA design that significantly improves over GPUs specifically targeted at running large models already? Where's the win?
The fundamental limit for hardware acceleration are number of gates you can squeeze on a die, right now. (Or, alternatively. memory bandwidth)
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by code_witch_sam on Friday June 02, 2023 @07:07PM
(#36171977)
analog chips like what MythicAI is developing seem like the next obvious leap forward for deploying inferences broadly. ASIC/FPGA wouldn't be much different than a GPU. ASIC seems like a brittle solution
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by wrs on Wednesday May 31, 2023 @10:04PM
(#36146815)
The minimum size of a model with equivalent performance is unknown (like so much else about LLMs), so requirements could be reduced.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by airgapstopgap on Wednesday May 31, 2023 @11:38PM
(#36147338)
Multi-Query Attention, used here, should make 40B inference viable on systems where even 33B LLaMA with Multi-Head is basically unusable, so sometimes improvements still come from software optimization (it's no free lunch though).
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by Tepix on Friday June 02, 2023 @07:33AM
(#36163776)
I think there is still lots room for improvement to reduce hardware requirements, such as 3-bit quantization or pruning weights from sparse models.
But if you're willing to spend $1500 on two used RTX 3090, it's the sweet spot in terms of the ability to run large models right now. Everything beyond that is much more expensive.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by dzhiurgis on Wednesday May 31, 2023 @10:34PM
(#36146974)
is there a model version where it can be deployed using huggingface's tool?
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by sireat on Thursday June 01, 2023 @02:41AM
(#36148196)
How is it possible to run this model on 2x 4090s ?
I thought that 4090s were "nerfed" and nvlink support removed - https://www.windowscentral.com/hardware/computers-desktops/n...
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by Timshel on Thursday June 01, 2023 @04:23AM
(#36148709)
Probably this : https://huggingface.co/docs/accelerate/usage_guides/big_mode...
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by MacsHeadroom on Friday June 02, 2023 @06:46AM
(#36163480)
Without NVLink different layers are loaded onto individual cards and only one card often has to wait for the other card, slowing down generation. It's still faster than CPU offloading. You can even mix and match GPUs like a 4090 + 3060.
Re:Falcon 40B LLM (which beats Llama) now Apache 2.0
by brianjking on Thursday June 01, 2023 @10:33AM
(#36151572)
How can it be deployed on a Huggingface Space or Colab notebook?
Inference is very slow right now but it works!