From the HuggingFace page:

> You will need at least 85-100GB of memory to swiftly run inference with Falcon-40B.

So it may be possible with less (swap around method), though not as efficiently and also slower.

For Falcon 40b, the 8-bit version would probably need about 48GB of VRAM while the 4-bit would need something closer to 28GB.

Currently on main, 8bit and 4bit quant is supported

One can simply do

```openllm start falcon --model-id tiiuae/falcon-40b-instruct --quantize int4```

Beware that there is no free lunch, meaning the quality of inference will degrade by alot when using int 4 quantization

For falcon 40b you probably need an A100 40gb or so.

Every model is drastically different.

If you want to run something on consumer hardware, your best bet is using anything ported to the ggml framework, especially if you're on Apple silicon.

Do you know any "standard" way or measures to determine the approximate hardware requirements of a model?

You want the entire model to fit in memory. So if you’re looking at a download size of, say, 100GB then don’t run on less than that. Your machine will swap to/from disk constantly, which will be slow and wear out an SSD.

If you want to train a model, that’s a different story.

Model size in GB = VRAM required for uncompressed inference (16bit aka "half precision) plus ~1-4GB for context. For 8bit you need half that. For GPTQ/4bit you need 1/4 that.

For example, assuming GPTQ 4bit a 100GB 16bit model needs 26-30GB of VRAM. The smallest video cards which meet this requirement will be 32GB or 40GB cards. (two 24GB cards in parallel work as well, e.g. 2x3090)

To add: usually when you go to download a ggml model you want a quantized version. People like TheBloke will usually have some RAM requirements for running it, eg: https://huggingface.co/TheBloke/vicuna-13b-v1.3-GGML

The number after q determines how many bits the weights are. Eg q4 means that is 4-bit.

If you use something like KoboldCPP you can only put some of the layers onto the GPU and be able to run larger models that way.

Eg the above linked Vicuna model requires about 10GB of memory at q4, but I have less VRAM than that. I can still run it though.

Let's say I wanted to use one of their quantized models with this OpenLLM project. How would I do that?

Sorry, I don't know. I suspect that it's not possible (yet?). OpenLLM lists a bunch of models in the Github Readme. I think the best way would be to use those for now.