Show HN: Khoj – Chat Offline with Your Second Brain Using Llama 2

Re:Show HN: Khoj – Chat Offline with Your Second Brain Using... by danparsonson on Sunday July 30, 2023 @08:49PM (#36937697)

So just to clarify, is that: Ideal is running the model on a GPU (any brand? Nvidia, AMD, etc.?) with 16GB of GPU RAM, less ideal is running it on the CPU, for which it needs 8GB system RAM? Presumably it will occupy all that memory while it's running?

What about if I have a GPU with 8GB?

[ Parent | on HN ]

Re:Show HN: Khoj – Chat Offline with Your Second Brain Using... by 110 on Sunday July 30, 2023 @10:27PM (#36938180)

Khoj uses Llama 2 7B, 4bit quantized. So it just needs 3.5Gb of RAM (GPU or System) [1].

Khoj and your other apps need more RAM themselves, so practically 8GB of System or GPU RAM should suffice.

Khoj has been tested with CUDA and Metal capable GPUs. So Nvidia and Mac M1+ GPUs should work. I'm think it'll work with AMD GPUs out of the box too but let me know if it doesn't for you? I can look into what needs to be done to get that to work.

[1]: The calculation is [params] * [bytes] GB RAM, so 7 * 0.5 = 3.5Gb

[ Parent | on HN ]