News for hackers. Stuff that matters.
Re:Dot: use of local LLMs and RAG in particular to interact ... by alexpinel on Sunday April 07, 2024 @12:48PM (#39961969)
Right now the minimum amount of RAM I would recommend is 16gb, I think it can run with less memory but that will require a few changes here and there (although they might reduce performance). I would also strongly recommend using a GPU over CPU, in my experience it can make the LLM run twice as fast if not more. Only Nvidia GPUs are supported for now and the CUDA toolkit 12.2 is required to run Dot.

There are no comments on this comment yet.