News for hackers. Stuff that matters.

A comment on Dot: use of local LLMs and RAG in particular to interact with documents. Parent comment

Re:Dot: use of local LLMs and RAG in particular to interact ... by alexpinel on Sunday April 07, 2024 @11:55AM (#39961567)
With those settings I would recommend GPU. CUDA acceleration really makes it faster, but keep in mind the CUDA toolkit 12.2 install will be a some 3-4gb
[ Parent | on HN ]
Re:Dot: use of local LLMs and RAG in particular to interact ... by pentagrama on Sunday April 07, 2024 @12:59PM (#39962057)
Thank you!
[ Parent | on HN ]