Re:PyTorch – Torchchat: Chat with LLMs Everywhere
by JackYoustra on Thursday August 01, 2024 @02:50PM
(#41132252)
Probably if you have any esoteric flags that pytorch supports. Flash attention 2, for example, was supported way earlier on pt than llama.cpp, so if flash attention 3 follows the same path it'll probably make more sense to use this when targeting nvidia gpus.
[ Parent
| on HN ]
Re:PyTorch – Torchchat: Chat with LLMs Everywhere
by sunshinesfbay on Thursday August 01, 2024 @06:27PM
(#41134279)
It would appear that Flash-3 is already something that exists for PyTorch based on this joint blog between Nvidia, Together.ai and Princeton about enabling Flash-3 for PyTorch: https://pytorch.org/blog/flashattention-3/
Re:PyTorch – Torchchat: Chat with LLMs Everywhere
by JackYoustra on Thursday August 01, 2024 @06:51PM
(#41134478)
Right - my point about "follows the same path" mostly revolves around llama.cpp's latency in adopting it.