News for hackers. Stuff that matters.
Re:PyTorch – Torchchat: Chat with LLMs Everywhere by JackYoustra on Thursday August 01, 2024 @02:50PM (#41132252)
Probably if you have any esoteric flags that pytorch supports. Flash attention 2, for example, was supported way earlier on pt than llama.cpp, so if flash attention 3 follows the same path it'll probably make more sense to use this when targeting nvidia gpus.
Re:PyTorch – Torchchat: Chat with LLMs Everywhere by sunshinesfbay on Thursday August 01, 2024 @06:27PM (#41134279)
It would appear that Flash-3 is already something that exists for PyTorch based on this joint blog between Nvidia, Together.ai and Princeton about enabling Flash-3 for PyTorch: https://pytorch.org/blog/flashattention-3/
Re:PyTorch – Torchchat: Chat with LLMs Everywhere by JackYoustra on Thursday August 01, 2024 @06:51PM (#41134478)
Right - my point about "follows the same path" mostly revolves around llama.cpp's latency in adopting it.