News for hackers. Stuff that matters.

sunshinesfbay

Re:PyTorch – Torchchat: Chat with LLMs Everywhere on Thursday August 01, 2024 @06:34PM (#41134334)
I think this is not about new Pytorch features, although it requires the latest Pytorch and Executorch making me think that some features in pytorch and executorch got extended optimized for this use case?

What makes this cool is that you can use the same model and the same library and apply to server, desktop, laptop and mobile on iOS and Android, with a variety of quantization schemes and other features.

Definitely still some rough edges as I'd expect from any first software release!

Re:PyTorch – Torchchat: Chat with LLMs Everywhere on Thursday August 01, 2024 @06:30PM (#41134309)
It's an end to end solution that supports the same model from server (including OpenAI API!) to mobile. To the extent that you just want to run on one specific platform, other solutions might work just as well?
Re:PyTorch – Torchchat: Chat with LLMs Everywhere on Thursday August 01, 2024 @06:28PM (#41134293)
Pretty cool! What are the steps to use these on mobile? Stoked about using ollama on my iPhone!
Re:PyTorch – Torchchat: Chat with LLMs Everywhere on Thursday August 01, 2024 @06:27PM (#41134279)
It would appear that Flash-3 is already something that exists for PyTorch based on this joint blog between Nvidia, Together.ai and Princeton about enabling Flash-3 for PyTorch: https://pytorch.org/blog/flashattention-3/