Quantized Llama models with increased speed and a reduced memory footprint

Re:Quantized Llama models with increased speed and a reduced... by EliBullockPapa on Thursday October 24, 2024 @04:45PM (#41939571)

Anyone know a nice iOS app to run these locally?

[ Parent | on HN ]

Re:Quantized Llama models with increased speed and a reduced... by simonw on Thursday October 24, 2024 @05:10PM (#41939828)

MLC Chat is a great iPhone app for running models (it's on Android too) and currently ships with Llama 3.2 3B Instruct - not the version Meta released today, its a quantized version of their previous release.

I wouldn't be surprised to see it add the new ones shortly, it's quite actively maintained.

https://apps.apple.com/us/app/mlc-chat/id6448482937

[ Parent | on HN ]

Re:Quantized Llama models with increased speed and a reduced... by Havoc on Thursday October 24, 2024 @08:43PM (#41941255)

Seems much more stable than the last time I tried it too

[ Parent | on HN ]

Re:Quantized Llama models with increased speed and a reduced... by Arcuru on Thursday October 24, 2024 @05:00PM (#41939702)

I access them by running the models in Ollama (on my own hardware), and then using my app Chaz[1] to access it through my normal Matrix client.

[1] - https://github.com/arcuru/chaz

[ Parent | on HN ]

Re:Quantized Llama models with increased speed and a reduced... by drilbo on Thursday October 24, 2024 @06:28PM (#41940483)

https://github.com/a-ghorbani/pocketpal-ai

This was just recently open sourced and is pretty nice. Only issue I've had is very minor UI stuff (on Android, sounds like it runs better on iOS from skimming comments)

[ Parent | on HN ]

Re:Quantized Llama models with increased speed and a reduced... by evbogue on Thursday October 24, 2024 @07:32PM (#41940878)

I'm on Android, however my somewhat elaborate solution was to install Ollama on my home laptop computer and then ssh in when I want to query a model. I figured that'd be better for my phone battery. Since my home computer is behind NAT I run yggdrasil on everything so I can access my AI on the go.

[ Parent | on HN ]

Re:Quantized Llama models with increased speed and a reduced... by behnamoh on Thursday October 24, 2024 @05:16PM (#41939884)

I've been using PocketGPT.

[ Parent | on HN ]