Ive got a similar story but with slightly harsher constraints. I also want to build private AI app but for the phone. As you know they have very little RAM per app and I needed to somehow circumvent it. So I picked qwen2.5-0.5B as it was good for the purpose of answering people and keeping the dialog and no overkill + whisper model to add stt feature (437 MB for both). I think that if your app does only one job, you can use smaller model and fine tune it a bit