News for hackers. Stuff that matters.

A comment on Octopus v2: On-device language model for super agent. Parent comment

Re:Octopus v2: On-device language model for super agent by vessenes on Wednesday April 03, 2024 @12:25PM (#39919439)
Yeah, good idea! I'm not sure if you would be successful mixing LoRA + functional tokens. If you could, that would be great. Then You could ship very light LoRA packs with repositories.

Their LoRA training was I think against their finetuned model, not Gemma-2B directly. But, seems worth playing with -- could be super useful.

[ Parent | on HN ]
Re:Octopus v2: On-device language model for super agent by AlexChen-NexaAI on Thursday April 04, 2024 @06:52PM (#39936796)
Yes, you can do that! We have experimented about this idea. The lora training is reported in the paper already, and it works well. I think it would be very interesting.
[ Parent | on HN ]