Llama 2 on togetherAI is as bad of a privacy nightmare as OpenAI

Re:Llama 2 on togetherAI is as bad of a privacy nightmare as... by cjonas on Friday September 08, 2023 @02:54PM (#37437815)

I was just talking to a colleague about the need for a service like this. Their pricing is much cheaper than self-hosting on AWS (ml.g5.48xlarge, needed to run 70B Lamma2, is $20.36/hr). But the TOS makes it completely unviable.

If someone can figure out how to standup self-hostable, fine-tuneable models with a serverless architecture, it could be a game changer (allowing devs to use something other than OpenAI/Claude).

[ Parent | on HN ]

Re:Llama 2 on togetherAI is as bad of a privacy nightmare as... by benxh on Friday September 08, 2023 @03:45PM (#37438479)

I've had some success using vast.ai[0] with the Oobabooga LLM WebUI (LLaMA2) instances. One click to start up, minimal editing in the interface settings to enable OpenAI compatible interface.

[0] https://cloud.vast.ai/

[ Parent | on HN ]

Re:Llama 2 on togetherAI is as bad of a privacy nightmare as... by yessen on Friday September 08, 2023 @04:45PM (#37439183)

We offer Llama-2-70B-chat api[0] at Deep Infra. $1 per 1M tokens, support streaming and OpenAI compatible endpoint[1].

0. https://deepinfra.com/meta-llama/Llama-2-70b-chat-hf 1. https://deepinfra.com/docs/advanced/openai_api

[ Parent | on HN ]

Re:Llama 2 on togetherAI is as bad of a privacy nightmare as... by cjonas on Friday September 08, 2023 @06:08PM (#37439967)

Nice, awesome that you can just repoint an OpenAI client to this service. I'll give it a try.

Do you have future plans to support fine-tuning via pay-per-token billing?

Related: Found an interesting Reddit post (rare) discussing how OpenAI manages to do this by loading the LoRA layer at runtime. https://www.reddit.com/r/MachineLearning/comments/15zl7ul/d_...

[ Parent | on HN ]