Re:Llama 2 on togetherAI is as bad of a privacy nightmare as...
by cjonas on Friday September 08, 2023 @02:54PM
(#37437815)
I was just talking to a colleague about the need for a service like this. Their pricing is much cheaper than self-hosting on AWS (ml.g5.48xlarge, needed to run 70B Lamma2, is $20.36/hr). But the TOS makes it completely unviable.
If someone can figure out how to standup self-hostable, fine-tuneable models with a serverless architecture, it could be a game changer (allowing devs to use something other than OpenAI/Claude).
Re:Llama 2 on togetherAI is as bad of a privacy nightmare as...
by benxh on Friday September 08, 2023 @03:45PM
(#37438479)
I've had some success using vast.ai[0] with the Oobabooga LLM WebUI (LLaMA2) instances. One click to start up, minimal editing in the interface settings to enable OpenAI compatible interface.
Re:Llama 2 on togetherAI is as bad of a privacy nightmare as...
by yessen on Friday September 08, 2023 @04:45PM
(#37439183)
We offer Llama-2-70B-chat api[0] at Deep Infra. $1 per 1M tokens, support streaming and OpenAI compatible endpoint[1].
0. https://deepinfra.com/meta-llama/Llama-2-70b-chat-hf 1. https://deepinfra.com/docs/advanced/openai_api
Re:Llama 2 on togetherAI is as bad of a privacy nightmare as...
by cjonas on Friday September 08, 2023 @06:08PM
(#37439967)
Nice, awesome that you can just repoint an OpenAI client to this service. I'll give it a try.
Do you have future plans to support fine-tuning via pay-per-token billing?
Related: Found an interesting Reddit post (rare) discussing how OpenAI manages to do this by loading the LoRA layer at runtime. https://www.reddit.com/r/MachineLearning/comments/15zl7ul/d_...
2.5 is interesting
> Your information, including Personal Data, is processed at the Company's operating offices and in any other places where the parties involved in the processing are located. It means that this information may be transferred to — and maintained on — computers located outside of Your state, province, country or other governmental jurisdiction where the data protection laws may differ from those in Your jurisdiction
> Your consent to this Privacy Policy followed by Your submission of such information represents Your agreement to that transfer.
And the table in the middle of the policy...
> Category Collected? Disclosed?
> Identifiers. Name, alias, postal address, unique personal identifier, online identifier, Internet Protocol (IP) address, email address, account name, or other similar identifiers YES YES
> Commercial information. Records of personal property, products or services purchased, obtained, or considered, or other purchasing or consuming histories or tendencies. YES YES
also interesting...
Invested in / partnered with NVIDIA & Hugging Face and others...