News for hackers. Stuff that matters.
Re:The Secret Sauce behind 100K context window in LLMs: all ... by espadrine on Sunday June 18, 2023 @04:45AM (#36378421)
For 1) a public cloud partnership is typically enough.

OpenAI didn’t build a bespoke supercomputer, but trained on Azure (with a preferential contract thanks to their investors): https://openai.com/gpt-4

> GPT-4 was trained on Microsoft Azure AI supercomputers

Cohere trained on GCP: https://techcrunch.com/2021/11/17/google-cloud-teams-up-with...

> Aidan Gomez, co-founder and CEO at Cohere[:] “We scrape the data to train these big models, we train them on massive TPU pods”

Stability AI trains on AWS: https://aws.amazon.com/blogs/machine-learning/stability-ai-b...

> With Amazon SageMaker, Stability AI will build AI models on compute clusters with thousands of GPU or AWS Trainium chips

Given Anthropic’s recent partnership announcement, they likely train on GCP nowadays: https://www.anthropic.com/index/anthropic-partners-with-goog...

> Anthropic will leverage Google Cloud's cutting-edge GPU and TPU clusters to train, scale, and deploy its AI systems

There are no comments on this comment yet.