News for hackers. Stuff that matters.
Re:The Secret Sauce behind 100K context window in LLMs: all ... by cavisne on Saturday June 17, 2023 @10:41PM (#36376840)
Claude is a mystery/surprise to me. My mental model has been to train these cutting edge closed source models you need 1) Bespoke supercomputer (no public cloud will cut it) 2) Great dataset (which takes a long time to collect unless you have a partnership with with a search engine) 3) Couple hundred lines of pytorch code to run on the supercomputer 4) A couple of employees with experience in the dark arts of malfunctioning GPU's and exploding gradients

Anthropic is a relatively new startup that probably has 3) & 4) from their history at OpenAI. But I don't see how they could have 1) & 2).

Re:The Secret Sauce behind 100K context window in LLMs: all ... by mareko on Saturday June 17, 2023 @10:46PM (#36376858)
For 2) it looks like they partnered with duckduckgo.
Re:The Secret Sauce behind 100K context window in LLMs: all ... by Kiro on Sunday June 18, 2023 @04:03AM (#36378215)
DDG has no search index (they are using Bing) so I wonder what that actually constitutes.
Re:The Secret Sauce behind 100K context window in LLMs: all ... by mkl on Sunday June 18, 2023 @04:19AM (#36378294)
DDG does have their own index, but also use Bing and many other sources. See the CEO's numerous comments to that effect: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu..., https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Re:The Secret Sauce behind 100K context window in LLMs: all ... by Kiro on Sunday June 18, 2023 @10:24AM (#36380550)
Only for things such as widgets, not regular search results.
Re:The Secret Sauce behind 100K context window in LLMs: all ... by espadrine on Sunday June 18, 2023 @04:45AM (#36378421)
For 1) a public cloud partnership is typically enough.

OpenAI didn’t build a bespoke supercomputer, but trained on Azure (with a preferential contract thanks to their investors): https://openai.com/gpt-4

> GPT-4 was trained on Microsoft Azure AI supercomputers

Cohere trained on GCP: https://techcrunch.com/2021/11/17/google-cloud-teams-up-with...

> Aidan Gomez, co-founder and CEO at Cohere[:] “We scrape the data to train these big models, we train them on massive TPU pods”

Stability AI trains on AWS: https://aws.amazon.com/blogs/machine-learning/stability-ai-b...

> With Amazon SageMaker, Stability AI will build AI models on compute clusters with thousands of GPU or AWS Trainium chips

Given Anthropic’s recent partnership announcement, they likely train on GCP nowadays: https://www.anthropic.com/index/anthropic-partners-with-goog...

> Anthropic will leverage Google Cloud's cutting-edge GPU and TPU clusters to train, scale, and deploy its AI systems