News for hackers. Stuff that matters.

A comment on ETH Zurich and EPFL to release a LLM developed on public infrastructure. Parent comment

Re:ETH Zurich and EPFL to release a LLM developed on public ... by menaerus on Monday July 14, 2025 @07:45AM (#44558992)
Wait, whole (english speaking) web content dataset size is ~50TB?
[ Parent | on HN ]
Re:ETH Zurich and EPFL to release a LLM developed on public ... by zX41ZdbW on Tuesday July 15, 2025 @02:41PM (#44574463)
Yes, if we take the filtered and deduplicated HTMLs of CommonCrawl. I've made a video on this topic recently: https://www.youtube.com/watch?v=8yH3rY1fZEA
[ Parent | on HN ]
Re:ETH Zurich and EPFL to release a LLM developed on public ... by menaerus on Wednesday July 16, 2025 @02:42AM (#44579379)
Fun presentation, thanks! 72min ingestion time for ~81TB of data is ~1TB/min or ~19GB/s. Distributed or single-node? Shards? I see 50 jobs are used for parallel ingestion, and I wonder how ~19GB/s was achieved since ingestion rates were far below that figure last time I played around with CH performance. Granted, that was some years ago.
[ Parent | on HN ]
Re:ETH Zurich and EPFL to release a LLM developed on public ... by zX41ZdbW on Wednesday July 23, 2025 @03:59PM (#44663326)
Distributed across 20 replicas.
[ Parent | on HN ]