News for hackers. Stuff that matters.
Re:LLMs use a surprisingly simple mechanism to retrieve some... by krainboltgreene on Thursday March 28, 2024 @01:12PM (#39854525)
> have access to virtually the entire internet

It isn't even close to 1% of the internet, much less virtually the entire internet. According to the latest dump, Common Crawl has 4.3B pages, but Google in 2016 estimated there are 130T pages. The difference between 130T and 4.3B is about 130T. Even if you narrow it down to Google's searchable text index it's "100's of billions of pages" and roughly 100P compared to CommonCrawl's 400T.

Re:LLMs use a surprisingly simple mechanism to retrieve some... by fspeech on Thursday March 28, 2024 @02:04PM (#39855237)
130T unique pages? That seems highly unlikely as that averages to over 10000 pages for each human being alive. If gp merely wants texts of interest to self as opposed to an accurate snapshot it seems LLMs should be quite capable, one day.
Re:LLMs use a surprisingly simple mechanism to retrieve some... by cypress66 on Friday March 29, 2024 @08:14AM (#39863100)
Is it? Every user profile in every website is a page. Every single tweet is a page.
Re:LLMs use a surprisingly simple mechanism to retrieve some... by fspeech on Thursday April 11, 2024 @03:23PM (#40005820)
Tweets count. HN posts also count (actually as high quality texts :). IOT devices reporting status based on the same templates should not qualify as unique pages (count the number of templates if you want). Now if I do a search for some news, many almost verbatim copies would show up. They should only count as one, as we are looking for unique texts!
Re:LLMs use a surprisingly simple mechanism to retrieve some... by whatever1 on Friday March 29, 2024 @01:02AM (#39860884)
The internet to me and to most of the people is the 10 first search results for the various terms we search for.