News for hackers. Stuff that matters.
Re:Discord, or the Death of Lore by zX41ZdbW on Tuesday March 07, 2023 @05:03AM (#35053585)
It sounds like CommonCrawl: https://commoncrawl.org/the-data/get-started/

You can download it, put it into ClickHouse, and get your own professional search engine.

I've made up the term "professional search engine". It's something like Google, but: - accessible by a few people, not publicly available; - does not have a sophisticated ranking or quorum pruning and simply gives your all the matched results; - queries can be performed in SQL, and the results additionally aggregated and analyzed; - full brute-force search is feasible.

PS. Yes, the Reddit dataset stopped updating.

There are no comments on this comment yet.