News for hackers. Stuff that matters.

A comment on Newsgroups and the Internet Archive: I Made a Difference.

Re:Newsgroups and the Internet Archive: I Made a Difference by zokier on Saturday March 24, 2018 @04:54PM (#16668872)
> After a month I had most newsgroups - excepting binaries - and it came to 800GB.

> Trying to index THAT lot was impossible

Stupid question, but why would indexing 800GB of newsgroup postings be impossible?

[ Parent | on HN ]
Re:Newsgroups and the Internet Archive: I Made a Difference by mailxplorer on Saturday March 24, 2018 @07:15PM (#16669605)
Clucene was way too slow for body text, more than 1GB. I had my own header parser in C++ (though you can do that in Python easily).

I'm trying again on that 800GB with KISS DB (append-only hashtable), and Elasticsearch. Doesn't matter if GPL because it's a website.

[ Parent | on HN ]
Re:Newsgroups and the Internet Archive: I Made a Difference by GuacheSuedeHN on Sunday March 25, 2018 @04:51AM (#16671358)
Do you mind sharing the code ? I think that is an interesting thing to see
[ Parent | on HN ]