News for hackers. Stuff that matters.
Re:DuckDuckHack is now in Maintenance Mode by romo3 on Sunday September 10, 2017 @11:38AM (#15212653)
Kiwix - most people are too conditioned to think that search has to happen online and don't even realize what is possible offline.

Entire web archives such as the entire dump of wikipedia and stackexchange (including media and indexes for search) can be stored locally. The missing piece is Google level search quality on the local machine. Given that brute force substring search can process Gigabytes in seconds nowadays. If you have enterprise grade server hardware things are reaching 1000GB/s. At this rate, there is no reason to think in a couple years local search of all known human knowledge can't happen on a local device at Google level result quality.

For anyone interested in the search space look into whats possible today in local offline search.

Re:DuckDuckHack is now in Maintenance Mode by tammer on Sunday September 10, 2017 @08:15PM (#15215005)
This is a great observation & seems to dovetail with technologies like IPFS.[1]

[1]: https://ipfs.io

Re:DuckDuckHack is now in Maintenance Mode by amelius on Sunday September 10, 2017 @11:48AM (#15212702)
You might be right, but human knowledge is also expanding, of course. The question is: will it expand faster than hardware capabilities?

Anyway, I wish we'd see more search and NLP related posts here on HN. It deserves far more attention than it gets.

Re:DuckDuckHack is now in Maintenance Mode by romo3 on Sunday September 10, 2017 @11:57AM (#15212742)
For the average person this rate does not matter. They don't need access to the cutting edge of quantum physics, astronomy, dance, art or javascript.

All you have to do is look at the speed at which new info is being added to Wikipedia and Stackoverflow which is stabilizing, i.e. it is not growing as it once was. Basic/foundational knowledge is more or less all covered. https://en.wikipedia.org/wiki/Wikipedia:Modelling_Wikipedia%...

And that sum total comes to 50-60 GB compressed. Think about that number. It's not big.

Re:DuckDuckHack is now in Maintenance Mode by weaksauce on Sunday September 10, 2017 @01:04PM (#15213087)
The sum total of our collective intelligence is equal to an install of gtaV... Crazy.
Re:DuckDuckHack is now in Maintenance Mode by panglott on Monday September 11, 2017 @08:29AM (#15218002)
Wikipedia is not the sum of our collective knowledge. It's little more than the preface.
Re:DuckDuckHack is now in Maintenance Mode by amelius on Sunday September 10, 2017 @01:50PM (#15213327)
We're talking about the "long tail" of information, which is huge also outside of science. Think popular culture.
Re:DuckDuckHack is now in Maintenance Mode by curioussavage on Sunday September 10, 2017 @01:32PM (#15213234)
It would be awesome if you could download dumps of wikepedia filtered by category so You can get the size down. Probably a lot of information that is useless to me in there
Re:DuckDuckHack is now in Maintenance Mode by freeflight on Sunday September 10, 2017 @04:44PM (#15214240)
Kiwix does this, at least to a certain degree: http://wiki.kiwix.org/wiki/Content
Re:DuckDuckHack is now in Maintenance Mode by TheGrassyKnoll on Sunday September 10, 2017 @08:58PM (#15215133)
Listen to Wikipedia http://listen.hatnote.com
Re:DuckDuckHack is now in Maintenance Mode by QAPereo on Sunday September 10, 2017 @02:57PM (#15213683)
NLP is rightly ignored.

https://en.m.wikipedia.org/wiki/Neuro-linguistic_programming...

Edit: Fortunately I'm left feeling foolish, rather than horrified.

Re:DuckDuckHack is now in Maintenance Mode by cantagi on Sunday September 10, 2017 @03:11PM (#15213768)
Re:DuckDuckHack is now in Maintenance Mode by feelin_googley on Sunday September 10, 2017 @11:25PM (#15215680)
The average user's needs are so small.

You do not even need "Google level" for most of today's web users.

You can deliver what users need with respect to web search with much less than "Google level".

For example a simple "<title>" search. This is how Google started.

The entry point into the web should be search for domains. A "<title>" search can do that.

Most users today do not do much searching within websites via Google. They search for websites using Google.

Anyway, you are right about storage space and offline search but obviously that truth misaligns with the "cloud" business narrative and coaxing users to store all their personal data in datacenters instead of on their desk or in their pocket.

Expect much opposition to this simple truth.

Re:DuckDuckHack is now in Maintenance Mode by exikyut on Monday September 11, 2017 @08:38AM (#15218040)
http://web.archive.org/ now provides full-text search, mostly of website titles.

Try it out. You'll find that it's... it feels like a trip back to 1998.

Re:DuckDuckHack is now in Maintenance Mode by detaro on Monday September 11, 2017 @08:49AM (#15218090)
I'd say especially the average user profits from a search system that's somewhat clever and finds things even if they do not ask the exactly right query.

And searching for domains is only a tiny part of it, especially now where a lot of information is stuck in general sites with a lot of content (wikis, Q&A sites, social media sites) and not on special-interest sites. And for many generic searches the special-interest domains are various levels of spam/affiliate marketing.

Re:DuckDuckHack is now in Maintenance Mode by hedora on Sunday September 10, 2017 @02:54PM (#15213662)
PCIe 3 x16 devices have a 16GB/s theoretical max, so 1000GB/s is still out of reach for single machine I/O (though it's not as though search needs anywhere near these bandwidths anyway).
Re:DuckDuckHack is now in Maintenance Mode by colechristensen on Sunday September 10, 2017 @06:53PM (#15214732)
The Intel i9-7900x has 44 PCIe 3.0 lanes and wikipedia tells me each lane has throughput 984.6 MB/s so there's ~40 GB/s, maybe fast compression could make a small integer multiple.

https://www.intel.com/content/www/us/en/products/processors/...

Re:DuckDuckHack is now in Maintenance Mode by brodock on Sunday September 10, 2017 @07:33PM (#15214880)
AMD Threadripper has 64 in all available models: https://en.wikipedia.org/wiki/Zen_(microarchitecture)
Re:DuckDuckHack is now in Maintenance Mode by scalyr2 on Sunday September 10, 2017 @03:16PM (#15213792)
Re:DuckDuckHack is now in Maintenance Mode by joshuamorton on Sunday September 10, 2017 @06:16PM (#15214587)
That blog seems to imply you're using a distributed architecture, ie. not a single machine.