News for hackers. Stuff that matters.
Re:Brave Search beta by 1vuio0pswjnm7 on Wednesday June 23, 2021 @12:07AM (#27600685)
"If things go as planned, this may become a paid, ad-free, zero-tracking search engine."

In 1998 when Google's founders announced their search engine they claimed it would be less commercial, more academic, more transparent and they would avoid the influence of advertising. Did things "go as planned." Not even close. What is the lesson here.

Meanwhile, every search submitted during the "beta" period is subject to none of those limitations.

"So, if I am not satisfied with Brave's result, Google's result is on the same page, or just one click away."

Brave is not the first to do that. Check out Gigablast, for example. If I am not mistaken, they also claim to use an "independent index". At least, they provide the source to a crawler and server.^1 That is what people should be excited about. Not Gigablast per se but the idea of an open source search engine that anyone can run. searx is another project worth looking at.^2

How many ways are results promoted and demoted; what are the factors used. Are these search engines that comenters are recommending in this thread transparent. (Making promises on a blog is not "transparency" IMO.) Where is the source code. What are the various server settings that alternatives like Google, DDG, Brave, Startpage, etc. never provide to users. This stuff should matter, yet the discussion of search engines always seems to devolve into personal usage anecdotes and "search shortcuts". Every user has different needs and preferences.

There are many knobs in web search that advertising-supporting tech companies providing "search engine" websites will never let users twiddle. The source code for those servers is not public.

1. To get an idea of type of settings users of popular search engines are not being allowed to control:

(Scroll down to "SEARCH CONTROLS") https://raw.githubusercontent.com/gigablast/open-source-sear...

2. https://github.com/searx/searx

For a current list of searx public instances

   curl https://searx.space/data/instances.json|grep -Eo '(A\+", .[^"]*.{4}[^"]*)'|cut -d/ -f5|uniq|sed 's>.*>https://&>'
Re:Brave Search beta by gigamatt on Wednesday June 23, 2021 @04:29PM (#27609881)
hi, gigablast creator, matt, here. thanks for mentioning gigablast. i've been coding web search engines for almost 25 years so it's always nice to see ppl recognize. i wish more ppl would care about these things. with enough people caring i think i (or we) could make gigablast into a super transparent, private search engine that doesn't rely on big tech like the other guys. really i just need more hardware at this point as that is the main technical hurdle for improving results quality and performance. if somebody would give me like $1M in amd-based minicomputers (i like minicomputers better than big servers - preferably asus) i think we could have something much better and faster, although what is there is pretty good -- this might be enough to really get things going.
Re:Brave Search beta by 1vuio0pswjnm7 on Wednesday June 23, 2021 @09:50PM (#27612941)
in the distribution available on github.com, included are some static binaries for various open source programs. any reason that a static gb binary could not be distributed as well. with linux, i use musl so prefer static to ones linked against gblic.
Re:Brave Search beta by 1vuio0pswjnm7 on Friday June 25, 2021 @02:04AM (#27627611)
anyway, thx for sharing some of your work. very much appreciated.
Re:Brave Search beta by yewenjie on Wednesday June 23, 2021 @02:16AM (#27601354)
> searx

I just gave the docker image a try. It works out of the box but the search time feels slightly more than that of Google - which is one of the reasons I gave up on DDG.

Also, I am curious - if I am hosting it on my own server and using Google as one of the engines - does that not mean my search ultimately goes to Google and they can still profile me?

Re:Brave Search beta by 1vuio0pswjnm7 on Wednesday June 23, 2021 @02:33AM (#27601447)
I just tested the 47 servers listed in https://searx.space/data/instances.json. I did not use a browser. No Javascript, cookies, etc.. A good number of them worked fine.

Who knows what the people running those instances do with the search data they acquire.

What I like about searx though is the list of search engines it potentially targets. Comprehensive lists of search engines on the internet are always valuable. I see searx as a supply of "parts" with which one can make something of their own. I have made a metasearch utility for myself.

Re:Brave Search beta by yewenjie on Wednesday June 23, 2021 @04:18AM (#27602109)
What does your metasearch approach look like?
Re:Brave Search beta by 1vuio0pswjnm7 on Wednesday June 23, 2021 @05:57AM (#27602667)
A. Text-only

B. Search from command line

C. Can open an index.html of saved search results in any browser; each query gets its own SERP; search results are saved in a directory that can be tarballed and compressed allowing simple transfer to any computer with a UNIX userland

D. Easy to add new sites; follows a failry standard template; currently at only eight sites, but adding more (like the ones in searx)

E. Requires only standard UNIX utilities; consists of small shell scripts of less than 2000 chars

F. Fast; no cruft

Unique features:

1. Streamlined SERP; URLs only, minimal HTML, i.e., <a>, <pre>, <ol>, <li>, <!-- -->; no images, Javascript or CSS; SERP contains timestamps in HTML comments to indicate when each query was submitted

2. Each SERP contains deduped batches of results from different search engines; source search engine indicated by short prefix; if desired, can resort to intersperse results from different sources, e.g., sort by URL

3. Continuation of search; allows retrieval 100s of results by spreading searches across periods of time too long for websites to track, thus allowing retrieval of large numbers of search results while avoiding ridiculously small result limits or temporary bans for "searching too fast" <--- I could not find anyone else using this approach

4. By default only minimum headers sent; custom headers can be sent when appropriate for particular site, e.g., DNT to findx.com; allows for complete customisation of presence/absence/content/order/case of HTTP headers, thus can potentially emulate any browser or other HTTP client (also supports HTTP/1.1 pipelining which curl cannot do)

5. Can be used with any TCP client; not limited to one library, e.g., libcurl; works great with proxies like stunnel and haproxy

6. URL params or hidden form fields that can potentially be used to link one SERP with another SERP are removed or rendered ineffective

Re:Brave Search beta by 1vuio0pswjnm7 on Thursday June 24, 2021 @05:28AM (#27615482)
This Brave search beta is quite slow and there is no way to access "page 2". One page of results. Hard to believe this is "good enough" in 2021.
Re:Brave Search beta by pythux on Thursday June 24, 2021 @05:39AM (#27615545)
Hey, Brave Search is currently hosted in the US (I am assuming you are on another country) and latencies will be bigger in other places. We will scale in the future, which should improve speed.

Regarding number of pages of results, please see my answer on another thread: https://news.ycombinator.com/item?id=27597911

TL;DR: we currently show the equivalent of 2 pages of results and will consider more in the future.