News for hackers. Stuff that matters.
Re:Internet Search Tips by ergot_vacation on Sunday April 18, 2021 @03:14PM (#26855380)
Some neat stuff here. Only VERY briefly mentioned (so briefly I missed it at first) however: Substituting Yandex for Google is great for many use cases. Being Russian, Yandex is no doubt heavily censored, but only for things important to Russian politics. Ironically, this means that for non-Russian users, it's considerably LESS censored than Google, which has SEVERELY crippled its search in recent years in the name of politics, "politics," DMCA madness, "right to be forgotten" etc.

The image search is especially impressive. Remember when Google Images used to give you actual results when trying to find the source of an obscure image? Yandex still does, and it does a bunch of other neat things too, like automatically trying to transcribe text from an image if it's text-heavy. My instinct is that a lot of this capacity exists in Google Images, but is either mostly hidden from the user or deliberately hobbled to stop the oh so evil content pirates.

Zero privacy of course. Assume the Russian government is watching in realtime as you hammer in another inane search. But for some use cases that's fine.

Re:Internet Search Tips by userbinator on Sunday April 18, 2021 @04:48PM (#26856217)
Bing is also significantly less censored, especially for adult content, but also has a smaller index than Google and less operators.
Re:Internet Search Tips by EE84M3i on Sunday April 18, 2021 @08:39PM (#26858033)
Mentioned in the article, but Yandex also does some degree of facial recognition (or at least similarity) when doing reverse image searches. It works surprisingly well.
Re:Internet Search Tips by Abishek_Muthian on Monday April 19, 2021 @01:04AM (#26859261)
I haven't used Yandex much, But I started noting down failed search results according to the search engines in the hopes of one day creating a 'Search Engine Wall of Shame' to provide genuine feedback to the Search Engines in the areas they could improve[1]. I'll try Yandex for such queries too.

[1] Those who are interested in a 'Search Engine Wall of Shame', URL for the discussion in my profile(#207).

Re:Internet Search Tips by derefr on Sunday April 18, 2021 @06:35PM (#26857155)
Is there a search portal that does backend-side searches of all these politically-disjoint large search providers, and then merges and deduplicates the result?
Re:Internet Search Tips by eli on Sunday April 18, 2021 @07:30PM (#26857575)
Hah that was a big thing in the mid 90s. And, remarkably, this site still seems to be going strong https://www.dogpile.com/

I thought it was a really dumb name at the time, but I remembered it some 20 years later.

Re:Internet Search Tips by 1vuio0pswjnm7 on Monday April 19, 2021 @12:18AM (#26859061)
I have actually been working on a prototype for something similar to the "meta-search" engines of the 90s, but the intent is to deal with the present situation of search engines placing limits on number of results, and to rid SERPs of needless cruft (Javascript, CSS, advertising, etc.) making them look more standardised according to personal taste. I never really liked meta-search engines because they were too slow to return results. Instead this script lets me query search engines directly, one or more at a time, in succession, or all at the same time, and merge the results into simple, aesthetically-pleasing HTML files.

I use a text-only browser to read HTML so what I am describing here is not designed with "modern" browsers in mind.

The approach I take to avoid search result limits is that I search from the command line and store the result URLs in standardised "search results files" in a "search directory", one file per unique query. When I reach the results limit for the particular search engine, I can repeat the search on another search engine. The search results file is created for two reasons: 1. it allows me to strip out all the cruft from SERPs and mix results from different search engines into one file that looks great in a text-only browser, and 2. it tells me where I left off for each search engine, so I can continue any search at a later time, i.e., get more results.

The script reads from the search directory and presents me with a menu of numbered searches. I continue a search at any time by selecting a number. For example:

   1 this is an example  
   2 foo
   3 foo bar
   4 foo bar baz
I typically browse results by pointing the text-only browser at the search directory.

The search result files are each named according to the URL-encoded search string. The file format is very simple. There are three type of lines: 1. a title tag for the search query, 2. a link for each result URL, and 3. an HTML comment for each HTTP request, indicating to the script where I left off, i.e., the last result number; this is more or less equivalent to a "Next page" or "More results" link. Each result URL and comment is prefixed with a search engine identifier, i.e., a prefix. Thus the script can read the search results file comments and I can easily see which results came from XYZ search engine versus ABC search engine.

  <title>this is an example</title>
  <!-- X q=this%20is%20an%20example -->
  X <a href="https://example.com">https://example.com</a><br>
  <!-- X q=this%20is%20and%20example&p=2 -->  
  <!-- A q=this%20is%20an%20example --> 
  A <a href="https://example.net/index.html">https://example.net/index.html</a><br>
  <!-- A q=this%20is%20an%20example&start=2 -->
This example search results file above shows 1 result retrieved from XYZ search engine (prefix "X") and another result retrieved from ABC search engine (prefix "A"). It shows comments indicating where to continue the search for each search engine. Normally there would be around 50-100 result URLs per HTTP request.

This approach assumes the search result limits are temporal, i.e., they are limits on how many results can be retrieved in a given period. That may not be true for every search engine.

Re:Internet Search Tips by 1vuio0pswjnm7 on Sunday April 18, 2021 @07:21PM (#26857496)
"Zero privacy of course."

Surely this is not suggesting that Google searches afford any privacy. ;)

While it might not be the Russian government who is watching, Google searches are certainly not ephemeral nor free from analysis in real-time. Aside from "things important to Russian politics", I would guess Google on behalf of its customers, who could be anyone, including governments, is far more interested in what someone is searching than the Russian government.

The point I am getting at here is that there is privacy from a government and there is privacy from a company. Each is a form of privacy, but only the company is in the business of commercialising the information it derives from violating privacy. (Not to mention that, at least in the US, the government is subject to a body of privacy law that does not apply to companies.)

Both the government and the company may violate privacy in the interests of staying in power. They could, e.g., suppress certain information when it is in their interests. However only the ad services company has the additional motivation to collect information to generate profits. The government may believe it has no choice but to monitor web search. The company OTOH freely chooses to monitor web search, as a business.

Anyway, here is a question I have about Yandex.

Google, Bing and other search engines such as DuckDuckGo now limit the number of results that can be retrieved in one session. Based on personal observation, with Google the ceiling is currently 300, with Bing and DDG, it's more like 250.^1 I wonder if Yandex is doing the same.

These limits by Google, Bing, DDG, etc. are eliminating "discoverability" via searching the web. "General" searches that would yield more than 300 results will not return more than 300 results. Users collecting large numbers of results from a general search is effectively prohibited. Google's idea of discovery is "I'm feeling lucky". Of all the silly changes Google has made to search, the most useless feature persists.

Perhaps "broad" searches do not benefit an online ad services business as much as more specific searches do. Limiting total results also puts more pressure on websites to try to be listed within the first 300. (Solution: Buy ads from Google.) A website who is at position 301 is undiscoverable thanks to Google's inexplicable truncation. Interestingly, on some, perhaps all, of their different "UI's", Google no longer numbers results.

1. To illustrate the truncation, try a search for a common string that would appear in the <title> tag of more than 300 pages on the web. https://www.google.com/search?q=title:[common string]&num=100&filter=0

Re:Internet Search Tips by mgraczyk on Sunday April 18, 2021 @07:38PM (#26857661)
Google uses search data to improve search and show you ads that you're more likely to click on. You can also use incognito/private mode to opt out of this.

Russia uses search data for things like applying pressure to political dissidents, blackmailing family members, harassing and attacking journalists etc. You can't easily opt out of this if you or your family is in Russia.

I consider the Google level of privacy completely acceptable, the Russian one not acceptable.

Re:Internet Search Tips by sellyme on Sunday April 18, 2021 @08:59PM (#26858153)
> Surely this is not suggesting that Google searches afford any privacy. ;)

Depends on who you want privacy from.

Google's entire business model relies on them being the only company on the planet who knows as much about me (or you, or any other reader) as they do. I know that anything I do while using their services is going to be logged and analysed by machines like crazy for the rest of time, but I'm also reasonably confident that they're not going to let the interns page through this data whenever they get bored, and I'm very confident that they're going to do everything in their power to prevent my complete browser history from ending up for sale on some data breach forum.

Re:Internet Search Tips by 1vuio0pswjnm7 on Tuesday April 20, 2021 @05:44PM (#26882192)
In some cases, I can get over 400 results with Google, but never more than 500.
Re:Internet Search Tips by RinTohsaka on Tuesday April 20, 2021 @10:44AM (#26875363)
Is there something I could self host that would perform aa well as yandex?