News for hackers. Stuff that matters.
Re:Lycos is still around and its search is pretty good by casion on Friday February 07, 2020 @08:17PM (#22272976)
I just searched the exact title of an article... and the result wasn't on the first page.

So... I tried it a few more times and I didn't get a single hit for any thing I tried. Google picked up each one with "I'm feeling lucky". DDG got them all too.

I wouldn't describe this as "pretty good", or even "good" or even "worth the time I spent this last 90 seconds".

Re:Lycos is still around and its search is pretty good by ChuckMcM on Saturday February 08, 2020 @12:35AM (#22274133)
Just for grins, try the same search on Bing and on Yandex. Look at the first 20 'organic' results (not always easy I know but its worth trying to figure out).

Search indexes have a reasonably unique fingerprint in the rank order of organic results for a given query. The ads and sponsored content are all over the map and perturbed by a/b tests and other things but the organic results typically come straight from the ranking algorithm.

Most "Search portals" (which is the web page that you type at to get your search engine result page from (SERP)) are either using one of the two English language indexes (Bing and Google) with their own ad network, or "blended" index where the search portal might index somethings like Wikipedia and Stack Overflow and then use Bing or Google for the long tail stuff.

The reason for this is economics, you can stand up a search portal on an AWS instance, feed all the queries to Bing out the back and sell ads on it and make a few bucks.

In contrast building your own index requires many more servers, crawling over the content you want to index (if you want to keep it fresh) and generally much larger network and storage costs. So its harder to pay for that with third party ad networks (not impossible, just harder). Pretty much everyone skips this approach for that reason.

Re:Lycos is still around and its search is pretty good by 80386 on Saturday February 08, 2020 @01:02AM (#22274222)
Looks like it's mostly Bing. First five results for 'big chungus':

Google:

- dictionary.com

- knowyourmeme

- artsatmichigan.umich.edu

- urbandictionary

- thetab

Bing:

- knowyourmeme

- silvergames

- wikipedia

- urbandictionary

- joke battles wikia

Lycos:

- knowyourmeme

- urbandictionary

- silvergames

- youtube (BIG CHUNGUS | Official Main Theme | Song by Endigo) [this is the sixth result on Bing]

- joke battles wikia

Yandex:

- big chungus wiki

- youtube: big chungus (song)

- knowyourmeme

- memepedia.ru

- urbandictionary

Re:Lycos is still around and its search is pretty good by growt on Saturday February 08, 2020 @01:58AM (#22274408)
Does anyone know how ddg and co. pay for that? Do they get special rates? Because the normal price for using the Search API of Google or bing seems too expensive to be viable.
Re:Lycos is still around and its search is pretty good by stereolambda on Saturday February 08, 2020 @09:02AM (#22275731)
> Do they get special rates?

I'd suspect so, the sticker price is for the little guy who cannot guarantee them a large volume of organic search queries. With a large volume of course you negotiate.

The most interesting thing here is Startpage. Apparently Google doesn't give access to general Search API (as opposed to custom search limited to some sites) for over a decade now. I idly wonder if folks at Startpage have to maintain an extremely plain and not terribly well marketed site lest they be evicted from their deal, presumably a very very old one. According to Wikipedia they probably use Google since the early 2000s at least.

Re:Lycos is still around and its search is pretty good by ChuckMcM on Saturday February 08, 2020 @03:24PM (#22278108)
By credit card? :-) Sillyness aside, it is a regular contract. About 6 years ago you could get Bing results through the Yahoo BOSS API[1]. I'm not sure what they charge these days but they used to charge about 80 cents per thousand queries.

You take the query terms and send it to an ad network (or ideally your own ad infrastructure) and the network sends back 1 - 10 advertisements based on the query terms. You put those and the "organic" results you got back from the API out as your search page.

You hope that in the thousand queries you process, some of the people click on the ad links instead of the organic links. If enough do, you "make" the difference between the money you get paid by the ad network and the money you paid to the search index.

If you can send a lot of queries, then you are more valuable to the search company than they are to you, and they will start to pay you to send them these queries. (this is what Google calls "traffic acquisition costs" in their earnings reports). But we are talking a lot of queries, you'd need a really popular Linux distribution or Web Browser to pull that off. Just a search landing page won't cut it.

The contract will also detail things that you can't advertise when you get results from the search index. For example in California Google can't allow anyone to show a payday loan ad next to Google results.

And there are subtleties within that as well, you typically get a fraction of the revenue on an ad that gets clicked on as the "first" click after showing the search engines results, but you can get more (or all) of the revenue on pages that you show after that. That trick is used by people who crawl all of Wikipedia and then show a version of Wikipedia with ads where they buy an ad on Google's results for Wikipedia type queries, and then send you to their own page / copy of Wikipedia with their much more profitable ads.

The more complex it gets the more sketchy the players involved.

[1] https://developer.yahoo.com/search/boss/boss_guide/BOSS_Gett...

Re:Lycos is still around and its search is pretty good by smt88 on Friday February 07, 2020 @10:08PM (#22273534)
> I just searched the exact title of an article... and the result wasn't on the first page.

This happens to me on Google a lot. I guess mileage varies.

Re:Lycos is still around and its search is pretty good by anoncake on Friday February 07, 2020 @08:35PM (#22273095)
Did you put the title in quotes?
Re:Lycos is still around and its search is pretty good by casion on Friday February 07, 2020 @08:59PM (#22273203)
I didn't do that on any of the services.
Re:Lycos is still around and its search is pretty good by zepto on Friday February 07, 2020 @10:21PM (#22273577)
Then you didn’t use lycos correctly.
[dead]
Re:Lycos is still around and its search is pretty good by anoncake on Saturday February 08, 2020 @04:47AM (#22274953)
Requiring the user to know how to use a tool in exchange for making it more useful is decidedly not the Apple mentality.
Re:Lycos is still around and its search is pretty good by riku_iki on Saturday February 08, 2020 @12:42AM (#22274156)
They probably didn't crawl it yet, likely having much less resources than google.