Ask HN: Has Google search become quantitatively worse?

Re:Ask HN: Has Google search become quantitatively worse? by wmil on Tuesday November 30, 2021 @11:27AM (#29393870)

It's analogous to predatory pricing. Google has all the necessary tech to go back to doing good searches. They can easily pivot back in a few months.

A new startup would need to dump loads of money into servers and building their own tech.

So there's just no way to recoup the cost of building a competitor.

[ Parent | on HN ]

Re:Ask HN: Has Google search become quantitatively worse? by systemvoltage on Tuesday November 30, 2021 @11:33AM (#29393943)

You might be surprised how much compute it takes.

Not the same thing but, entire SO website runs on like 2 door sized racks. Search engine might be a different thing but if you have funding to get started, hardware costs aren’t going to be impossibly huge. Most is labor (engineering).

I’m curious, how much compute power it takes to index the whole web? I presume queries are super fast.

[ Parent | on HN ]

Re:Ask HN: Has Google search become quantitatively worse? by estebarb on Tuesday November 30, 2021 @12:36PM (#29394868)

The monthly snapshots of CommonCrawl contains around 64k files of HTML, metadata and text only. The text only and metadata are around 250MB each, so in total the corpus would be around 16TB of text and 16TB of metadata. I don't know how much it costs to them, but when I used my own crawler I was able to download 500k webpages in around 4 hours (using the $80/month basic DigitalOcean server... let's say a total of $1/1M web pages, just the crawling process).

[ Parent | on HN ]

Re:Ask HN: Has Google search become quantitatively worse? by alangibson on Tuesday November 30, 2021 @11:52AM (#29394204)

IMHO, indexing the whole web is an antifeature for technical and research topics. But there are projects like CommonCrawl that crawl a good portion of the web and give it away free.

[ Parent | on HN ]