vivekraghunatha
vivekraghunatha on Hacker News
We strive to earn our users trust every day. We read every piece of feedback that comes in and try to make the product better. At the end of the day, we only make money if our users use our product and opt to pay for it, so our incentives are as aligned with our users as they can be.
We wrote an op-ed last month in Fast Company re: the challenges of crawling at scale in today's world: https://www.fastcompany.com/90759792/with-google-dominating-...
tldr; there are two big challenges in crawling the web:
* quantity -- how do you crawl the web at O(B) pages per day
* quality -- how do you make sure you are crawling the high quality parts
On the quantity side, the web is a much trickier place to crawl than 10-15 years ago. Even if you build a system that is well behaved, respects rate limits and work w/ webmasters and CDNs to make sure you are treated as a good bot, the following things will bite you:
* some sites only allow googlebot/bingbot via robots
* even when sites allow all search crawlers, you'll see 429s, 503s, crawl delay directives that limit you to very low qps, and other mysterious errors. (most retailers and aggregators)
* working w/ webmasters works only if you manage to get a response from them (good luck)
* many sites require JS rendering (which is 100x more expensive in terms of the number of assets you are crawling)
(OTOH, Cloudflare is a force for good: https://radar.cloudflare.com/verified-bots)
On the quality side, crawl prioritization works best when you have click data, which most small search engines don't have enough of. In the absence of that, good seed sets and high quality inlinks are your friend.
In other news, eng blogs are being written up as we speak. Will post here on HN as they roll out. Also, like marginalia_nu pointed out, we share whatever we've been working on on a weekly basis on our Twitter account.
The signals that matter the most: 1. Anchor text (and all variants of smearing and distinguishing between high quality and low quality anchors) 2. In aggregate, which pages got clicked on on any given query (and all smearing variants -- using ngrams, embeddings, ...). At Neeva, we use it for retrieval and scoring. 3. Query understanding signals mined from the query-click bi-partite graph and the query-query session refinement graph. 4. Page summarization signals built on top of 1 and 2 and body text. 5. To a lesser extent, query-independent page quality signals
Whether you use term-based retrieval or nearest neighbor (embedding) retrieval, a heuristic combination of signals or LambdaMart, whether you calibrate to human eval or clicks, whether your topical relevance function is hand crafted or uses a combination of deep learning and IR signals are all details past that.
tldr; there's a lot of craft in a search ranker, and no one silver bullet. Definitely not just PageRank.
The mobile app ads businesses (UAC campaigns) at Google are just too profitable, and depend too much on IDFA/ADID tracking for the Android team to be able to do anything meaningful here. So we all know the stakes here, and to put this abstract "tracking" into "real world" perspective, a big use case of ADID/IDFA tracking is gaming advertisers identifying "whales" (kids/adults who spend a shit ton of money on their games) and accurately targeting them across all their usage on the phone. If your kids are being tricked into spending hundreds of dollars by these ads, this is why.