News for hackers. Stuff that matters.
Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @11:57AM (#26593607)
The bigger problem, to me, is not around crawling. It's the asymmetrical power Google has after crawling.

Google is obviously on a mission to keep people on Google owned properties. So, they take what they crawl and find a way to present that to the end user without anyone needing to visit the place that data came from.

Airlines are a good example. If you search for flight status for a particular flight, Google presents that flight status in a box. As an end user, that's great. However, that sort of search used to (most times) lead to a visit to the airline web site.

The airline web site could then present things Google can't do. Like "hey, we see you haven't checked in yet" or "TSA wait times are longer than usual" or "We have a more-legroom seat upgrade if you want it".

Google took those eyeballs away. Okay, fine, that's their choice. But they don't give anything back, which removes incentives from the actual source to do things better.

You see this recently with Wikipedia. Google's widgets have been reducing traffic to Wikipedia pretty dramatically. Enough so that Wikipedia is now pushing back with a product that the Googles of the world will have to pay for.

In short, I don't think the crawler is the problem. And I don't think Google will realize what the problem is until they start accidentally killing off large swaths of the actual sources of this content by taking the audience away.

Re:Only Google is really allowed to crawl the web by throwaway_kufu on Friday March 26, 2021 @12:06PM (#26593725)
They are not just taking away internet traffic, but in the flights example, they actually acquired an aggregate flight/travel company and so they are actually entering markets and competing with their own ad customers.

Then it comes fully circle to Google unfairly using their market position vis-a-vis data, search and advertising. It’s a win-win Google lets the data dictate which markets to enter and on one hand they can jack up advertising fees on customers/competitors and unfairly build their own service into search above both ads and organic results.

Re:Only Google is really allowed to crawl the web by danielscrubs on Friday March 26, 2021 @12:32PM (#26594049)
Be careful when using Google Flight, last time I checked they use significantly less margins between flights so it’s shorter trips but much riskier.
Re:Only Google is really allowed to crawl the web by aetherane on Friday March 26, 2021 @12:58PM (#26594336)
You can screwed any time you book a connecting flight on two different airlines even if the times aren't tight. For instance if one is cancelled.

If you use the same airline they will make sure you get to the destination.

Re:Only Google is really allowed to crawl the web by matwood on Friday March 26, 2021 @01:07PM (#26594458)
That's true, but it can save you a ton of money. You just have to be aware of the risks and plan accordingly.

I have typically used this strategy when flying back to the US from the EU. Take an EZJet or similar low cost airline from random small EU city to a larger EU city like Paris, London, Frankfurt, etc... and book the return trip to the US from the larger city. I've also been forced to do this from some EU cities since there was no connecting partner with a US airline.

Re:Only Google is really allowed to crawl the web by hodgesrm on Friday March 26, 2021 @06:19PM (#26597684)
The difference is mind-boggling in some cases. On one trip in 2019 I had the following coach fair choices for SFO - Moscow return trip tickets booked 3 weeks prior to departure.

* UA or Lufthansa round trip (single carrier) $3K

* UA round trip SFO - Paris + Aeroflot round trip Paris - Moscow: $1K

No amount of search could reduce the gap. I went with the second option. The gap is even bigger if you have a route with multiple segments.

Re:Only Google is really allowed to crawl the web by sitkack on Friday March 26, 2021 @08:07PM (#26598486)
https://www.airtreks.com/ will do this for you with a person. phenomenal service.

To anyone from airtreks, I love you so much!

Re:Only Google is really allowed to crawl the web by throwaway1777 on Friday March 26, 2021 @01:35PM (#26594766)
Yeah this strategy is good, but you need to allow a long layover like 6 hours if you have to go through immigration and change airports for the connection which happens pretty often with ryanair and ezjet. It’s a big pain, but it does save money.
Re:Only Google is really allowed to crawl the web by cbenneh on Friday March 26, 2021 @02:55PM (#26595588)
If you're booking each leg with different carrier, I find it best to pay the little extra with kiwi.com and they give you guarantee for the connection. I missed connection twice and they always got me on the next flight to the destination for free.
Re:Only Google is really allowed to crawl the web by Cullinet on Friday March 26, 2021 @09:24PM (#26598985)
in my ideal world the software ITA wrote for airlines and is now owned by Google would be in the hands of consumers and the airlines could have adapted to shifts in demand probably without the need for abrupt cessation of services and human fatigue on industry employees caused when route optimisation analysis tempts executives with what I suspect are ultimately fictitious net present savings.
Re:Only Google is really allowed to crawl the web by HenryBemis on Friday March 26, 2021 @06:59PM (#26598010)
> even if the times aren't tight

Depending on the definition of "tight" each of us have. I remember having 40mins in Munich, and that is a BIG airport. Especially if you disembark on one side of the terminal and your flight is on the far/opposite end. That's 25-30mins brisk walking. With 5000 people in-between you could as well miss your flight. No discussion about stopping to get a coffee or a snack.. you'll miss your flight.

Re:Only Google is really allowed to crawl the web by sershe on Saturday March 27, 2021 @01:39AM (#26600197)
It doesn't really matter if it's on the same airline, it just has to be in the same reservation. Usually, that is the same thing; however on international and hyper-local (the kind that end with a Cessna) flights you'd often have several airlines with codeshares, and you could buy two separate tickets on the same airline if you wanted.
Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @11:45PM (#26599739)
Another approach that's interesting is "buy long, fly short". Sometimes buying A->B->C and getting off at B is cheaper than just buying A->B. But, airlines can cancel the the A->B->C flight and replace it with an A->B and A->C, and place you on the A->C flight.
Re:Only Google is really allowed to crawl the web by slymon99 on Friday March 26, 2021 @12:38PM (#26594110)
Can you elaborate on this? Do you mean shorter layovers?
Re:Only Google is really allowed to crawl the web by bombcar on Friday March 26, 2021 @12:49PM (#26594243)
It sounds like it - and third-party companies will often show you flights that involve different companies on the different legs - which can leave you in a pickle because technically each airline's job is to get you to the end of THIER flight, not the entire journey.
Re:Only Google is really allowed to crawl the web by Scoundreller on Friday March 26, 2021 @01:17PM (#26594579)
And sometimes with a change of airport!
Re:Only Google is really allowed to crawl the web by foepys on Friday March 26, 2021 @02:10PM (#26595115)
I remember when in Germany some budget airlines used to say they'd fly to "Frankfurt" (FRA) but actually flew to "Frankfurt-Hahn" (HHN) - 115km away. After arrival in HHN they put you on a bus to FRA that took about 2 hours.
Re:Only Google is really allowed to crawl the web by SV_BubbleTime on Friday March 26, 2021 @01:24PM (#26594658)
Oh don’t worry, you have 15 on-paper minutes to go from A1 to A70 in Detroit... in January... and the shuttle is down.
Re:Only Google is really allowed to crawl the web by kingo55 on Friday March 26, 2021 @06:07PM (#26597572)
Even before it gets to that point, they routinely display snippets off regular websites and show ads next to it.

Keeping users from clicking through to organic results helps them generate more revenue.

Re:Only Google is really allowed to crawl the web by killjoywashere on Friday March 26, 2021 @08:39PM (#26598685)
Here’a a thought: most companies don’t actually want to serve a ton of extra pages. For example, airlines just want to fly passengers. They don’t care who puts those butts in seats and they would fully acknowledge that they aren’t able to deliver a better flight search than Google can. I mean, sure, some small team of web developers at every airline is pissed, but the CEO needs butts in seats to keep the pilot and service unions off her back. Her own web dev team is the least of her problems.
Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @10:29PM (#26599347)
"For example, airlines just want to fly passengers. They don’t care who puts those butts in seats"

Not sure who you talked to, but I've never heard that before. They all want to sell more direct and forego GDS fees and/or other types of fees and commissions. I'd love to see a quote from an airline VP or above that they don't care about their distribution model, boosting direct sales percentages, etc.

Re:Only Google is really allowed to crawl the web by hawaiianbrah on Friday March 26, 2021 @10:11PM (#26599237)
Sure, but if you aren’t controlling the experience of getting butts in seats, it’s harder to upsell and make even _more_ money from those butts.
Re:Only Google is really allowed to crawl the web by chii on Friday March 26, 2021 @10:53PM (#26599487)
the OP wasn't talking about google competing with airlines, but with other flight aggregating and search/booking services, by abusing their monopoly on web search.
Re:Only Google is really allowed to crawl the web by markdown on Friday March 26, 2021 @07:47PM (#26598370)
> Google lets the data dictate which markets to enter and on one hand they can jack up advertising fees on customers/competitors and unfairly build their own service into search above both ads and organic results.

Just like Amazon with Amazon Basics.

Re:Only Google is really allowed to crawl the web by Cullinet on Friday March 26, 2021 @09:15PM (#26598930)
isn't the solution to disaffective aqui - hiring the restoration of the ability to IPO companies like ITA Software so there's another way to reach financial security for talented programmer - entrepreneurs? for that matter, do the conditions for recreating the frequency of lower level programmer millionaires (hardly a family home debt free today) like Microsoft created, require the recreation also of senior executive abuse of option schemes? It seems to me that making it a reasonable chance of becoming at least financially secure for not irrational amounts of dedication and 90 weeks, and underwriting that with the greater robustness of larger companies and hence livable salaries, instead of trying to sustain the apparent startup free for all figuring to a common heat death of the advertising budgetary universe?
Re:Only Google is really allowed to crawl the web by marshmallow_12 on Friday March 26, 2021 @12:20PM (#26593906)
Aren't there anti trust laws to prevent this kind of thing?
Re:Only Google is really allowed to crawl the web by Majromax on Friday March 26, 2021 @01:34PM (#26594763)
Antiturst laws are hard to enforce in the United States.

Monopolies themselves aren't illegal. To be convicted of an antitrust violation, a firm needs to both have a monopoly and needs to be using anticompetitive means to maintain that monopoly. The recent "textbook" example was of Microsoft, which in the 90s used its dominant position to charge computer manufacturers for a Windows license for each computer sold, regardless of whether it had Windows installed or was a "bare" PC.

Depending on how you define the market, Google may not even have a monopoly. It's probably dominant enough in web search to count, but if you look at its advertising network it competes with Facebook and other ad networks. In the realm of travel planning (to pick an example from these comments), it's barely a blip.

Furthermore, Google can potentially argue it's not being anticompetitive: all businesses use their existing data to optimize new products, so Google could claim that it not doing so would be an artificial straightjacket.

Re:Only Google is really allowed to crawl the web by arrosenberg on Friday March 26, 2021 @03:20PM (#26595850)
It's not that hard, we're just out of practice due to the absurd Borkist economic theories we've been operating under for 40+ years. The laws are all there if the head of the DOJ antitrust division has the gumption to go reverse some bad precedents.

> In the realm of travel planning (to pick an example from these comments), it's barely a blip.

They used their monopoly in web search to gain non-negligible marketshare in entirely unrelated industry. That's text book anti-competitive behavior.

Google can argue whatever they want, but the argument that they're enabling other businesses is a bad one. It casts Google as a private regulator of the economy, which is exactly what antitrust laws are intended to deal with.

Re:Only Google is really allowed to crawl the web by pmiller2 on Friday March 26, 2021 @02:36PM (#26595363)
Is web search even a "market" independent of ads?
Re:Only Google is really allowed to crawl the web by rijoja on Friday March 26, 2021 @03:23PM (#26595884)
yes
Re:Only Google is really allowed to crawl the web by pmiller2 on Friday March 26, 2021 @03:59PM (#26596268)
Where's the money?
Re:Only Google is really allowed to crawl the web by ClumsyPilot on Friday March 26, 2021 @10:44PM (#26599425)
That's like arguing that newspapers are not a market, because it makes money from ads.
Re:Only Google is really allowed to crawl the web by pmiller2 on Friday March 26, 2021 @10:46PM (#26599439)
No, I was asking if web search was a market independent of ads.

BTW, newspapers also make money from subscriptions and sales of copies, so your analogy is doubly wrong.

Re:Only Google is really allowed to crawl the web by throwaway_kufu on Saturday March 27, 2021 @12:08PM (#26603293)
In collecting and selling your data to 3rd parties.
Re:Only Google is really allowed to crawl the web by supernovae on Friday March 26, 2021 @12:57PM (#26594335)
Just tell people to stop using google. Go direct.
Re:Only Google is really allowed to crawl the web by zentiggr on Friday March 26, 2021 @02:34PM (#26595352)
Upvoted - regardless how pointless some people might think this comment is, it really is the ONLY way that Google is going to drop out of its aggregate lead position.

Enough people realizing Google is trapping and cannibalizing traffic to the other sites it feeds off of, and choosing to do other things EXCEPT touching Google properties, is THE ONLY way they'll be unseated.

No clear legal path to stop a bully means it's an ethical / habit path.

Not saying there's any easy way, just that this is it.

Re:Only Google is really allowed to crawl the web by smaryjerry on Saturday March 27, 2021 @10:19AM (#26602422)
I find those little snippets actually mostly worthless, maybe because I’ve seen enough of them taken out of context or basically using a snippet from someone who figured out SEO properly, meanwhile the correct information may be down a couple links or not there at all.
Re:Only Google is really allowed to crawl the web by throwaway_kufu on Saturday March 27, 2021 @12:04PM (#26603261)
Laws are one thing and enforceability another.

In fact Google had to make certain concessions in order for the Google Flights acquisition to get regulatory approval.

IIRC a Chinese firewall between Google data and Google Flights...but like many regulations they were likely written by Google lobbyists aka the industry experts. Because at the end of the day Google flights: 1. Still has the built in widget above organic results and 2. They still bid on their own ad spots jacking up costs on competition which is ultimately passed on to consumers.

Re:Only Google is really allowed to crawl the web by midoBB on Friday March 26, 2021 @12:44PM (#26594192)
Anti-trust in the US tend to not hit the big tech players as much they do other sectors. Also there is actually a debate in the judicial system about the extent of Anti trust laws themselves.
Re:Only Google is really allowed to crawl the web by sitkack on Friday March 26, 2021 @08:10PM (#26598505)
Chicago school basically published a bunch of position papers that made feudal corporations a legal entity that "aren't monopolies" because the give things away for free. Because the consumer isn't paying, it can't be bad.
Re:Only Google is really allowed to crawl the web by sangnoir on Friday March 26, 2021 @02:06PM (#26595086)
The current anti-trust doctrine in the US has a goal of protecting consumers - not competition. What Google is doing is arguably great for consumers but awful to their competitors/other organizations. Technically, companies can simply block Google using robots.txt - but in reality that will lose them more money than the current partial disintermediation by Google is costing them - and Google knows this.

It's a tall order to convince the courts that Google's actions consumers, or is illegal: after all, being innovative in ways that may end up hurting the competition is a key feature of a capitalist society - proving that a line has been crossed is really hard, by design.

Re:Only Google is really allowed to crawl the web by speeder on Friday March 26, 2021 @02:30PM (#26595297)
consumers are in this case the advertisers.

google has a monopoly on search ads and does enforce it, being a drain on the economy since in many fields you only succeed if you spend on search ads

Re:Only Google is really allowed to crawl the web by sangnoir on Friday March 26, 2021 @03:21PM (#26595867)
> consumers are in this case the advertisers.

If someone could convince the courts that this is correct, then I'm sure Google would lose. However, I bet dollars to donuts Google's counter-arguement would be that the people doing the searching and quickly finding information are also consumers, and they outnumber advertisers and may be harmed by any proposed remediation in favor of advertisers.

Re:Only Google is really allowed to crawl the web by basch on Friday March 26, 2021 @04:14PM (#26596408)
googles answer to this at yesterdays hearing..

Search isnt a single category. If you break it down, they arent a monopoly. For example. 1/2 of PRODUCT SEARCHES begin on Amazon. It's probably hard to argue Google as a monopoly if who they see as their main competitor has half the market share.

Re:Only Google is really allowed to crawl the web by smaryjerry on Saturday March 27, 2021 @10:32AM (#26602527)
The US is so behind in identifying markets in technology which is what is leading to this dominance by a few companies and their resulting monopoly like power. We had already figured out that you can be dominant in only a subset of a market. For example Disney was forced to sell off fox sports channels when it purchased fox because it already owned espn and would have dominated sports TV. That’s the thing it wasn’t even just TV but a subset, sports TV. That identification is where we are behind. As of now no, one in the FTC knows what makes a market or why say YouTube and Facebook may both show large amounts of video content but are absolutely not competitors in video content space. It is because the functions are completely different. YouTube is barely a social network at all despite having users and comments and pages. Facebook is hardly a video platform at all because it isn’t profitable for users to focus on Facebook videos and make ad money.
Re:Only Google is really allowed to crawl the web by sn_master on Friday March 26, 2021 @10:51PM (#26599477)
Anecdotally, its true for books too. Amazon is a great way to figure out which books are the best reviewed before deciding to get one (whether paper, kindle or 'other means').
Re:Only Google is really allowed to crawl the web by jacquesm on Friday March 26, 2021 @07:47PM (#26598371)
That's so disingenuous there should be a new term for it.
Re:Only Google is really allowed to crawl the web by sitkack on Friday March 26, 2021 @10:11PM (#26599243)
gSplain or gWash
Re:Only Google is really allowed to crawl the web by anoncake on Saturday March 27, 2021 @06:20AM (#26601280)
Which is shortsighted. If competition did not benefit consumers, there would be no need for it anyway.
Re:Only Google is really allowed to crawl the web by throwaway_kufu on Saturday March 27, 2021 @12:10PM (#26603318)
Anti-trust above all recognizes competition benefits consumers. And so “unfairly competing” is prohibited, because it is bad for the market, thus bad for consumers.
Re:Only Google is really allowed to crawl the web by adamcstephens on Friday March 26, 2021 @12:31PM (#26594026)
Yes, but they lack enforcement.
Re:Only Google is really allowed to crawl the web by samuelizdat on Friday March 26, 2021 @12:30PM (#26594024)
That depends, would Google let us know?
Re:Only Google is really allowed to crawl the web by rijoja on Friday March 26, 2021 @03:25PM (#26595911)
not if they could avoid it
Re:Only Google is really allowed to crawl the web by onlyrealcuzzo on Friday March 26, 2021 @12:28PM (#26593993)
Wikipedia isn't monetized. Doesn't it benefit them if Google is serving their content for free and people are finding the information they want without having to hit Wikipedia??

And also, isn't Google the largest sponsor for Wikipedia already? In 2019 - Google donated $2M [1]. In 2010, Google also donated $2m [2].

[1] https://techcrunch.com/2019/01/22/google-org-donates-2-milli...

[2] https://en.wikipedia.org/wiki/Wikimedia_Foundation

Re:Only Google is really allowed to crawl the web by kelnos on Friday March 26, 2021 @03:12PM (#26595771)
> Wikipedia isn't monetized.

No, but they often ask for donations when you visit the site, which people won't see if they just see the in-line blurb from Wikipedia on the Google results page.

> In 2019 - Google donated $2M [1]. In 2010, Google also donated $2m [2].

$2M is a pittance compared to what I expect Google believes is the value of their Wikipedia blurbs. If Wikipedia could charge for use of this data (which another commenter claims they are working on doing), they could easily make orders of magnitude more money from Google.

Of course, my expectation is that Google would rather drop the Wikipedia blurbs entirely, or source the data elsewhere, than pay significantly more.

Re:Only Google is really allowed to crawl the web by tylerhou on Friday March 26, 2021 @03:36PM (#26596040)
Unlikely that Wikipedia will be able to charge for content, seeing as all of their content is CC-BY-SA licensed. https://en.wikipedia.org/wiki/Wikipedia:Licensing_update

They may be able to charge for bandwidth (if you want to use a Wikipedia image, you can use Wikipedia's enterprise CDN instead of their own), but their licensing allows me to rehost content as long as I follow the attribution & sublicensing terms.

Google has no problem operating their own CDNs, so I find it unlikely that Wikipedia will be able to monetize Google search results in such a manner as you described.

Disclaimer: I work for Google; opinions are my own.

Re:Only Google is really allowed to crawl the web by aabhay on Friday March 26, 2021 @08:20PM (#26598559)
Fine, the content is free. But if your crawlers want access to the content, then pay! Simple as that.
Re:Only Google is really allowed to crawl the web by rezonant on Friday March 26, 2021 @10:24PM (#26599304)
Will it be a flat fee, so that I, a lowly one-man crawler developer will not be able to afford it? Will it be that only Google can afford it, thus making their monopoly position even stronger?

Is there a Wikipedia crawling "welfare" program if I'm not a trillion dollar mega company?

Re:Only Google is really allowed to crawl the web by dannyw on Friday March 26, 2021 @11:08PM (#26599565)
Actually, yes! The current API is free. The new Enterprise API is paid.
Re:Only Google is really allowed to crawl the web by aabhay on Saturday March 27, 2021 @12:07PM (#26603290)
Sure! Apply to become a crawler. And if you meet certain criteria and your crawlers don’t exceed a quota then have at it. The key is not to make it technically challenging, but to erect a legal barrier.
Re:Only Google is really allowed to crawl the web by rincebrain on Friday March 26, 2021 @12:42PM (#26594163)
Wikimedia recently announced Wikimedia Enterprise for "organizations that want to repurpose Wikimedia content in other contexts, providing data services at a large scale".

So they're pretty clearly looking to monetize organizations which consume their data in a for-profit context.

Re:Only Google is really allowed to crawl the web by dathinab on Friday March 26, 2021 @02:31PM (#26595313)
monetizing != for-profit

You could e.g. just cover operational cost and/or improve the service quality from it.

Re:Only Google is really allowed to crawl the web by pmiller2 on Friday March 26, 2021 @02:48PM (#26595518)
I think they may have meant "(organizations) (which consume their data in a for-profit context)."
Re:Only Google is really allowed to crawl the web by dmitriid on Friday March 26, 2021 @12:41PM (#26594150)
Google was/is also the largest sponsor of Mozilla. This doesn't stop Google from sabotaging Mozilla.

2 mln is probably Google's hourly profit. For that they get one of the biggest knowledge bases in the world. It's basically free as far as Google is concerned.

The instant Google becomes confident they can supplant Wikipedia, they will.

Re:Only Google is really allowed to crawl the web by jedberg on Friday March 26, 2021 @01:51PM (#26594937)
> 2 mln is probably Google's hourly profit.

You don't have to guess, their numbers are public. In 2020 they made $40B in profit, so it takes them about 27 minutes to make $2M in profit.

Re:Only Google is really allowed to crawl the web by billiam on Friday March 26, 2021 @03:02PM (#26595652)
NOT a sponsor of Mozilla. Google buys web traffic (as default search engine) for ~$300M and turns it into several times that $ in ad revenue.
Re:Only Google is really allowed to crawl the web by kelnos on Friday March 26, 2021 @03:14PM (#26595787)
Not sure why you're being downvoted; I completely agree with what you're saying (modulo questionable usage of "sponsor"). If Wikipedia were to try to charge for this use of their data, Google would likely make it a priority to drop the Wikipedia blurbs, either without replacement, or with data sourced elsewhere.
Re:Only Google is really allowed to crawl the web by will4274 on Friday March 26, 2021 @06:43PM (#26597889)
> Google would likely make it a priority to drop the Wikipedia blurbs, either without replacement, or with data sourced elsewhere.

That's an odd way of phrasing things. If Wikipedia were to take away free access to their data, Google wouldn't be dropping Wikipedia, Wikipedia would be dropping Google. This line of thinking "you took this when I was giving it away for free, but now I want to charge for it, so you are expected to keep paying for it" is incorrect.

Re:Only Google is really allowed to crawl the web by zdragnar on Friday March 26, 2021 @03:59PM (#26596261)
Given the scale that google already operates at, I don't doubt that they would just take a copy of thr content and rebrand it as a google service, complete with user contribution.

Then, after two or five years, let it fester then abandon it. Nobody gets promoted for keeping well oiled machines running.

Re:Only Google is really allowed to crawl the web by dmitriid on Friday March 26, 2021 @05:44PM (#26597331)
Remember Knol? https://en.wikipedia.org/wiki/Knol?wprov=sfti1

It was actually good for writing stuff when I tried it. Never brought in enough traffic. Killed.

Re:Only Google is really allowed to crawl the web by magicalist on Friday March 26, 2021 @12:43PM (#26594179)
> Google was/is also the largest sponsor of Mozilla. This doesn't stop Google from sabotaging Mozilla.

Google isn't a sponsor of Mozilla, they're a customer. Do people think Google is "sponsoring" Apple with $1.5 billion a year too?

Re:Only Google is really allowed to crawl the web by foobarian on Friday March 26, 2021 @12:52PM (#26594271)
> they're a customer.

The cynic in me thinks the product is anti-trust insurance.

Re:Only Google is really allowed to crawl the web by saagarjha on Friday March 26, 2021 @08:59PM (#26598815)
$1.5 billion a year? You're off by an order of magnitude; the number is thought to be over $10 billion a year.
Re:Only Google is really allowed to crawl the web by dmitriid on Friday March 26, 2021 @02:37PM (#26595376)
Google being Apple's customer doesn't mean Google isn't sponsoring Mozilla.

These are two very different companies with a very different relationship with Google. And very different influences on Google.

Google wants to be on iOS. It brings customers to Google. A lot of them. iOS is possibly more profitable to Google than Android even with all the payments Apple extracts from them.

Google needs Mozilla so that Google may pretend that there's competition in browser space and that they don't own standards committees. The latter already isn't really true, and Google increasingly doesn't care about the former.

Re:Only Google is really allowed to crawl the web by tomp on Friday March 26, 2021 @02:22PM (#26595224)
Well then they can't nag users to donate to Jimmy Wales' trust fund.
Re:Only Google is really allowed to crawl the web by minikites on Friday March 26, 2021 @02:12PM (#26595132)
Couldn't you make a similar argument about for-profit uses of free/libre software? The software serves a useful purpose, who cares where it came from?
Re:Only Google is really allowed to crawl the web by onetimemanytime on Friday March 26, 2021 @03:59PM (#26596263)
>>Google donated $2M [1]. In 2010, Google also donated $2m [2].

$2 Million a year? Now I know why Googlers complained about having one less olive in their lunch salad.

How much does Google PROFIT from Wikipedia and how much does Wikipedia loses in fundraising when Google fails to send users to the info provider?

Re:Only Google is really allowed to crawl the web by lupire on Friday March 26, 2021 @05:49PM (#26597381)
Wikipedia is drowning in money so this whole line of discussion is weird.

And most of the value of wikipedia is created by its unpaid users, not Wikimedia foundation.

Re:Only Google is really allowed to crawl the web by magicalist on Friday March 26, 2021 @12:41PM (#26594158)
> You see this recently with Wikipedia. Google's widgets have been reducing traffic to Wikipedia pretty dramatically.

Wikipedia visitors, edits, and revenue are all increasing, and the rate that they're increasing is increasing, at least in the last few years. Is this a claim about the third derivative?

> Enough so that Wikipedia is now pushing back with a product that the Googles of the world will have to pay for.

The Wikimedia Enterprise thing seems like it has nothing to do with missing visitors and that companies ingesting raw Wikipedia edits are an opportunity for diversifying revenue by offering paid structured APIs and service contracts. Kind of the traditional RedHat approach to revenue in open source: https://meta.m.wikimedia.org/wiki/Wikimedia_Enterprise

Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @12:52PM (#26594273)
See https://searchengineland.com/wikipedia-confirms-they-are-ste... from 2015. Google's widgets that present Wikipedia data do reduce visitors to Wikipedia.

Or see page views on English Wikipedia from 2016-current: https://stats.wikimedia.org/#/en.wikipedia.org/reading/total... Looks pretty flat, right? Does that seem normal?

As for Wikimedia Enterprise, you do have to read between the lines a bit. "The focus is on organizations that want to repurpose Wikimedia content in other contexts, providing data services at a large scale".

Re:Only Google is really allowed to crawl the web by SamBam on Friday March 26, 2021 @01:06PM (#26594450)
The first link doesn't seem quite conclusive (see the part at the bottom), and also doesn't give evidence that Google's widgets are to blame.

The flattening of users could also be due to a general internet-wide reduction in long-form (or even medium-form) non-fiction reading. How are page views for The New York Times?

Seems like it should be simple to A/B test, though. Obviously Google could do it themselves by randomly taking away the widget, but would could also see whether referrals from non-Google search engines (though they are themselves a tiny percentage) continue to increase while Google remains flat.

Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @01:21PM (#26594623)
Edit: Removed bad "simple english graph", thanks. Though the regular english wikipedia traffic is flat from 2016-present.

As for NYT, is there a better proxy to compare to? There's no public pageview stats and they have a paywall.

Re:Only Google is really allowed to crawl the web by magicalist on Friday March 26, 2021 @01:31PM (#26594738)
That first graph is Simple English, not English, and is in millions, not billions. They also explicitly call out the methodology change in 2015...
Re:Only Google is really allowed to crawl the web by gtm1260 on Friday March 26, 2021 @03:59PM (#26596266)
I'm not sure I agree with this. I think airline websites are so garbage filled that they've driven people to use the simple alternative of the google flights checkout.

It's a bit of a vicious cycle, but In general most websites are so chock filled with crap that not having to click into them for real is a relief!

Re:Only Google is really allowed to crawl the web by cyberpunk on Friday March 26, 2021 @04:39PM (#26596663)
BA had some tracking request inline on the “payment processing” page which when blocked by my pihole prevents me from ever getting to the confirmation page, just have to refresh your email and wait for the best.

I have no idea how these companies, which make quite a decent amount of money at least up until 2020, can have such utterly poor sites.

I once counted some 20+ redirects on a single request during this process heh..

Re:Only Google is really allowed to crawl the web by bombcar on Friday March 26, 2021 @05:31PM (#26597196)
I don’t know what they’re doing but most every single sign on tool I’ve seen redirects 10-20 times during the sign on process (and then dumps you to the homepage to navigate your way back).
Re:Only Google is really allowed to crawl the web by merlinscholz on Friday March 26, 2021 @05:47PM (#26597359)
Probably to get first party cookies on a handful of domains
Re:Only Google is really allowed to crawl the web by creato on Friday March 26, 2021 @04:19PM (#26596462)
Yeah, the Google flights issue is difficult. On one hand, the business practice is problematic. On the other hand, Google flights is so much better than its competitors it's ridiculous.

If there was a way to split Google flights into a separate company and somehow ensure it wouldn't devolve into absolute trash like its competitors, that would be a good thing.

Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @04:28PM (#26596551)
It was ITA and prior to Google buying them, did a pretty good business selling backend flight shopping services to aggregators and airlines.

Shopping for flights is a surprisingly technically difficult thing to do well.

Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @04:17PM (#26596434)
I'm talking about flight status. Not Google Flights, shopping, or booking.

There are events associated with flight status that Google doesn't know. Like change fee waivers, cash comp awards to take a later or earlier flight, seat upgrades, etc.

Re:Only Google is really allowed to crawl the web by gxs on Friday March 26, 2021 @04:11PM (#26596367)
It's not Google's prerogative to scrape a website and display its content, no matter how awful the website.
Re:Only Google is really allowed to crawl the web by michaelmrose on Friday March 26, 2021 @04:23PM (#26596497)
If 1 airline let me view information in a friendly fashion and the other didn't I would do business with the first.

Lest we forget the money in that scenario is from butts in seats not clicks on a website. The particular example is ill chosen as google is actually taking on a cost, taking nothing, and gifting the airline a better ui.

Re:Only Google is really allowed to crawl the web by BEEdwards on Friday March 26, 2021 @04:25PM (#26596518)
If you make an awful website that can be scrapped it's a matter of when not if someone will take your data and give it to your consumers whether your trying to upsell them or not...
Re:Only Google is really allowed to crawl the web by treis on Friday March 26, 2021 @12:18PM (#26593872)
>However, that sort of search used to (most times) lead to a visit to the airline web site.

I don't think that's correct. In the old days you'd either call a travel agent or use an aggregator like expedia.

Google muscles out intermediaries like Expedia, Yelp, and so on. It's not likely much better or worse for the end user or supplier. Just swapping one middleman for another.

Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @12:20PM (#26593895)
I can't prove it was that way, but I spent a lot of time in the space. For a long time, the airline's site used to be the top organic result, and there was no widget. Similar for other travel related searches (not just airlines) over time. Google has been pushing down organic results in favor of ads and widgets for a long time...and slowly, one little thing at a time. Like no widgets -> small widget below first organic result -> move the widget up -> make it bigger -> etc.
Re:Only Google is really allowed to crawl the web by supernovae on Friday March 26, 2021 @01:03PM (#26594420)
I don't think google muscling out intermediaries like Expedia is a good thing.

Just for example, Expedia is probably 5% of Google's total revenue and Google doesn't like slim margin services by and large that can't be automated.

Travel is fairly high-touch - people centric. It doesn't fit Google's "MO".

But... its shitty that google can play all sides of the markets while holding people ransom to mass sums of money to pay to play on PPC where google doesn't... i think that's where the problem shines.

In essence, you're advocating that eBay goes away because google could do it... they could.. and eBay is technically just an intermediary, but do we want everything to be googlefied?

Google bought up/destroyed other aggregators - remember the days of fatwallet, priceline, pricewatch, shopzilla and such when they used to focus on discounts/coupons/deals and now they're moving more towards rewards/shopping/experience - it used to be i could do PPC on pricewatch and reach millions of shoppers are a reasonable rate, but now that google destroyed them all, the PPC rate on "goods" is absurdly high and not having an affordable market means only the amazons and walmarts can really afford to play...

it used to be you could niche out, but even then, that's getting harder

Re:Only Google is really allowed to crawl the web by treis on Friday March 26, 2021 @01:54PM (#26594968)
>In essence, you're advocating that eBay goes away because google could do it... they could.. and eBay is technically just an intermediary, but do we want everything to be googlefied?

I don't think I'm really advocating for it as much as I see as a more or less neutral change.

That said, I'm pretty ambivalent about Google. Their size is a concern, but they also tend to be pretty low on the dark pattern nonsense. eBay, to use an example you gave, screwed me out of some buyer protection because of poor UX and/or bug (I never saw the option to claim my money after the seller didn't respond). In this specific instance Google ends the process by sending you to the airline to complete the booking. That, imho, is likely better than dealing with Expedia.

Re:Only Google is really allowed to crawl the web by supernovae on Friday March 26, 2021 @05:49PM (#26597382)
Companies opt in to sites like Expedia and list their properties/flights/vacations on their marketplace and they pay a commission for those being booked. Expedia doesn't just crawl them and demand a royalty for sending them traffic...

Google has a huge pay 2 play problem with PPC... i've worked for Expedia so that's the only reason i know this :)

It's the reason companies work with Expedia many times because they don't have the leverage expedia group does...

i see it as unnatural change btw... "borg" if you will.

Re:Only Google is really allowed to crawl the web by darkwater on Friday March 26, 2021 @05:02PM (#26596904)
It's actually pretty different because another middleman can basically arise only if it's a big success in the iOS App Store because coming up in Google searches would be impossible and more or less the same in the Play Store. So, Google is not just yet another intermediary.
Re:Only Google is really allowed to crawl the web by josefx on Friday March 26, 2021 @12:22PM (#26593935)
Only if Google stays around long term. I wouldn't be surprised if each free product on its graveyard took down a dozen of competing products before it was killed of.
Re:Only Google is really allowed to crawl the web by pc86 on Friday March 26, 2021 @12:26PM (#26593970)
Then someone can start a competitor up again, right? Assuming there's actually a market for it.
Re:Only Google is really allowed to crawl the web by josefx on Friday March 26, 2021 @12:44PM (#26594181)
Not every market is lucrative in the extreme and it can take a long time to recover from being "disrupted". I think it is also a common practice for larger shopping chains to dump prices when they open a new location in order to clear out the local competition, so the damage it causes is well understood to be long lasting.
Re:Only Google is really allowed to crawl the web by bouncycastle on Friday March 26, 2021 @12:14PM (#26593823)
In regards to airlines, Google and Amadeus have a partnership I believe. Amadeus is the main source of data for many of these airline websites. If Google gets the data from Amadeus directly and not these websites, they are just cutting out the middleman. I don't shed a tear for any of these middleman (together with their Dark Pattern UX design).
Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @12:15PM (#26593835)
Amadeus isn't a source of flight status. It is a source for (some) planned schedules and fares. Global distribution systems are a complex topic that's hard to sum up on HN. For flight status, Google is pulling from OAG and Flight Aware, and also from airline websites. Though they don't show airline sites as a source.
Re:Only Google is really allowed to crawl the web by lelanthran on Friday March 26, 2021 @05:55PM (#26597446)
> And I don't think Google will realize what the problem is until they start accidentally killing off large swaths of the actual sources of this content by taking the audience away.

What makes you think they care? Killing off the sources of content might even be there goal. If they kill off sources of content, they'd be more than happy to create an easier-to-datamine replacement.

Hypothetically, if they killed off wikipedia, they are best placed to use the actual wikipedia content[1] in a replacement, which they can use for more intrusive data-mining.

Google sells eyeballs to advertisers; being the source of all content makes them more money from advertisers while making it cheaper to acquire each eyeball.

[1] AFAIK, wikipedia content is free to reuse.

Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @09:04PM (#26598855)
I'm not suggesting it's illegal. There are a great many practices that are legal that I dislike.
Re:Only Google is really allowed to crawl the web by jeffbee on Friday March 26, 2021 @12:27PM (#26593982)
You're wrong on a lot of facts here. Google Flights doesn't get its data just by crawling, they get it from Sabre, the FAA, Eurocontrol, etc. Airlines are, obviously, extremely pleased to disseminate this information. Google Flights "gives back" in the exact same way as any other travel outlet: they book passengers.

As for Wikipedia, the WMF is quite happy that most of their traffic is now served by Google. WMF is in the business of distributing knowledge, not in the eyeballs business. Serving traffic is just a cost for them. The main problem has been that the average cost for Wikipedia to serve a page has gone up, because many readers read it via Google, and more people who visit Wikipedia are logged-in authors, which costs them more to serve. I'm sure there's an easy solution to this problem (for example, beneficiaries of Wikipedia can donate compute facilities and services, or something along those lines).

Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @12:35PM (#26594075)
They don't get individual flight status (what I was talking about) from Sabre or the FAA or Eurocontrol. I didn't get into fares and planned schedules and Google Flights, that's a different topic. I was talking about the big widget you get for queries on status for a particular flight, which is not Google Flights.

They have relented in some ways, rolling out stuff in the widget like: "The airline has issued a change fee waiver for this flight. See what options are available on American's website"

But obviously, that kind of stuff isn't shown on Google for quite some time after it exists on the source site. And the widget pushes the organics off the fold unless you have a huge monitor.

As for Wikipedia, I was referring to this: https://news.ycombinator.com/item?id=26487993

"Airlines are, obviously, extremely pleased to disseminate this information"

In the same way that publishers love AMP, yes. They don't actually like it, but they are forced to make the best of it.

Re:Only Google is really allowed to crawl the web by jeffbee on Friday March 26, 2021 @01:06PM (#26594436)
Oh, status. I was thinking of schedules. Still, what is the point for the consumer of being directed to an airline's terrible status page? And are they even capable of being crawled? Looking at American's site (it was the most ghastly airline that sprang to mind) I don't see how a crawler would be able to deal with it, and indeed the Google snippet for AA flight status, on the aa.com result which is far down in the results page, just says "aa.com uses cookies" which is about what you'd expect.

In this case, I want to be sent literally anywhere but aa.com.

Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @01:30PM (#26594715)
"what is the point for the consumer of being directed to an airline's terrible status page?"

One example...

If you back up a bit, the widget didn't used to tell you there was a change fee waiver when the flight was full, while aa.com did.

That's an actual, tangible benefit that a consumer might want, worth real money. You can also even often "bid" on a dollar amount to receive if you're willing to change flights. Google doesn't present that info today.

There are more examples. My perspective isn't that Google should lead you to aa.com, but I do feel it's a bit dishonest that the widget is so large it pushes aa.com below the fold. It doesn't need to be that large.

Re:Only Google is really allowed to crawl the web by benatkin on Friday March 26, 2021 @12:02PM (#26593676)
That's the result of the crawling, and it preventing competition. Google would much prefer that people complain about the details while ignoring the root cause.
Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @12:04PM (#26593706)
I don't understand that. The crawling access is mostly the same as it ever was. Google's SERP pages are not. A mutually beneficial search engine that respects it's sources would still crawl the same. Google used to be that.

The core problem is incentives: http://infolab.stanford.edu/~backrub/google.html "we believe the issue of advertising causes enough mixed incentives that it is crucial to have a competitive search engine that is transparent and in the academic realm."

Re:Only Google is really allowed to crawl the web by benatkin on Friday March 26, 2021 @12:08PM (#26593756)
That's incorrect. Before the search oligopolies formed, new search engines could start up. There was excite, hotbot, altavista, and more. Now they don't have access. Search these comments for census.gov.
Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @12:11PM (#26593802)
There are companies that do pretty well in this space, like ahrefs, for example. They do resort to trickery, like proxy clients that look like home computers or cell phones. But, if a small entity like ahrefs can do it, anyone can do it.

In a nutshell, though, I don't see equal access for all crawlers changing anything. Maybe that's the first barrier they hit, but it isn't the biggest or hardest one by far. Bing has good crawler access, but shit market share.

Re:Only Google is really allowed to crawl the web by veltas on Friday March 26, 2021 @12:23PM (#26593948)
I swear something like 50% of those digests are totally incorrect as well. It's amazing they have kept the feature because it has never had a very high signal-to-noise ratio. I never trust what's presented in these digests without double-checking the source page.
Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @12:30PM (#26594019)
I remember when rich snippets (one type of those widgets) came out there were a lot of funny examples. One for a common query about cancer treatments that pulled data from a dodgy holistic site saying that "carrots cured most types of cancer" (or something like that).

There was a similar one where Google emphatically claimed a US quarter was worth five cents in a pretty and large snippet graphic.

Re:Only Google is really allowed to crawl the web by BeFlatXIII on Friday March 26, 2021 @02:23PM (#26595230)
The most memorable rich snippet humor I've seen is a horse breeder sharing a story of how her searches gave snippets with my little ponies as the preview image.
Re:Only Google is really allowed to crawl the web by Mauricebranagh on Friday March 26, 2021 @12:58PM (#26594338)
I recall in the last uk election google got the infographic of party leaders about 60-70% wrong.

And quite often a people also ask refine is just some random guys comment from redit.

Re:Only Google is really allowed to crawl the web by bombcar on Friday March 26, 2021 @05:41PM (#26597300)
Have you heard the story of Thomas Running? It’s a story Google will tell you.

(Search who invented running)

Re:Only Google is really allowed to crawl the web by Camillo on Friday March 26, 2021 @08:57PM (#26598804)
> The airline web site could then present things Google can't do. Like "hey, we see you haven't checked in yet" or "TSA wait times are longer than usual" or "We have a more-legroom seat upgrade if you want it".

If I'm a passenger, there's plenty of ways for airlines to notify me. If I'm searching for a flight status online, it's because I'm picking someone up. If I want more information, I'll click through. I don't see how either me or the airline are hurt.

> You see this recently with Wikipedia. Google's widgets have been reducing traffic to Wikipedia pretty dramatically. Enough so that Wikipedia is now pushing back with a product that the Googles of the world will have to pay for.

Why is that even a problem for Wikipedia?

Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @09:11PM (#26598907)
It's a progression. It's not a huge problem, by itself, for either. But Google shareholders want to continue the same YoY gains. The only cash cow is search, so they continue to take screen real estate that used to go to others, and take it for themselves. Whether that's more ads, or more widgets, or whatever.

Yes, it's legal. But it does reduce visitor interactions for those sites. Reduced visitor interactions isn't good for web sites...it takes away incentives, reduces brand value, reduces revenue. Eventually, that is not great for consumers.

Ever been a middleman? Squeeze your suppliers enough, and you kill them. The next supplier will pre-emptively cut quality, features, etc, because they know you're going to try and squeeze them to death.

"If I'm searching for a flight status online, it's because I'm picking someone up"

That's one use case, it's not all of them. There's middle ground too, like the bones they throw Wikipedia in the form of links for more info.

Re:Only Google is really allowed to crawl the web by plumeria on Friday March 26, 2021 @08:35PM (#26598650)
> You see this recently with Wikipedia. Google's widgets have been reducing traffic to Wikipedia pretty dramatically. Enough so that Wikipedia is now pushing back with a product that the Googles of the world will have to pay for.

Do you have a link of that product/service from Wikipedia?

Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @09:01PM (#26598825)
Re:Only Google is really allowed to crawl the web by plumeria on Saturday March 27, 2021 @12:41AM (#26599977)
Thanks
Re:Only Google is really allowed to crawl the web by JKCalhoun on Friday March 26, 2021 @12:44PM (#26594185)
> In short, I don't think the crawler is the problem.

Except that, allow other companies to crawl/compete, and you can take eyeballs away from Google (which may well then return eyeballs to Wikipedia so long as the Google competitors don't also present scraped data).

Re:Only Google is really allowed to crawl the web by ChrisArchitect on Friday March 26, 2021 @02:48PM (#26595514)
They're making it easier to search for flights and arrange a trip. It's UX and makes me not hate the airlines/travel process as much. And I end up buying the flight from the airline anyways, and in many cases doing the arranging on the airline site in the end once it's determined, so Google is giving that back. They're not taking stuff from the airlines, I mean what ads and stuff are on the airline sites anyways specifically during the search process. Where they are taking away is from the Expedia's and other aggregation sites that offer a garbage/hodgepodge experience that drives people crazy.
Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @03:32PM (#26595992)
You're talking about Google Flights, which is completely unrelated to flight status.
Re:Only Google is really allowed to crawl the web by ilaksh on Friday March 26, 2021 @05:31PM (#26597193)
The way that the web has been fundamentally broken by Google and other companies is one of the reasons I am excited about an alternative protocol called Gemini. It doesn't replace the web entirely, but for basic things like exchanging information, it's great. https://gemini.circumlunar.space/
Re:Only Google is really allowed to crawl the web by nathanh4903 on Saturday March 27, 2021 @02:10PM (#26604147)
>Google's widgets have been reducing traffic to Wikipedia pretty dramatically.

But wouldn't this be a good thing? Since wikipedia is a nonprofit aiming to provide knowledge, google stealing & caching their content might help serve information to more while reducing wikipedia's server load, so IMO it might not be so bad for wikipedia.

Re:Only Google is really allowed to crawl the web by Siira on Friday March 26, 2021 @05:48PM (#26597368)
Large swaths of web are garbage. Wasting people's time and attention on visiting pointless sites for something presentable in a small box is obviously not economical.

And if some of the sources somehow die? New sources will spring up. It doesn't matter.

Re:Only Google is really allowed to crawl the web by nextaccountic on Saturday March 27, 2021 @01:14AM (#26600100)
> You see this recently with Wikipedia. Google's widgets have been reducing traffic to Wikipedia pretty dramatically. Enough so that Wikipedia is now pushing back with a product that the Googles of the world will have to pay for.

What product is that?

Re:Only Google is really allowed to crawl the web by stjohnswarts on Friday March 26, 2021 @08:20PM (#26598556)
I think the flight arrivals/departures is a bad example. A good example might be putting flights.google.com on the first page or even allow it to exist.
Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @09:06PM (#26598866)
Not sure what you mean. Both for flight status as well as flight shopping, Google drops a huge widget at the top and pushes everything else down, below the visible fold.
Re:Only Google is really allowed to crawl the web by 838812052807016 on Friday March 26, 2021 @05:29PM (#26597181)
Standardized interoperability enables overall progress.

Every airline doesn't need their own webpage. They could all provide a standard API.

Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @07:16PM (#26598135)
"Every airline doesn't need their own webpage. They could all provide a standard API."

That's sort of how it works in the corporate booking tool world. It is decidedly not a better experience for end users, IMO.

There's quite a lot about each airline that is different, so any unified approach is a lowest common denominator. You'll notice things like loyalty points, for example, have more rich data on the airline's website. And that some fares are ONLY on the website. Or that seat maps have more useful detailed info, etc.

And that's all shopping/booking. Departure control, flight status, upgrade/downgrade, check-in, seat upgrades, standby, etc, are for the most part only on the airline's website.

Re:Only Google is really allowed to crawl the web by dan-robertson on Friday March 26, 2021 @12:15PM (#26593840)
The way to look at this from Google’s point of view is to realise that most websites are slow and bad[1], so if Google sent you there you would have a bad experience with a bad slow website trying to find the information you want. Google want to make it better for you.

[1] it feels like Google have contributed a lot to websites being slow and bad with eg ads, amp, angular, and probably more things for the other 25 letters of the alphabet.

Re:Only Google is really allowed to crawl the web by zentiggr on Friday March 26, 2021 @02:48PM (#26595515)
> Google want to make it better for you.

Hehe, sure, nothing nefarious or greedy here... move along, move along, nothing to see...

Re:Only Google is really allowed to crawl the web by devoutsalsa on Friday March 26, 2021 @03:28PM (#26595944)
I've noticed that sometimes Google had updated flight information before the displays at the airport.
Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @03:47PM (#26596149)
For the most part individual airports own that infrastructure. So it's hard to generalize. For most types of notable flight status/time changes, however, airlines usually know first.

There are exceptions, like an airport-called ground stop.

Re:Only Google is really allowed to crawl the web by wbl on Friday March 26, 2021 @05:17PM (#26597061)
Does the concergie of a hotel take anything away when he informs you that your flight has been delayed?
Re:Only Google is really allowed to crawl the web by tyingq on Friday March 26, 2021 @08:30PM (#26598612)
It's hard for me to make that an apt analogy. She's not well known as a portal to find websites, which is what Google had been for most of its existence.

It's pretty difficult to come up with a non-computer analogy for how Google works now. Pick a different space, and the power imbalance is quite clear. If they wanted, they could destroy StackExchange very quickly with these widgets.

Re:Only Google is really allowed to crawl the web by mdaniel on Saturday March 27, 2021 @12:03AM (#26599809)
Only if you are presuming they are going to make a question answering widget, too, since the content on stackexchange doesn't materialize out of thin air
Re:Only Google is really allowed to crawl the web by supert56 on Friday March 26, 2021 @01:00PM (#26594364)
Perhaps I am misunderstanding or over simplifying things but it always surprises me that there are legal cases brought against companies who scrape data when so many of Google's products are doing exactly this.

It definitely feels like one set of rules for them and a different set for everyone else.

Re:Only Google is really allowed to crawl the web by Spivak on Friday March 26, 2021 @02:00PM (#26595026)
I mean it's not that weird that a company would authorize major search engines scraping them but no one else.

I don't really see this as Google playing by different rules so much as economic incentives being aligned in Google's favor.

Re:Only Google is really allowed to crawl the web by lupire on Friday March 26, 2021 @05:53PM (#26597426)
Google doesn't scrape anything that the site owner objects to.