News for hackers. Stuff that matters.
Re:Google's fact-checking bots build vast knowledge bank by ck2 on Friday August 22, 2014 @12:27PM (#8212772)
Isn't it nice that millions of people made web pages that Google decided to scrape to harvest the work of others and run ads next to it for themselves?

Now try scraping Google and see what they do to you.

Re:Google's fact-checking bots build vast knowledge bank by dm2 on Friday August 22, 2014 @01:17PM (#8212980)
You can ban GoogleBot easily, just put a line in your robots.txt file, but then people won't be able to easily find your site using Google services.

If you provide value to Google they will make an API to allow accessing that data easier.

By scraping do you mean scraping their search results? They offer this, which is nice: https://developers.google.com/custom-search/

Many large sites don't allow scraping because of unnecessary server load (denial of service sometimes) so they'll offer an API where you can download content in a controlled (and monitorable) manner.

Re:Google's fact-checking bots build vast knowledge bank by frik on Friday August 22, 2014 @01:34PM (#8213070)
We need a robots.txt and "noindex" metatag standard for emails.

If one sends an email to an GMail/Outlook.com/Yahoo email address, one should be able to opt-out of their email crawler, advertisement analysis, artificial intelligence analysis, etc.

Re:Google's fact-checking bots build vast knowledge bank by dm2 on Friday August 22, 2014 @01:46PM (#8213125)
The user who receives it can: https://support.google.com/ads/answer/2662922?hl=en

I don't think they do too much storing of email details, they know that it's a sensitive area and that an employee will eventually blow the whistle and it will hurt user trust, which is a big part of their business.

Re:Google's fact-checking bots build vast knowledge bank by frik on Friday August 22, 2014 @01:59PM (#8213181)
I meant it the other direction.

1) Alice sends an email to Bob (GMail user).

2) Bob receives the email, meanwhile Google scrapes the content and extracts its meaning to show Bob some ads and use the facts to improve Google's A.I.

Alice wants a way to mark her email content as "no index".

So that email service provider don't crawl through the content. Exactly like the robots.txt for domains or the "noindex" metatag in HTML head element!

Re:Google's fact-checking bots build vast knowledge bank by lern_too_spel on Friday August 22, 2014 @06:45PM (#8214285)
Why should Alice get to restrict what Bob can do with his inbox?
Re:Google's fact-checking bots build vast knowledge bank by frik on Saturday August 23, 2014 @01:38AM (#8215108)
It's not about Bob, he can do what he want.

It's about the email service provider, that should stop analyze the email text to extract its meaning. Gmail uses it to display ads to Bob, builds a shadow profile for Alice (like Facebook) and trains an artificial intelligence (see headline link).

Re:Google's fact-checking bots build vast knowledge bank by rstupek on Friday August 22, 2014 @05:19PM (#8214018)
It would be nice if there were a way to provide google, bing, et. al. the data so they were constantly crawling sites. A non-insignificant amount of our bandwidth is used up daily by search engines spidering sites.
Re:Google's fact-checking bots build vast knowledge bank by vijayr on Friday August 22, 2014 @12:36PM (#8212803)
Those millions of people want google to scrape and harvest, in the hope that they will rank higher etc etc.

If an unknown person tries to scrape, he/she will promptly get banned by those very same people (Google wouldn't like someone scraping their stuff either).

Different players different rules, I guess.

Re:Google's fact-checking bots build vast knowledge bank by yutah on Friday August 22, 2014 @01:10PM (#8212950)
Which large website did you get banned from for scraping? I did some scrapping and never got banned... perhaps my scraping's rate was not too fast.
Re:Google's fact-checking bots build vast knowledge bank by adventured on Saturday August 23, 2014 @05:47AM (#8215409)
If you try to mass scrape almost any major site (millions of pages of content) they'll block you.

For example, if you went one by one through Stack Overflow and sucked out every question and answer, your scraper bot would get banned (unless you're doing one request per minute, in which case you'll never finish).

Or if you tried to scrape Twitter.