News for hackers. Stuff that matters.
Re:Matt Cutts is looking for scraper sites by VikingCoder on Friday February 28, 2014 @07:40AM (#7318380)
Scrapers lift the full content, wholesale, without attribution.

You may as will just show http://images.google.com and complain that it's scraping. Or http://news.google.com.

In general, do you think Wikipedia gets more traffic because Google exists, or do you think Google gets more traffic because Wikipedia exists? Meaning, which affect is larger? I'm pretty sure the answer to this is obvious.

And if more scrapers donated millions to the site they scrape from, the world would be a much better place.

http://wikimediafoundation.org/wiki/Press_releases/Wikimedia...

Re:Matt Cutts is looking for scraper sites by josefresco on Friday February 28, 2014 @08:08AM (#7318494)
One man's "scrapper" is another man's "aggregator".

How do you think Google would view my site if I wrapped Wikipedia's content, with back link and ran my own ads alongside that content? I would imagine not very positively.

Also, is it okay that a bigger entity scrapes my content just because they send me traffic? You might not want to bite the hand that feeds you, but it still doesn't make it right.

Re:Matt Cutts is looking for scraper sites by ds9 on Friday February 28, 2014 @10:06AM (#7319037)
Google does not reproduce whole articles, only short excerpts to help searchers decide whether it's relevant to what they're looking for - and with clear indication of the source and in a context where it's understood that Google is showing the blurb only to pointing to the source where it was found.

This is technically scraping but it's hardly comparable to the bottom-feeders that plagiarize for money. (Edit: according to 'pud' on this page, Google uses a Wikipedia index so it's not scraping, but it is in the case of other sites that Google indexes.)

And yes, it's OK both legally and ethically if you do the same to Wikipedia - like Google that is, just for indexing purposes and not using whole articles.

Re:Matt Cutts is looking for scraper sites by Silhouette on Friday February 28, 2014 @12:20PM (#7320091)
Google does not reproduce whole articles, only short excerpts to help searchers decide whether it's relevant to what they're looking for

And what about, for example, Google's image search tool, where the image itself might be what their user is searching for, and where Google controversially changed their system a little while ago to show full-size images in-SERP and de-emphasize forwarding search users to the original source? Or Google Cache, if it's reproducing material that has since been taken down deliberately from the original source?

To add insult to injury, some Google services still appear to rely on the original source's bandwidth to serve things like images (not to mention avoiding a certain legal argument about copyright infringement), thus violating the basic principle of netiquette that has been good manners ever since people actually used the word netiquette that you don't hotlink other people's stuff on your site.

Re:Matt Cutts is looking for scraper sites by josefresco on Friday February 28, 2014 @10:40AM (#7319304)
You're comparing what Google does to another extreme when you say things like "bottom-feeders that plagiarize for money"

Surely you don't believe that all "scapers" are bottom feeders? It's like saying every criminal is a murderer. There's a whole bunch of grey area in between, and this is where the criticism of Google's harsh penalties is valid.

Re:Matt Cutts is looking for scraper sites by AJ007 on Friday February 28, 2014 @11:39AM (#7319791)
You are at least partially incorrect.

Last year Google was testing reproducing entire Wikipedia articles within their site for their mobile site. You could read the full article within going to Wikipedia (allowed by Creative Commons, of course.) Between that and what they did with Google Images, I would say this reveals intention and is the direction web publishers should expect Google to be headed in.

In order for Google to continue to meet their growth targets they must increase the percentage of outgoing click from free to paid.

Re:Matt Cutts is looking for scraper sites by ezequiel-garzon on Friday February 28, 2014 @09:33AM (#7318836)
The de facto standard robots.txt is pretty likely to be respected by Google, so it's fairly easy to stop their scraping your site. Yes, it is opt-out, bit I'd expect it to be.

It may be quite frustrating for an upstart to be denied access while Google is explicitly allowed, but that's another matter.

Re:Matt Cutts is looking for scraper sites by zone411 on Friday February 28, 2014 @02:13PM (#7320816)
Wikipedia's top rankings are actually a big problem. I know of a site that was the first to put up high-quality reference-type content on the Web and for a while getting reasonable traffic from Google. Wikipedia's editors copied that content into thousands of articles in various ways. Thousands with attribution or copying just the facts and thousands without and copying more than just the facts.

This original site is now getting so little traffic from Google that more people visit it from the trickle of these bottom-of-the-page Wikipedia links than from Google itself. Its traffic was also badly hurt by Google's Panda algorithm, which I think clearly proves how flawed it is since this algorithm was supposed to do the exact opposite.

Because of this situation, if somebody thinks of spending money to create high quality reference-type content, I would strongly advise against it. You have no chance vs. Wikipedia's poorly-written articles repurposing your content and Google's flawed algorithms.

Re:Matt Cutts is looking for scraper sites by lisper on Friday February 28, 2014 @02:29PM (#7320905)
It seems a bit odd for you to be so cagey about the identity of this "original site" while at the same time lamenting that they aren't getting the traffic they deserve. Why don't you tell us who they are?
Re:Matt Cutts is looking for scraper sites by zone411 on Friday February 28, 2014 @02:54PM (#7321043)
It's because I don't speak for the owners of the site and I'd rather make sure they don't mind me putting it out there like this. I could let you know privately, if you'd like to check my story for yourself, though.
Re:Matt Cutts is looking for scraper sites by lisper on Friday February 28, 2014 @07:53PM (#7322559)
Why on earth would they mind?
Re:Matt Cutts is looking for scraper sites by zone411 on Friday February 28, 2014 @08:55PM (#7322739)
I'm not sure if they do mind. I do know that their relationship with Google is important to them when it comes to their much larger and more successful projects and that this site has been mostly left behind, so they may not want to bring it up in the context of this Hacker News post, even in the unlikely case that it resulted in this site getting its traffic back. Why not just email me and I'll show you a simple content site with minimal traffic, not using any black or gray-hat SEO tactics, with high-quality, original (to the Web) content, referenced in thousands of Wikipedia articles and you can decide for yourself if my post was truthful.
Re:Matt Cutts is looking for scraper sites by dennisgorelik on Saturday March 01, 2014 @11:25PM (#7327701)
Could you get permission from the owners and publish it here?
Re:Matt Cutts is looking for scraper sites by danielbarla on Friday February 28, 2014 @07:54AM (#7318440)
That's one way of looking at it, on the other hand, they link to the original URL, passing traffic back to the original source. Most "scraper" sites take the content, wrap it in their own similar outer layer, and try to take ad revenue. E.g. I've seen my own StackOverflow answers copied, word for word, to a scraper site and presented under a made-up name.
Re:Matt Cutts is looking for scraper sites by dangrossman on Friday February 28, 2014 @08:18AM (#7318532)
StackOverflow actually allows this; all their data is Creative Commons licensed, and they publish the full database dump on the Internet Archive.

https://archive.org/details/stackexchange

Re:Matt Cutts is looking for scraper sites by jbinto on Friday February 28, 2014 @08:23AM (#7318548)
Do the terms of the license allow for this kind of abuse?

Just because something is CC doesn't mean you can do whatever you want with it.

Re:Matt Cutts is looking for scraper sites by dangrossman on Friday February 28, 2014 @08:29AM (#7318577)
Yes, they do; it's not abuse when you're given explicit permission. CC BY-SA means you can do whatever you want with it as long as you attribute the source as specified.
Re:Matt Cutts is looking for scraper sites by leephillips on Friday February 28, 2014 @08:49AM (#7318664)
"as long as you attribute the source"

danielbarla said that they presented the material under a false name; this goes beyond copying and becomes plagiarism, which I can't imagine is an intended result of the CC license.

Re:Matt Cutts is looking for scraper sites by aroch on Friday February 28, 2014 @09:31AM (#7318829)
Is the source 'User X' or 'StackOverflow'? When you reference CC BY-SA code you don't reference the people who, say, checked it into git but rather the whole repo.
Re:Matt Cutts is looking for scraper sites by Flimm on Friday February 28, 2014 @12:22PM (#7320098)
CC BY-SA is short for Creative Commons Attribution Share-Alike. BY means you must attribute, and SA means you must license any distributed derivative works under the same license (copyleft). Attribution on its own is not enough.
Re:Matt Cutts is looking for scraper sites by grey-area on Friday February 28, 2014 @10:20AM (#7319147)
No, attribution is required.
Re:Matt Cutts is looking for scraper sites by jliptzin on Friday February 28, 2014 @09:36AM (#7318848)
Interesting, from the file sizes you can quickly gauge the relative popularity of each subject.
Re:Matt Cutts is looking for scraper sites by tobehonest on Friday February 28, 2014 @07:58AM (#7318456)
By having a tl;dr about the actual Wikipedia page, there is no need for the user to click on the link. Following what you're saying, Google as wrapped it in their own layer, and trying to take ad revenue.
Re:Matt Cutts is looking for scraper sites by smoyer on Friday February 28, 2014 @08:28AM (#7318568)
Actually, I find that having a tl;dr will rarely answer the question(s) I have on a topic, but it will commonly show me whether I've found the right wikipedia page. I usually either click-through or refine my search.
Re:Matt Cutts is looking for scraper sites by bushido on Friday February 28, 2014 @09:36AM (#7318850)
They don't actually link to the wikipedia URL. They mask a link that leads to another Google page "/url?sa=t&rct=j&q=&...." which in turn responds with a 200 OK page that redirects to Wikipedia.

Sure it passes the keywords etc. But this likely reduces the number of people visiting Wikipedia, while increasing Google's ad revenues, if anyone but Google did this they'd be potential blacklisted by Google.

Re:Matt Cutts is looking for scraper sites by VikingCoder on Friday February 28, 2014 @10:43AM (#7319320)
Actually, they do link to the wikipedia URL.

href="http://en.wikipedia.org/wiki/Scraper_site" appears directly in the source code of that web page.

It also has an onmousedown handler that rewrites the URL to point at Google, so they can tell which link you clicked, to improve their ranking system. And Google works very closely with sites to make sure the sites know how to understand the referrals.

Re:Matt Cutts is looking for scraper sites by jpeterson on Friday February 28, 2014 @07:53AM (#7318433)
Relax, it's a joke.
Re:Matt Cutts is looking for scraper sites by MWil on Friday February 28, 2014 @07:48AM (#7318413)
If Google only needs to visit Wikipedia's "scraper" page once a day or less, but serves it out to others with attribution, isn't that helping Wikipedia by lowering traffic COSTS?
Re:Matt Cutts is looking for scraper sites by baldfat on Friday February 28, 2014 @07:50AM (#7318420)
BUT it gives full attribution to http://en.wikipedia.org/wiki/Scraper_site???
Re:Matt Cutts is looking for scraper sites by DanBC on Friday February 28, 2014 @07:53AM (#7318436)
That is what paren comment says.