News for hackers. Stuff that matters.
Re:Wikimedia is moving to Gitlab by jefftk on Wednesday October 28, 2020 @12:19PM (#24920598)
Can you say more about how it's a problem if people can view things without logging in? Naively I would have seen that as a plus.
Re:Wikimedia is moving to Gitlab by dfabulich on Wednesday October 28, 2020 @12:44PM (#24920943)
If you allow new users to create user profiles with links, and those user profiles are visible to Google, spammers will create a bunch of new user accounts and fill them with spam links.

The easiest way to prevent this is to block Google from seeing user profiles by requiring login to see the profiles.

Re:Wikimedia is moving to Gitlab by chrisweekly on Wednesday October 28, 2020 @01:37PM (#24921720)
> "and those user profiles are visible to Google"

googlebot adheres to robots.txt, right?

in which case couldn't self-hosted gitlab admins add a robots.txt entry for the profile page url?

Re:Wikimedia is moving to Gitlab by jefftk on Wednesday October 28, 2020 @02:21PM (#24922276)
That requires the spammers to notice that it's blocked in robots.txt, which seems optimistic
Re:Wikimedia is moving to Gitlab by pitay on Wednesday October 28, 2020 @05:28PM (#24924333)
There is also adding the rel="nofollow ugc" to user submitted links that removes the benefits of linking for spammers.

https://support.google.com/webmasters/answer/96569

Re:Wikimedia is moving to Gitlab by dspillett on Wednesday October 28, 2020 @03:02PM (#24922781)
Most bots aren't going to bother checking if you've done that, so while they'll not get the expected benefit you'll still get the spam.
Re:Wikimedia is moving to Gitlab by jancsika on Wednesday October 28, 2020 @12:54PM (#24921091)
Because spammers fill the publicly viewable things-- like snippets and user profiles-- with spam. If it can be viewed without logging in, then they get it indexed with Google and it dilutes the search results.

Nobody views snippets and user profiles as a common part of daily development, so it takes time away from development to investigate those things to prune them. And if you don't prune it fast enough, it gets into the search results at which point it's even more of a pain in the ass to remove (even using Google's webmaster tools).

Re:Wikimedia is moving to Gitlab by dragonwriter on Wednesday October 28, 2020 @01:07PM (#24921295)
> Because spammers fill the publicly viewable things-- like snippets and user profiles-- with spam. If it can be viewed without logging in, then they get it indexed with Google and it dilutes the search results.

Isn't then the problem not that it is viewable but that it's not excluded from indexing by robots.txt?

Re:Wikimedia is moving to Gitlab by LeifCarrotson on Wednesday October 28, 2020 @12:54PM (#24921089)
Anything visible without login will be visible to people who want to follow a bare URL to your tracker and visible to a search engine crawler, making it visible to people without a URL who just search for the issue. That is, indeed, a plus.

But even if you require login to post stuff to the issue tracker, creating a login and posting a comment has been trivially automated.

You're no longer running a useful issue tracker, you're running a free ad network: you're hosting a dozen useful issues and a thousand advertisements and blackhat SEO comments for spammers.

If it's not visible to search engines, and your repo doesn't get much traffic, it's not nearly as valuable to spammers. It's kind of like cutting off your nose to spite your face, but those basic economics of cost to spammers, cost to users, value to spammers, and value to users are the only rules that you can really apply when hosting content on the Internet.

Re:Wikimedia is moving to Gitlab by jefftk on Wednesday October 28, 2020 @01:06PM (#24921283)
> creating a login and posting a comment has been trivially automated

Isn't this what various CAPTCHA tools handle?

You can also require email address validation.

Re:Wikimedia is moving to Gitlab by marcinzm on Wednesday October 28, 2020 @12:24PM (#24920660)
1. Create user account

2. Create spam content (snippets, profile, etc.)

3. Get spam content search indexed

4. Profit

Re:Wikimedia is moving to Gitlab by dtech on Wednesday October 28, 2020 @12:39PM (#24920873)
a free user sign up is not going to prevent scraping...
Re:Wikimedia is moving to Gitlab by cortesoft on Wednesday October 28, 2020 @12:44PM (#24920938)
I think the idea is that if you can't view issues without logging in, then google won't index your issues (because it can't view them), so you won't get people spamming in order to get into google
Re:Wikimedia is moving to Gitlab by noizejoy on Wednesday October 28, 2020 @04:19PM (#24923662)
Over the years I've frequently seen Google search results showing things that require login to the indexed site. Has that changed?
Re:Wikimedia is moving to Gitlab by marcinzm on Wednesday October 28, 2020 @07:28PM (#24925351)
I believe that requires the site giving google’s bot a logged in view rather than something google does themselves.
Re:Wikimedia is moving to Gitlab by wolco2 on Wednesday October 28, 2020 @02:20PM (#24922258)
It also means 95% of people will stop casually viewing issues.
Re:Wikimedia is moving to Gitlab by wyldfire on Wednesday October 28, 2020 @12:44PM (#24920952)
The problem isn't scraping, it's spam.