News for hackers. Stuff that matters.
Re:Wikimedia is moving to Gitlab by jancsika on Wednesday October 28, 2020 @12:06PM (#24920417)
Anyone thinking of moving to their own Gitlab instance with Gitlab CE-- either stay on Github or prepare to waste your time dealing with user spam bots that pollute your site's search results.

In other words-- if you want the common use case for a FOSS project:

1. publicly viewable main repository with publicly viewable issue tracker

2. requirement to log in to view all snippets, user profiles, perhaps even other repos as enforced by administrator settings (otherwise SEO bots will leverage these features to eat your search results)

3. anyone with an email can sign up to post issues to the main repo's issue tracker

There is no combination of settings in Gitlab CE to achieve this. Any sane approach has to leave out step #2. That means that your Gitlab instance gets hammered with user spam from bots which then get indexed in Google search results for your site.

Worse, Gitlab has no tools to make it easy to remove the user spam (and obviously no tools to prevent it from happening).

Just run a public-facing Gitlab CE instance for a few days. Search for one of the spam snippets you collect, and you'll find results for all the FOSS projects out there running their own Gitlab instances.

I've never seen any solutions offered by Gitlab for this, nor frankly any interest in the myriad bug reports about them addressing this at all.

Edit: typo

Re:Wikimedia is moving to Gitlab by phikai on Wednesday October 28, 2020 @12:49PM (#24921023)
Hi! I'm the PM at GitLab who works on Snippets, so thanks for providing this feedback. We do have Recaptcha support which can be configured - are you seeing these kinds of issues with that enabled/configured?

One item that is on the roadmap that is coming and may be of interest is `Optional Admin Approval for local user sign up` - https://gitlab.com/groups/gitlab-org/-/epics/4491.

I'm not in the group working on that, but it does appear to be coming soon and would limit the ability of newly created accounts from doing anything until they're approved.

Re:Wikimedia is moving to Gitlab by protoduction on Wednesday October 28, 2020 @01:06PM (#24921288)
Hi phikai,

I built a privacy friendly alternative to ReCaptcha called FriendlyCaptcha [1], is there a possibility to see this integrated as a more user friendly alternative?

Happy to chat (e-mail in profile)

[1] https://friendlycaptcha.com/

Re:Wikimedia is moving to Gitlab by barnabask on Wednesday October 28, 2020 @01:26PM (#24921565)
Man this needs more attention, cool project. I see you tried to submit to HN a couple of times and didn't get traction, that's too bad. Don't give up!
Re:Wikimedia is moving to Gitlab by aeyes on Wednesday October 28, 2020 @02:28PM (#24922362)
Is the demo somehow tweaked to be less hard?

On my machine it doesn't take any time to solve it and I see no signs of CPU usage. Even trying a couple of times in incognito mode and watching CPU immediately after loading the page for the first time.

On many sites creating a profile takes a few seconds. Loading one of my CPU cores for another 5 seconds doesn't really bother me if I wanted to create massive amounts of profiles/posts. I'll still do over 100 per minute on a standard desktop PC.

Re:Wikimedia is moving to Gitlab by protoduction on Wednesday October 28, 2020 @02:50PM (#24922633)
The default difficulty is set to a difficulty that makes sense on websites that have a varied audience (which includes some ancient browsers on old devices).

The solver runs in WebAssembly and is really really fast (~4M hashes per second) - but not every browser supports WASM yet (around 0.3% empirically). The JS fallback is around 10 times slower (more in 5+ year old browsers) - for those users you want at least a decent solve time too.

For Gitlab's audience the difficulty can probably be increased a lot - it all depends on the website and usecase. I'm sure the JS fallback's performance can be improved (it involves a lot of operations on 64bit ints that need to be represented as two numbers in JS), happy to accept PRs [1] :)

[1]: https://github.com/FriendlyCaptcha/friendly-pow/blob/master/...

Re:Wikimedia is moving to Gitlab by thinkloop on Wednesday October 28, 2020 @03:24PM (#24923041)
What are your thoughts on performing a quick intial test on each client to measure their performance then tailoring the puzzle to be difficult enough for each?
Re:Wikimedia is moving to Gitlab by unilynx on Wednesday October 28, 2020 @03:30PM (#24923112)
Once the spammer figures out what you're doing, he'll just throttle the CPU for the duration of the quick test.

Depending on how smart the test is, just having Date.now() return values with a -12000, -11000, -10000 offsets the first few calls might even do it

Re:Wikimedia is moving to Gitlab by sytse on Wednesday October 28, 2020 @01:14PM (#24921385)
That looks cool! Can someone create an issue to add support for this to GitLab? And maybe we can consider switching GitLab.com to this as well.
Re:Wikimedia is moving to Gitlab by robotmay on Wednesday October 28, 2020 @02:25PM (#24922326)
I'm personally interested in this too so I've created one :D https://gitlab.com/gitlab-org/gitlab/-/issues/273480
Re:Wikimedia is moving to Gitlab by sytse on Wednesday October 28, 2020 @03:11PM (#24922892)
Thanks for creating this! I think adding support for this in GitLab is a no-brainer. After that we can consider enabling it for GitLab.com
Re:Wikimedia is moving to Gitlab by birdsbirdsbirds on Wednesday October 28, 2020 @01:37PM (#24921721)
Hopefully you are successful, but how can you scale? If it takes 5 seconds on a desktop, then a server can solve 500.000 captchas per month. At $5 per month, a spammer can still send 1.000 messages for a cent.
Re:Wikimedia is moving to Gitlab by protoduction on Wednesday October 28, 2020 @01:52PM (#24921915)
It's not enabled yet in production - but the main mechanism is by increasing the difficulty as more requests are made from an IP in a certain timeframe (it's basically rate limiting at that point). Think: every 3rd request in a minute doubles the difficulty with some cooldown period.

With that the cost (and complexity) of an attack can hopefully be in the same ballpark (or higher) than ReCaptcha - without your end user having to label cars or send data to Google.

But in the end a determined spammer will get through any captcha cheaply (for reference: ReCaptcha solves are sold by the thousands for $1) - we just hope we can do better than ReCAPTCHA, especially UX-wise.

Re:Wikimedia is moving to Gitlab by leonidasv on Wednesday October 28, 2020 @09:52PM (#24926239)
I love this concept of proof-of-work captchas, but there's a growing number of tools and ways to bypass IP blocks via IP rotation[1], specially after the explosion of IaaS providers. How do you intend to tackle this?

[1] Some examples: https://rhinosecuritylabs.com/aws/bypassing-ip-based-blockin... https://oxylabs.io/products/real-time-crawler https://github.com/alex-miller-0/Tor_Crawler https://www.scrapinghub.com/crawlera/

Re:Wikimedia is moving to Gitlab by njitram on Thursday October 29, 2020 @07:35AM (#24929388)
There are free and paid list of all ip addresses from datacenters like https://udger.com/resources/datacenter-list, they probably existing for specifically preventing this, so maybe thats an option here.
Re:Wikimedia is moving to Gitlab by coder543 on Wednesday October 28, 2020 @02:14PM (#24922186)
The obvious follow-up question is how IPv6 impacts this, because I think it's supposed to be easy for someone to get their hands on a decent chunk of IPv6 addresses.

Maybe the difficulty could scale as a property of how similar the IP address is to previously seen addresses... so the addresses in the same /64 block would be very closely related, for example. (I think that's how IPv6 works... but definitely something I haven't researched lately, so I could just sound very confused)

Re:Wikimedia is moving to Gitlab by protoduction on Wednesday October 28, 2020 @02:31PM (#24922399)
I don't have all the answers yet, but indeed rate limiting a larger block (at least /64), or even at multiple prefix sizes with different weighting makes sense.
Re:Wikimedia is moving to Gitlab by zahllos on Wednesday October 28, 2020 @08:18PM (#24925709)
So the way this is supposed to work is that providers hand out /48s and each site should be allocated a /64. In practice if you for example rent a VPS, you'll be handed a /64 for it by your service provider from their /48.

I would personally treat any /64 as the same. Depending on your local network setup the second half of the address could be anything and could change frequently. You might also get multiple addresses. Whereas getting a new /64, or /48, requires slightly more effort.

Of course there's a risk you'll block a /64 and that takes out some whole company or whatever, but I've seen that happen to corporate proxies that got flagged as a source of spam as well so this is not an easy problem even without the 2^128 address space.

Re:Wikimedia is moving to Gitlab by ognarb on Thursday October 29, 2020 @08:41AM (#24929863)
Your website mention that friendlycaptcha is open source but looking at the license in the repository, it is a custom license that can't be defined as open source. Can you change it to source available?
Re:Wikimedia is moving to Gitlab by typenil on Wednesday October 28, 2020 @05:15PM (#24924229)
Love to see this. ReCaptcha is nothing short of a menace. I'll take a shot at this for my next project
Re:Wikimedia is moving to Gitlab by laughinghan on Wednesday October 28, 2020 @03:57PM (#24923427)
There doesn't appear to be any discussion on your website or on GitHub about why, to be blunt, this is even a good idea in the first place.

A classic 2004 paper, "Proof-of-Work" Proves Not to Work [0], explained that the fundamental problem with proof-of-work bot filters is that attackers will always be able to solve the cryptographic puzzle faster than legitimate users. A touch of security-through-obscurity can help at the margins, but you chose Blake2b, which is used by cryptocurrencies like Zcash, Siacoin, and Nano [1], and as a result there are optimized GPU algorithms (first Google result [2]) and FPGA designs (one of the top Google results [3]). Have you run the numbers on any of those?

The closest to any discussion of these numbers that I saw was a mention on your website that it may take up to 20s on mobile; for comparison, the much-hated image CAPTCHA takes about 6-12s on average for native English speakers, and 7-14s for non-native speakers [4].

In another comment you bring up the idea of starting with a lower difficulty, and increasing it with repeated requests from the same IP address (IPv4, I assume). Unfortunately, access to unique IPv4 addresses is highly correlated with access to more compute power: laptops and desktops in developed countries are most likely to be in a household with a unique IPv4 address, whereas mobile devices on 4G internet and households in developing countries are more likely to be behind Carrier-Grade NAT [5], where thousands or millions [6] of hosts share a pool of a handful or dozens of IPv4 addresses. (The exact same concern applies to IPv6 /64 prefixes.)

This means that mobile devices will face a "double-jeopardy": your service will present them with higher proof-of-work difficulties because the same IPv4 address is shared by more people, and at the same time, the mobile device solves the proof-of-work slower for the same difficulty than a desktop.

Do you have documented anywhere on your website or GitHub how you address these concerns?

[0]: https://www.cl.cam.ac.uk/~rnc1/proofwork.pdf

[1]: https://en.bitcoinwiki.org/wiki/Blake2b

[2]: https://github.com/zhq1/sgminer-blake2b

[3]: https://xilinx.github.io/Vitis_Libraries/security/2020.1/gui...

[4]: http://theory.stanford.edu/people/jcm/papers/captcha-study-o...

[5]: https://en.wikipedia.org/wiki/Carrier-grade_NAT

[6]: Yes, millions. RFC 6598 reserved a /10 for them, which is 4 million unique IPv4 addresses: https://tools.ietf.org/html/rfc6598

Re:Wikimedia is moving to Gitlab by coder543 on Wednesday October 28, 2020 @04:16PM (#24923631)
I'm not associated with the project in any way, but your well researched comment did miss at least one important factoid.

This comment:

> The closest to any discussion of these numbers that I saw was a mention that it may take up to 20s on mobile; for comparison, the much-hated image CAPTCHA takes about 6-12s on average for native English speakers, and 7-14s for non-native speakers.

Missed this quote from the website:

> As soon as the user starts filling the form it starts getting solved

> By the time the user is ready to submit, the puzzle is probably already solved.

The time spent solving reCAPTCHA is active user involvement. The time being spent on Friendly Captcha is passive and can overlap with time being spent filling out a form.

"up to 20 seconds" was also seemingly presented as a worst-case scenario. Most users' devices would presumably be faster than that, but I don't know how the author researched that conclusion on how performance scales. Friendly Captcha does report back some information on how long it is taking users to solve the captcha, and it looks like website owners could use that to adjust the difficulty based on the needs of their specific audience and how tolerant they are of untargeted spam.

The stuff you point out about Blake2b seems entirely legitimate, and I wonder if an Argon variant would be more appropriate to avoid specialized hardware being quite so problematic.

Personally, I really like the idea of Friendly Captcha. Certainly, there are problems with any captcha implementation. People can rant for many, many paragraphs about websites that use reCAPTCHA... I'm not surprised to see someone ripping apart a different captcha system. The ideal solution would be for spammers to just stop being so obnoxious... but good luck with that plan.

Re:Wikimedia is moving to Gitlab by laughinghan on Wednesday October 28, 2020 @04:41PM (#24923888)
The time being spent on Friendly Captcha is passive and can overlap with time being spent filling out a form.

Great point!

I wonder if an Argon variant would be more appropriate

The creators of Argon2 actually also created a memory-hard proof-of-work function they call MTP (for "Merkle Tree Proof", which is a terrible name, totally un-Googleable; I always have to search for the title of their paper, "Egalitarian Computing"): https://arxiv.org/pdf/1606.03588.pdf

A bug bounty for it was sponsored by Zcoin, which is nice. Zcoin is actually considering moving away from it, but mainly because the proof size of 200kb is prohibitive, which is less of a concern for a captcha system: https://forum.zcoin.io/t/should-we-change-pow-algorithm/477

I'm not surprised to see someone ripping apart a different captcha system

I really don't mean to rip it apart. I just wanted to see some discussion, any discussion, of the well-known flaws with the idea and what ideas OP has to address them.

Re:Wikimedia is moving to Gitlab by protoduction on Wednesday October 28, 2020 @05:40PM (#24924421)
It is also important to note that the 6-12 seconds and 7-14 seconds reported in the paper is for the garbled text CAPTCHAs, not for image labeling tasks (fire hydrants, cars, etc).
Re:Wikimedia is moving to Gitlab by protoduction on Wednesday October 28, 2020 @05:20PM (#24924273)
I'll try to provide my thoughts on each of the issues you've mentioned, let me know if there's something I missed.

On using blake2b: I chose blake2b as I was looking to use a hash function that is small in implementation, readily available and already optimized. With WebAssembly the solver can achieve (close to native) speeds and be least be an order of magnitude or two closer to optimized GPU algorithms.

Using specialized hardware, image tasks (and even more so audio tasks which must be present for accessibility reasons) have the same issue that they can be solved by GPU algorithms (i.e. machine learning, in which even a low percentage success rate would already be enough). If you search on GitHub you will find there are more ML captcha cracking repos than captcha implementations - they are probably even easier to get started with than adapting GPU miner code.

Image/Audio Captcha vs ML is an arms race that can be beat for split seconds of compute (even on CPU) or cheap human labeling: it's just as broken. FriendlyCaptcha optimizes for the end user (privacy + effort + accessibility) by not engaging in the arms race - I think it makes a better trade-off. Like the sibling comment pointed out the captcha solving can happen entirely in the background so that hopefully it doesn't even make the user wait.

As for rate limiting/difficulty adjustment: it's not perfect and it could lead to problems if you share the IP with a spammer (and let's be realistic: even with a million users on one IP there won't be tens of users signing up to some forum per minute). Also normal captchas have problems here though: users from these locales already get presented with much more difficult+frequent recaptcha tasks (I also doubt they are localized: American sidewalks are harder to label if you've never seen one in real life). Setting a reasonable upper limit to difficulty may be good enough here.

On not using blake2b: I have considered mutating the hashing algorithm every day randomly to make writing an optimized solver for it all that more difficult - but that would mean one could no longer self-serve the JS+WASM and be done with it. I won't rule it out for FriendlyCaptcha v2 if this does ever become a real problem.

Swapping out the hash function should be easy (the puzzles are versioned to allow for this). If you have a different function in mind and someone implements it in Assemblyscript (so we also have a JS fallback) then we can definitely consider it.

Re:Wikimedia is moving to Gitlab by laughinghan on Saturday October 31, 2020 @02:55PM (#24954197)
Thanks for your detailed response.

I've seen all the projects claiming to have broken ReCAPTCHA—often using Google's own ML services, hilariously—but it's unclear to me how broken image/audio CAPTCHAs are in practice (and the number of GitHub repos doesn't seem like a good measure to me). If they really are completely broken, then why are they still so widely used? If they really are completely broken by ML, how do human CAPTCHA-solving services stay in business?

FriendlyCaptcha optimizes for the end user (privacy + effort + accessibility)

Good point. I am concerned though that burning CPU cycles on the proof-of-work uses battery life if the end-user is on mobile, without getting their getting any choice in the matter. What if, given an informed choice, they would have preferred an image CAPTCHA? (On the other hand, that could use more cellular data. Might be good to run the numbers on this too.)

even with a million users on one IP there won't be tens of users signing up to some forum per minute

I think this is a bad choice of threat model. "Some forum" would likely be better off with simpler measures, like a hidden honeypot textbox: https://dev.to/felipperegazio/how-to-create-a-simple-honeypo...

Re:Wikimedia is moving to Gitlab by NorwegianDude on Wednesday October 28, 2020 @07:51PM (#24925525)
Cool project, but I do find it quite ironic that it's named friendly captcha when it's not a captcha.
Re:Wikimedia is moving to Gitlab by Eldt on Wednesday October 28, 2020 @07:59PM (#24925573)
How would you define "CAPTCHA"?
Re:Wikimedia is moving to Gitlab by perryizgr8 on Thursday October 29, 2020 @12:02AM (#24927059)
The original expansion was "Completely Automated Public Turing test to tell Computers and Humans Apart".
Re:Wikimedia is moving to Gitlab by jimmydorry on Wednesday October 28, 2020 @11:52PM (#24927006)
CAPTCHA: a computer program or system intended to distinguish human from machine input, typically as a way of thwarting spam and automated extraction of data from websites

I would say this Oxford Languages dictionary definition is close enough.

Re:Wikimedia is moving to Gitlab by webphineas on Wednesday October 28, 2020 @01:50PM (#24921898)
Really nice! Finally someone is using the blockchain technology in a meaningful way!
Re:Wikimedia is moving to Gitlab by laughinghan on Wednesday October 28, 2020 @03:10PM (#24922881)
This doesn't use a blockchain, it uses a Hashcash-style proof-of-work function (an idea that predates the Bitcoin by decades): https://en.wikipedia.org/wiki/Hashcash
Re:Wikimedia is moving to Gitlab by redbergy on Wednesday October 28, 2020 @03:07PM (#24922859)
Awesome work, I will be giving this a try in my next project
Re:Wikimedia is moving to Gitlab by remram on Wednesday October 28, 2020 @06:01PM (#24924608)
> up to 20 seconds on old smartphones

That sounds like a very battery-unfriendly idea.

Re:Wikimedia is moving to Gitlab by protoduction on Wednesday October 28, 2020 @06:25PM (#24924815)
It's not perfect, but maxing a single core for 20 seconds on an older smartphone is a necessary evil for this kind of captcha.

The alternative: loading a third party script and multiple images (~2MB) to label for ReCAPTCHA and spending time performing the task also takes some battery (and mental) power.

Re:Wikimedia is moving to Gitlab by jancsika on Wednesday October 28, 2020 @01:11PM (#24921357)
> We do have Recaptcha support which can be configured - are you seeing these kinds of issues with that enabled/configured?

Thanks, I have used Recaptcha for a long time now. It made no difference.

> One item that is on the roadmap that is coming and may be of interest is `Optional Admin Approval for local user sign up` - https://gitlab.com/groups/gitlab-org/-/epics/4491.

Yes, that would be a very sensible solution and welcome feature for my use case here.

Unfortunately, from the bottom of that issue tracker:

"Yikes. I'm glad we did the further breakdown and pre-work. It's a bit cringeworthy looking back and seeing I estimated a 5"

Re:Wikimedia is moving to Gitlab by mushakov on Wednesday October 28, 2020 @02:15PM (#24922198)
Hi! I'm a PM at GitLab. Please see my reply above for more details but TL;DR we shipped the first iteration of the `Optional Admin Approval for local user sign up` feature in 13.5. I'd love your feedback! Please comment on the epic if there are other changes for this feature that would help your use case https://gitlab.com/groups/gitlab-org/-/epics/4491
Re:Wikimedia is moving to Gitlab by jancsika on Wednesday October 28, 2020 @03:45PM (#24923296)
Thanks for the update. I can certainly manage user sign-up from the admin tab for the time being. Once it's hooked into email, I believe that will make things maintainable again for me.

From a UX standpoint it's still sub-par. Someone who wants to report an issue doesn't want to wait an arbitrary amount of time to be allowed to report an issue. They are ready to report it at that moment.

And as an admin, I don't want to have to approve new users on a schedule to ensure the delay is low enough that they are still willing to submit the issue after I approve them. I'd much prefer they go ahead and submit the content, especially so that I can use it in my review of whether to approve the sign up or not.

I seem to remember some pattern in Gitlab where my login period timed out before I finished making a comment. When I logged back in, Gitlab had somehow saved my comment content so that I could then post it so that others could see it. Is there any way to use that pattern for users who haven't been approved yet? So that they can post content, but with a warning shown to them that other users won't see it until the sign-up is approved.

Re:Wikimedia is moving to Gitlab by mushakov on Wednesday October 28, 2020 @09:28PM (#24926119)
That's a really interesting idea! Users could have limited interactions with the instance and content queued up until approved by an administrator. I created an issue to capture this. https://gitlab.com/gitlab-org/gitlab/-/issues/273542
Re:Wikimedia is moving to Gitlab by rightbyte on Wednesday October 28, 2020 @01:07PM (#24921307)
Relying on Google's Spying-as-a-Service tooling is not very FOSS at all.

There need to be other ways to reach out to users who block Google.

Re:Wikimedia is moving to Gitlab by encom on Wednesday October 28, 2020 @01:23PM (#24921517)
I immediately back out whenever encounter Recaptcha.

The other day I was forced to endure it, because I wanted to delete my ancient Minecraft account, since Microsoft pulled a Facebook and are going to require a Microsoft account to play going forwards. Without exaggeration, it took me 15 minutes of training Google surveillance AI (had to solve it three times), for Recaptcha to let me in. I guess Google really hates me.

Re:Wikimedia is moving to Gitlab by dalmo3 on Wednesday October 28, 2020 @07:50PM (#24925523)
Yesterday I spent the longest ever with a recaptcha, about 2-3 minutes, at a frigging checkout page. I decided to endure it just because I really needed that ergonomic kb+mouse combo.

Hopefully they'll allow me to solve captchas for longer without getting a RSI.

Re:Wikimedia is moving to Gitlab by wolco2 on Wednesday October 28, 2020 @02:12PM (#24922170)
Are you sure you are human?
Re:Wikimedia is moving to Gitlab by myself248 on Wednesday October 28, 2020 @02:34PM (#24922445)
I'm human enough, and I've been a licensed driver long enough, to recognize that rumble strips at the side of a road are not crosswalks. But apparently enough bots thought they were that the system is now trained on that 'fact', and I as a human am forced to misidentify rumble strips as crosswalks to pass as human.

It's bizarre.

Re:Wikimedia is moving to Gitlab by ignoranceprior on Thursday October 29, 2020 @09:19AM (#24930153)
ReCaptcha also thinks that mailboxes are parking meters, for some reason.
Re:Wikimedia is moving to Gitlab by encom on Wednesday October 28, 2020 @02:29PM (#24922371)
Re:Wikimedia is moving to Gitlab by Kwpolska on Wednesday October 28, 2020 @05:43PM (#24924452)
Try reCAPTCHA’s audio version (the headphones icon), it’s much easier than guessing what images it wants you to click (if you speak English, have headphones, and are not hearing-impaired).
Re:Wikimedia is moving to Gitlab by WalterSear on Wednesday October 28, 2020 @04:41PM (#24923879)
This sounds like it has the potential to be a modern version of the credit score: avoid it enough, and you become persona non grata. That is, for more than 15 minutes.
Re:Wikimedia is moving to Gitlab by db48x on Wednesday October 28, 2020 @04:11PM (#24923578)
I do the same thing.
Re:Wikimedia is moving to Gitlab by meibo on Wednesday October 28, 2020 @09:23PM (#24926091)
You're doing something very wrong if you take 15 minutes to solve these and aren't on Tor. Even on public VPN and Firefox this doesn't happen usually. I know people that pick the wrong options to fuck with their models though, and then go on HN to complain about recaptcha being annoying.
Re:Wikimedia is moving to Gitlab by randunel on Thursday October 29, 2020 @08:54AM (#24929960)
I have similar issues. I do not pick the wrong options. It also doesn't take me too long to solve the captchas, leading to "too many queries from your ip address". This is what internet users deal with when blocking most google services.
Re:Wikimedia is moving to Gitlab by mushakov on Wednesday October 28, 2020 @02:11PM (#24922147)
Thanks for bringing up this epic in the conversation phkai. I'm a PM at GitLab for our Auth group and am working on the `Optional Admin Approval for local user sign up` feature. I'm happy to tell y'all that we shipped the first iteration of this in our 13.5 release. You can find more information in our release blog https://about.gitlab.com/releases/2020/10/22/gitlab-13-5-rel... . I've also updated the epic with more information about its current status https://gitlab.com/groups/gitlab-org/-/epics/4491#status-upd....
Re:Wikimedia is moving to Gitlab by kemayo on Wednesday October 28, 2020 @05:18PM (#24924254)
For this specific case, the Wikimedia Foundation has explicitly stated that "It is the Free Software release of GitLab that runs optional non-free software such as Google Recaptcha to block abuse, which we do not plan to use." So, not incredible helpful at the moment.

Also, is manual approval for new signups a good idea for a large FOSS project? It seems like a pretty big barrier to legitimate discussion.

Re:Wikimedia is moving to Gitlab by anarcat on Thursday October 29, 2020 @08:55AM (#24929975)
We (at torproject.org) also adopted GitLab CE recently and we had to close down registrations because of abuse. Tens (hundreds?) of seemingly fake accounts were created in the two weeks we had registrations opened and we had to go through each one of those to make sure they were legitimate. In our case, snippets were not directly the problem: user profiles were used as spam directly.

We can't use ReCAPTCHA or Akismet for obvious privacy reasons. The new "admin approval" process in 13.5 is interesting, but doesn't work so well for us, because it's hard to judge if an account should be allowed or not.

As a workaround, we implemented a "lobby": a simple Django app that sits in front of gitlab to moderate admissions.

https://gitlab.torproject.org/tpo/tpa/gitlab-lobby/

The idea is people have to provide a reason (free form text field) to justify their account. We'd also like people to be able to file bugs from there directly, in one shot.

We're also thinking of enabling the service desk to have that lower bar for entry, but we're worried about abuse there as well.

Having alternatives to ReCAPTCHA would be quite useful for us as well.

Re:Wikimedia is moving to Gitlab by MrStonedOne on Wednesday October 28, 2020 @03:18PM (#24922978)
You have to remove incentives. Block the viewing of these snippets by logged out users by default and require opt-in and a way to whitelist snippets by snippet or user. Same for user profiles
Re:Wikimedia is moving to Gitlab by noizejoy on Wednesday October 28, 2020 @04:13PM (#24923595)
I don't think this is targeting human views - but it's targeting Google for SERP (Search Engine Results Pages) boost.
Re:Wikimedia is moving to Gitlab by MrStonedOne on Wednesday October 28, 2020 @05:10PM (#24924191)
That's the point. Having a way to disable search engines would also work, but wouldn't be obvious to spammers so they would still try to spam. Disabling all users by default works to remove the incentive to try
Re:Wikimedia is moving to Gitlab by gaba on Thursday October 29, 2020 @03:36PM (#24934666)
Is this something that we will have in the CE version (the open licensed one) or it will only go to the enterprise one?
Re:Wikimedia is moving to Gitlab by 67868018 on Wednesday October 28, 2020 @01:57PM (#24921974)
None of your captcha settings work, not even the invisible captcha setting that requires enabling a feature flag.
Re:Wikimedia is moving to Gitlab by xiphias2 on Wednesday October 28, 2020 @04:06PM (#24923526)
Have you thought of the option of disabling links? That would make SEO spam impossible
Re:Wikimedia is moving to Gitlab by pitay on Wednesday October 28, 2020 @05:24PM (#24924299)
Is just adding the attribute rel="nofollow ugc" to any links in submitted content may be good enough. This tells search engines to not index, or tag them as suspicious, allowing them them to identify SEO spam more easily. [1]

Having both options would be great.

[1] https://support.google.com/webmasters/answer/96569

Re:Wikimedia is moving to Gitlab by gramakri on Wednesday October 28, 2020 @01:24PM (#24921534)
The spam is infuriating (not GitLab's fault, of course). Atleast, on our instance at https://git.cloudron.io, we got massive snippet spam. After we disabled snippets, we got massive spam on the issue tracker (!). The way we "fixed" is by turning on mandatory 2FA for all users.

As a general lesson, what we learnt is these are not bots. These are real humans working in some poor country manually creating accounts (always gmail accounts) and pasting all sorts of random text. Some of these people even setup 2FA and open issues with junk text, it's amazing. Unfortunately, GitLab from what I can tell cannot make issues read-only to non project members (i.e I only want project members to open issues, others can just read and watch issues).

Currently, our forum spam (https://forum.cloudron.io) is way more than GitLab spam. On the forum, we even have Captcha enabled (something we despise) but even that doesn't help when there are real humans at work.

Re:Wikimedia is moving to Gitlab by Symbiote on Wednesday October 28, 2020 @03:30PM (#24923105)
We had one of the "real humans" write to us (in issues) asking us to leave his spam up for "just a few hours".

We implemented a filter anyway.

(This was not Gitlab, but a specific form on our unique website.)

Re:Wikimedia is moving to Gitlab by packetlost on Wednesday October 28, 2020 @05:41PM (#24924437)
> asking us to leave his spam up for "just a few hours"

What... why? What is their goal???

Re:Wikimedia is moving to Gitlab by vvpan on Wednesday October 28, 2020 @05:46PM (#24924488)
Feeding their family?
Re:Wikimedia is moving to Gitlab by packetlost on Wednesday October 28, 2020 @05:51PM (#24924521)
But like... who's paying for that kind of spam??
Re:Wikimedia is moving to Gitlab by Symbiote on Thursday October 29, 2020 @03:47AM (#24928161)
In our case, mostly pirate TV streaming services. (Watch NBA games live etc.)
Re:Wikimedia is moving to Gitlab by mgbmtl on Wednesday October 28, 2020 @06:26PM (#24924836)
A lot of people unfortunately seem to bite on those "SEO experts" kind of emails. I had a few clients ask me if they should give it a try, since, "why not, it's cheap".
Re:Wikimedia is moving to Gitlab by csdreamer7 on Wednesday October 28, 2020 @01:29PM (#24921616)
Why are they posting random text in Gitlab?
Re:Wikimedia is moving to Gitlab by grey-area on Wednesday October 28, 2020 @03:24PM (#24923048)
This is a typical spam profile. Usually they contain links, which search engines follow.

https://forum.cloudron.io/user/cardioaseg

Re:Wikimedia is moving to Gitlab by edflsafoiewq on Wednesday October 28, 2020 @05:09PM (#24924178)
The link contains rel=nofollow.
Re:Wikimedia is moving to Gitlab by grey-area on Thursday October 29, 2020 @04:37AM (#24928421)
That doesn't matter, see other comments below on Google's changing treatment of this attribute.

Also you'll find spambots posting on any open form on the internet even if it doesn't do them any good, because much of it is automated, so even if you hide the results the spam will still come in.

Re:Wikimedia is moving to Gitlab by leipert on Wednesday October 28, 2020 @05:54PM (#24924549)
I don’t know for sure, but I think our Markdown implementation adds nofollow.
Re:Wikimedia is moving to Gitlab by brlewis on Wednesday October 28, 2020 @11:22PM (#24926819)
I used to think that spammers would stop if their spamming didn't win them any results. But they don't care. They spread their spam as widely as possible without trying to prune out the places where it does them no good.
Re:Wikimedia is moving to Gitlab by gramakri on Wednesday October 28, 2020 @03:04PM (#24922809)
I am not entirely sure. See https://forum.cloudron.io/users, if you go to say page 10 or something you will see all sorts of nonsense. I am still trying to figure what the best way to fight this spam (because captcha is enabled and required to even create accounts). But these are real people and not bots. I know this because they even post new messages all the time.
Re:Wikimedia is moving to Gitlab by fancyfish on Wednesday October 28, 2020 @03:23PM (#24923039)
Definitely the SEO backlinks- for example one profile I see is linking to an Indian escort service in the profile.
Re:Wikimedia is moving to Gitlab by coder543 on Wednesday October 28, 2020 @04:40PM (#24923867)
Maybe GitLab needs an option to disable external linking, and filter any comment that contains an external link automatically
Re:Wikimedia is moving to Gitlab by csdreamer7 on Wednesday October 28, 2020 @06:04PM (#24924626)
Or a nofollow option (add rel=nofollow)
Re:Wikimedia is moving to Gitlab by dnsmichi on Thursday October 29, 2020 @06:57AM (#24929146)
That's a great idea. We have discussed ways of getting a trust level, and enable this for specific groups. Discourse uses the same system for preventing spam. "Good" bots detect the rel=nofollow and do not come back.

See my proposal here: https://gitlab.com/gitlab-org/gitlab/-/issues/14156#note_258...

Re:Wikimedia is moving to Gitlab by dnsmichi on Thursday October 29, 2020 @01:18PM (#24932795)
Iterating on my original thought, here is a smaller feature request for self-hosted GitLab instances. This can help GitLab.com too: https://gitlab.com/gitlab-org/gitlab/-/issues/273618
Re:Wikimedia is moving to Gitlab by coder543 on Thursday October 29, 2020 @01:22PM (#24932827)
I still think an even better path to success is to allow entirely disabling linking for non-admins.

Google no longer treats "nofollow" as strongly as it used to: https://webmasters.googleblog.com/2019/09/evolving-nofollow-...

Re:Wikimedia is moving to Gitlab by dnsmichi on Thursday October 29, 2020 @04:50PM (#24935656)
Thanks for sharing, I have added it to the issue, maybe you want to join the discussion there :) https://gitlab.com/gitlab-org/gitlab/-/issues/273618#note_43...

Just so that I can follow - URLs posted by non-admins should not render as HTML URLs at all? Wouldn't that be quite limiting for OSS project members for example?

Re:Wikimedia is moving to Gitlab by coder543 on Thursday October 29, 2020 @04:58PM (#24935785)
My opinion on the topic isn't definitive by any means, but I think a lot of projects would do just fine without allowing arbitrary hyperlinks to be added by non-admins.

I think being able to link to related issues and link into the code is still important, for example.

It's certainly a trade off, but spammers want it to be rendered as a link.

Re:Wikimedia is moving to Gitlab by riffic on Wednesday October 28, 2020 @01:34PM (#24921675)
getting that sweet sweet seo backlink juice
Re:Wikimedia is moving to Gitlab by gkop on Wednesday October 28, 2020 @01:37PM (#24921722)
Isn’t that why we add rel=nofollow to low friction user submitted links on our platforms?
Re:Wikimedia is moving to Gitlab by ivank on Wednesday October 28, 2020 @01:53PM (#24921920)
Google changed the interpretation of those a year ago. https://webmasters.googleblog.com/2019/09/evolving-nofollow-...
Re:Wikimedia is moving to Gitlab by nurettin on Wednesday October 28, 2020 @02:29PM (#24922370)
> Looking at all the links we encounter can also help us better understand unnatural linking patterns.

It appears as though they want to mark these links in order to prevent inorganic SEO, not help it.

Re:Wikimedia is moving to Gitlab by nerdponx on Wednesday October 28, 2020 @02:48PM (#24922609)
I don't get it. They post all this spam in the hopes that people click on the links therein, thereby boosting the ranking of those sites? Does that actually work at all?
Re:Wikimedia is moving to Gitlab by rudedogg on Wednesday October 28, 2020 @03:20PM (#24922998)
It doesn’t actually require anyone clicking on the links. Google sees inbound links and uses that as a factor when calculating the ranking of the linked page.
Re:Wikimedia is moving to Gitlab by IggleSniggle on Wednesday October 28, 2020 @05:53PM (#24924537)
I thought that was how it worked like a decade or more ago, but not today.
Re:Wikimedia is moving to Gitlab by technion on Wednesday October 28, 2020 @07:47PM (#24925495)
Regardless of whether it works, people still pay for it. I have a Facebook ad right now that says "Get over 500,000 backlinks for $29.99". No doubt it's someone with a bot that spams comment forms.
Re:Wikimedia is moving to Gitlab by mpol on Wednesday October 28, 2020 @05:47PM (#24924500)
A service like Stop Forum Spam might be a solution to this. It checks for IP address and email address and gives it a value based on how likely it is assumed to be a spammer.

When they have to set up a new email account and maybe even a new IP address for every few accounts, it gets to be a lot of work soon.

https://www.stopforumspam.com/

Re:Wikimedia is moving to Gitlab by Siira on Wednesday October 28, 2020 @02:16PM (#24922201)
How do you know they are real humans? I imagine bots doing 2FA would still be cheaper.
Re:Wikimedia is moving to Gitlab by gramakri on Wednesday October 28, 2020 @03:06PM (#24922837)
I know this because in our forum we have LOTS of "spam" users - https://forum.cloudron.io/users . These users will go into posts and actually make "helpful" comments. Like, "Oh I tried this solution but I found that my disk my full. Deleting data fixed my problem". It almost seems genuine but they build reputation and once they have some votes, they go back and edit all the comments to have links.
Re:Wikimedia is moving to Gitlab by eznzt on Wednesday October 28, 2020 @03:12PM (#24922909)
Banning entire countries helps a lot. I don't want to name certain countries, but let's assume it's one where it's common to see human corpses floating on a big river.
Re:Wikimedia is moving to Gitlab by jychang on Wednesday October 28, 2020 @05:32PM (#24924359)
That doesn't help narrow it down. I live in Seattle and the first think I thought of was a popular tiktok of teens finding a corpse in the river last month.
Re:Wikimedia is moving to Gitlab by reaperducer on Wednesday October 28, 2020 @05:56PM (#24924568)
The word you missed was "common."
Re:Wikimedia is moving to Gitlab by d3nj4l on Wednesday October 28, 2020 @09:02PM (#24925968)
Apart from the fact that banning an entire country from contributing to their code would be antithetical to the Wikimedia foundation, if you're implying the country which I think you're implying (which is also where I live, btw) you'll:

1. Ban a burgeoning tech industry which has produced over 20 unicorns,receives billions in funding from across the world and produces world-class tech talent; 2. Ban millions of other OSS developers from contributing; and 3. Just lead to SEO spammers picking out other impoverished countries to spam from, which means finally you'll end up with only people from the "west" being able to contribute in any way.

Re:Wikimedia is moving to Gitlab by boneitis on Wednesday October 28, 2020 @02:29PM (#24922374)
Many bots are likely still powered under the hood by humans.

On my backlog of projects to do is to make a browser extension that solves the more obnoxious captchas for me, as I'm regularly behind vpn and fall into ridiculously long solve loops.

On the most popular api i could find, $10 buys you a shockingly LOT of solves (not that I've tested it yet). It is automatable but ultimately still powered by humans.

Re:Wikimedia is moving to Gitlab by dannyw on Wednesday October 28, 2020 @02:41PM (#24922524)
It’s incredibly sad how the open web is being destroyed by google’s recaptcha.
Re:Wikimedia is moving to Gitlab by Kalium on Wednesday October 28, 2020 @04:51PM (#24923991)
Without google's recaptcha, do you think there would be less spam?

Personally, I suspect there would be more without at least some speed bumps to raise the cost of spamming. I would absolutely love for there to be better options than recaptcha that meets the same needs around bot-detection, price, implementation effort, and accessibility. It is, sadly, the best option I've seen on offer.

You're right. The scenario we're in is incredibly sad. It would be wonderful if the individual actors involved had better options to meet their needs.

Re:Wikimedia is moving to Gitlab by nerdkid93 on Wednesday October 28, 2020 @03:00PM (#24922770)
I'd argue that it's equally sad to see the open web get destroyed by massive DDoS attacks and malicious actors. How would you keep your own website up if it was constantly being attacked?
Re:Wikimedia is moving to Gitlab by hombre_fatal on Wednesday October 28, 2020 @09:56PM (#24926260)
You're barking up the wrong tree. Bad actors create abuse and spam which they can do because of fundamental weaknesses in the design of the internet. People trying to solve that reality with Recaptcha (and Cloudflare for that matter) aren't the ones destroying the internet.
Re:Wikimedia is moving to Gitlab by boneitis on Wednesday October 28, 2020 @10:29PM (#24926446)
I don't think it's so much the wrong tree as it is but one tree in a forest to be barking up.

All the maturely developed bot filters frequently throw me in an endless battery of tests that have me giving up in frustration before finally making it through to content I'm requesting.

> aren't the ones destroying the internet

IMO they are every bit as much destroying it as the abusers they're claiming to fend off.

Re:Wikimedia is moving to Gitlab by boneitis on Wednesday October 28, 2020 @02:56PM (#24922714)
I'm totally in that camp of opinion, although I'll acknowledge the escalating abuses carried out by both "sides."

In the meantime, i hope to have the savviness to program my own way out of unsolvable captchas.

Re:Wikimedia is moving to Gitlab by FredFS456 on Wednesday October 28, 2020 @02:56PM (#24922711)
Already exists: https://github.com/dessant/buster

Edit: on re-read, you meant solving using humans. Buster uses speech-to-text APIs to solve.

Re:Wikimedia is moving to Gitlab by boneitis on Wednesday October 28, 2020 @02:59PM (#24922749)
Every lead to spice(e:solve, lol gboard) my problems is probably worth a peek. I'll take a look, thank you.
Re:Wikimedia is moving to Gitlab by pcmaffey on Wednesday October 28, 2020 @06:00PM (#24924596)
Could add nocrawl to your robots.txt and advertise the fact on signup page that search engines won’t find this content.
Re:Wikimedia is moving to Gitlab by sytse on Wednesday October 28, 2020 @12:29PM (#24920746)
At GitLab Inc. we have a Trust and Safety team https://about.gitlab.com/handbook/engineering/security/opera... that prevents spam.

So far that functionality has lived in separate repositories from the core codebase since few people needed it, the cycle time was quicker, and it is an advantage to not have the spammers see the code.

If there is strong interest in collaborating on this I'm sure they will be happy to engage. I'll ask them how best to structure this.

Re:Wikimedia is moving to Gitlab by ran3824692 on Wednesday October 28, 2020 @12:44PM (#24920942)
There's been a gitlab bug for almost 3 years to stop relying on recaptcha, https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45684 Debian, KDE and Gnome have never wanted to make their users run Google's nonfree javascript blob to contribute on their gitlab instance. There's been interest, Gitlab has done very little about it. Edit: other bugs about this can be found here https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46548
Re:Wikimedia is moving to Gitlab by GLJHunt on Wednesday October 28, 2020 @04:00PM (#24923461)
We have a team currently working on improving the detection and mitigation of spam. We continue to look for ways to improve the security and user experience of our product. Our product includes the Akismet Spam filter which you can read more about in our handbook: https://about.gitlab.com/handbook/support/workflows/managing.... Further, Gitlab.com includes the ability to report abuse directly to our trust & safety team here: https://about.gitlab.com/handbook/engineering/security/opera... however, the report abuse feature on self-managed reports back to the instance admin. We are also currently developing an anti-spam feature intended to further improve spam detection & mitigation. This is set to be enabled on GitLab.com within 3 months.
Re:Wikimedia is moving to Gitlab by nico_h on Wednesday October 28, 2020 @07:23PM (#24925317)
As mentioned above in the thread, multiple times, maybe a simpler solution to reduce spam is to remove incentives by:

- removing links (making them as plain text forcing users to copy paste them..) - hiding links from non-registered users (plain text to non-registered users, clickable for registered users), - blocking links from search engine crawlers (robots.txt / rel=nofollow...).

Maybe these fall in the "for each complex problem there is simple but wrong solution" but it sounds like it's worth a try.

Re:Wikimedia is moving to Gitlab by mpol on Wednesday October 28, 2020 @05:54PM (#24924545)
(I already replied on a different thread but this might make more sense)

A service like Stop Forum Spam might be a solution to this. It checks for IP address and email address and gives it a value based on how likely it is assumed to be a spammer.

When they have to set up a new email account and maybe even a new IP address for every few accounts, it gets to be a lot of work soon.

https://www.stopforumspam.com/

It has a very simple API and is not that hard to implement (really, I have done it myself :) )

Re:Wikimedia is moving to Gitlab by GLJHunt on Wednesday October 28, 2020 @06:11PM (#24924693)
Appreciate the response - I'll look into now
Re:Wikimedia is moving to Gitlab by mpol on Wednesday October 28, 2020 @07:11PM (#24925232)
Okay, thank you. I see Gitlab is mostly Ruby. Just to get a general idea of the code this is a simple PHP function to use it:

https://plugins.trac.wordpress.org/browser/gwolle-gb/trunk/f...

That function can be called when the register form has been submitted. It will return true or false. Forget about the transient stuff, that is just WordPres caching stuff.

You don't need an API key like with Akismet. You would only need it if you want to add or remove entries from the SFS database. It really is much simpler. Ofcourse you might want to have a checkbox in the settings. But still, in an afternoon you might be able to finish this :)

Wish you the best.

Re:Wikimedia is moving to Gitlab by lbierner on Wednesday October 28, 2020 @08:35PM (#24925814)
Great suggestion, this looks like a very straightforward service and implementation. All open source as well.
Re:Wikimedia is moving to Gitlab by sytse on Wednesday October 28, 2020 @01:00PM (#24921180)
I think the code of this problem is that it is hard to identify if a user is a bot or a human. I've not seen any elegant free solutions to this.
Re:Wikimedia is moving to Gitlab by ran3824692 on Wednesday October 28, 2020 @01:27PM (#24921577)
That is not the core of the problem. Spammers are humans, and sometimes they will solve recaptchas in large quantities to get their spam through. Its about having a multipronged approach for administrators to stay ahead of them. For some examples of free solutions see https://www.mediawiki.org/wiki/Manual:Combating_spam. It's even possible to connect spamassassin to forms. Gitlab needs tools and automation that detects and rolls back spam, bans users, knobs to tune restrictions and rate limits based on how spammers are acting. Gitlab inc just hasn't seemed to care much to help people trying to use Gitlab and keep their software freedom.
Re:Wikimedia is moving to Gitlab by sytse on Wednesday October 28, 2020 @01:49PM (#24921887)
I think the focus of our Trust and Safety team has been on GitLab.com and not on all GitLab instances. We'll discuss changing this.
Re:Wikimedia is moving to Gitlab by ran3824692 on Wednesday October 28, 2020 @02:11PM (#24922159)
Thank you.
Re:Wikimedia is moving to Gitlab by edchan on Friday October 30, 2020 @01:28AM (#24939442)
GitLab team member here. We just added a new page to our Handbook where we share approaches to preventing, detecting and mitigating spam on self-managed instances of GitLab. https://about.gitlab.com/handbook/engineering/security/opera...

We want to hear from you! Instructions on how to contact us: https://about.gitlab.com/handbook/engineering/security/opera...

Re:Wikimedia is moving to Gitlab by robotmay on Wednesday October 28, 2020 @02:51PM (#24922645)
I'm curious about the spamassassin integration. Do you know of any open source projects currently using it for a web application?
Re:Wikimedia is moving to Gitlab by trynewideas on Wednesday October 28, 2020 @06:17PM (#24924737)
I'll be curious to see whether they even use GitLab user auth. For Gerrit (and Phabricator), Wikimedia already requires contributors to have a dev account on Wikimedia's LDAP system: https://wikitech.wikimedia.org/wiki/Help:Create_a_Wikimedia_...
Re:Wikimedia is moving to Gitlab by jefftk on Wednesday October 28, 2020 @12:19PM (#24920598)
Can you say more about how it's a problem if people can view things without logging in? Naively I would have seen that as a plus.
Re:Wikimedia is moving to Gitlab by dfabulich on Wednesday October 28, 2020 @12:44PM (#24920943)
If you allow new users to create user profiles with links, and those user profiles are visible to Google, spammers will create a bunch of new user accounts and fill them with spam links.

The easiest way to prevent this is to block Google from seeing user profiles by requiring login to see the profiles.

Re:Wikimedia is moving to Gitlab by chrisweekly on Wednesday October 28, 2020 @01:37PM (#24921720)
> "and those user profiles are visible to Google"

googlebot adheres to robots.txt, right?

in which case couldn't self-hosted gitlab admins add a robots.txt entry for the profile page url?

Re:Wikimedia is moving to Gitlab by jefftk on Wednesday October 28, 2020 @02:21PM (#24922276)
That requires the spammers to notice that it's blocked in robots.txt, which seems optimistic
Re:Wikimedia is moving to Gitlab by pitay on Wednesday October 28, 2020 @05:28PM (#24924333)
There is also adding the rel="nofollow ugc" to user submitted links that removes the benefits of linking for spammers.

https://support.google.com/webmasters/answer/96569

Re:Wikimedia is moving to Gitlab by dspillett on Wednesday October 28, 2020 @03:02PM (#24922781)
Most bots aren't going to bother checking if you've done that, so while they'll not get the expected benefit you'll still get the spam.
Re:Wikimedia is moving to Gitlab by jancsika on Wednesday October 28, 2020 @12:54PM (#24921091)
Because spammers fill the publicly viewable things-- like snippets and user profiles-- with spam. If it can be viewed without logging in, then they get it indexed with Google and it dilutes the search results.

Nobody views snippets and user profiles as a common part of daily development, so it takes time away from development to investigate those things to prune them. And if you don't prune it fast enough, it gets into the search results at which point it's even more of a pain in the ass to remove (even using Google's webmaster tools).

Re:Wikimedia is moving to Gitlab by dragonwriter on Wednesday October 28, 2020 @01:07PM (#24921295)
> Because spammers fill the publicly viewable things-- like snippets and user profiles-- with spam. If it can be viewed without logging in, then they get it indexed with Google and it dilutes the search results.

Isn't then the problem not that it is viewable but that it's not excluded from indexing by robots.txt?

Re:Wikimedia is moving to Gitlab by LeifCarrotson on Wednesday October 28, 2020 @12:54PM (#24921089)
Anything visible without login will be visible to people who want to follow a bare URL to your tracker and visible to a search engine crawler, making it visible to people without a URL who just search for the issue. That is, indeed, a plus.

But even if you require login to post stuff to the issue tracker, creating a login and posting a comment has been trivially automated.

You're no longer running a useful issue tracker, you're running a free ad network: you're hosting a dozen useful issues and a thousand advertisements and blackhat SEO comments for spammers.

If it's not visible to search engines, and your repo doesn't get much traffic, it's not nearly as valuable to spammers. It's kind of like cutting off your nose to spite your face, but those basic economics of cost to spammers, cost to users, value to spammers, and value to users are the only rules that you can really apply when hosting content on the Internet.

Re:Wikimedia is moving to Gitlab by jefftk on Wednesday October 28, 2020 @01:06PM (#24921283)
> creating a login and posting a comment has been trivially automated

Isn't this what various CAPTCHA tools handle?

You can also require email address validation.

Re:Wikimedia is moving to Gitlab by marcinzm on Wednesday October 28, 2020 @12:24PM (#24920660)
1. Create user account

2. Create spam content (snippets, profile, etc.)

3. Get spam content search indexed

4. Profit

Re:Wikimedia is moving to Gitlab by dtech on Wednesday October 28, 2020 @12:39PM (#24920873)
a free user sign up is not going to prevent scraping...
Re:Wikimedia is moving to Gitlab by cortesoft on Wednesday October 28, 2020 @12:44PM (#24920938)
I think the idea is that if you can't view issues without logging in, then google won't index your issues (because it can't view them), so you won't get people spamming in order to get into google
Re:Wikimedia is moving to Gitlab by noizejoy on Wednesday October 28, 2020 @04:19PM (#24923662)
Over the years I've frequently seen Google search results showing things that require login to the indexed site. Has that changed?
Re:Wikimedia is moving to Gitlab by marcinzm on Wednesday October 28, 2020 @07:28PM (#24925351)
I believe that requires the site giving google’s bot a logged in view rather than something google does themselves.
Re:Wikimedia is moving to Gitlab by wolco2 on Wednesday October 28, 2020 @02:20PM (#24922258)
It also means 95% of people will stop casually viewing issues.
Re:Wikimedia is moving to Gitlab by wyldfire on Wednesday October 28, 2020 @12:44PM (#24920952)
The problem isn't scraping, it's spam.
Re:Wikimedia is moving to Gitlab by ognarb on Wednesday October 28, 2020 @12:29PM (#24920732)
Strange I never saw this behaviour on our Gitlab instance invent.kde.org.
Re:Wikimedia is moving to Gitlab by marcinzm on Wednesday October 28, 2020 @02:46PM (#24922585)
You seem to be using a central login system (https://identity.kde.org/) that requires going to a separate website to create an account which presumably is non-standard enough to throw off most bots.
Re:Wikimedia is moving to Gitlab by ran3824692 on Wednesday October 28, 2020 @01:01PM (#24921205)
invent.kde.org uses the nonfree google Recaptcha, that prevents it mostly. Not very nice for KDE to make people run nonfree software blob in their browser that gives up their freedom, gives up their privacy to google and trains Google's proprietary machine learning models.
Re:Wikimedia is moving to Gitlab by mattl on Thursday October 29, 2020 @12:27PM (#24932235)
Where does it use that?
Re:Wikimedia is moving to Gitlab by wdb on Wednesday October 28, 2020 @02:20PM (#24922255)
Personally, I also think, GitHub's Checks API, and it Github bots. Gives a much better experience compared to GitLab.

On a daily basis I am confused about how diff's are being rendered in GitLab merge requests, as it has a weird way to render ${blah} in strings in lines.

Also when you want to check if an issue exists in GitLab own repository, you always end up in some jungle of redirected tickets. Just now I got redirected to three different tickets , as they switch projects or something. Really annoying.

Re:Wikimedia is moving to Gitlab by boleary-gl on Wednesday October 28, 2020 @02:27PM (#24922349)
Can you share an example or screenshot of what you mean around the rendering issue in merge requests?

As for the issue redirects - it does stink. It's an artifact of our move to a single code base for CE and EE [1]. A lot of issues have long-standing SEO so the "old" issue often comes up in a Google search.

[1] https://about.gitlab.com/blog/2019/08/23/a-single-codebase-f...

Re:Wikimedia is moving to Gitlab by wdb on Friday October 30, 2020 @01:09PM (#24944391)
Yes, for example, in today's pull request, I added a new file and now in my Typescript file it renders things like this, which I find confusing: https://imgur.com/a/C82hcMK
Re:Wikimedia is moving to Gitlab by zeeZ on Wednesday October 28, 2020 @12:35PM (#24920827)
In general settings, you can check "Public" under "Restricted visibility levels". According to the blurb, "selected levels cannot be used by non-admin users for groups, projects or snippets".

Is that not what you want with #2?

Re:Wikimedia is moving to Gitlab by jancsika on Wednesday October 28, 2020 @12:46PM (#24920980)
It's what I want for #2, but it has the unfortunate side-effect of restricting visibility for my main public repo. My #1 goal above is for people to be able to clone from my main repo without logging in.
Re:Wikimedia is moving to Gitlab by zeeZ on Wednesday October 28, 2020 @12:57PM (#24921136)
I see. And as soon as you make the repo public you end up with public issues again, unless you restrict them to project members...

And a workaround of auto syncing just the code to a public repo where issues and stuff is disabled isn't available natively in CE.

Re:Wikimedia is moving to Gitlab by mgbmtl on Wednesday October 28, 2020 @06:31PM (#24924890)
If you can, place your Gitlab CE instance behind an LDAP server. Have another site handle signups. (admitedly, setting up something with LDAP is often a massive pain. I duct-tape around it by using LdapJS on top of a CMS)

I've had a handful of projects where human spammers will bother to create an account and jump through the loops, but in the 2-3 years of running a Gitlab instance, which has 1300 users, I only had 2-3 incidents (we keep an eye on recent projects, snippets, etc).

Re:Wikimedia is moving to Gitlab by chillfox on Thursday October 29, 2020 @09:30AM (#24930242)
The GitLab LDAP config is pretty easy.
Re:Wikimedia is moving to Gitlab by kodah on Wednesday October 28, 2020 @03:52PM (#24923378)
I would encourage folks to look at Gitea.io. I run that on Kubernetes alongside Drone and it basically replicates all the most important parts of GitHub.
Re:Wikimedia is moving to Gitlab by ben0x539 on Wednesday October 28, 2020 @01:06PM (#24921291)
You'd think Wikimedia in particular has experience with the issue of spam bots polluting the site's search result.
Re:Wikimedia is moving to Gitlab by abbe98 on Wednesday October 28, 2020 @01:21PM (#24921489)
Is this mainly a concern for the Gitlab issue tracker? Wikimedia will continue to use Phabricator for issue tracking, Gitlab CE will only be used for CI and code review/hosting...
Re:Wikimedia is moving to Gitlab by ran3824692 on Wednesday October 28, 2020 @01:34PM (#24921682)
No. Spammers will create repos and user profiles and snippets and anything they can with spam in them.
Re:Wikimedia is moving to Gitlab by abbe98 on Wednesday October 28, 2020 @05:33PM (#24924367)
I would imagine authentication being done through Wikimedia's existing LDAP or Mediawiki solution and I hope that features that already exists in Phabricator(such as snippets) will be disabled.
Re:Wikimedia is moving to Gitlab by oconnor663 on Wednesday October 28, 2020 @12:13PM (#24920521)
Is it possible to configure a robots.txt file to accomplish #2?
Re:Wikimedia is moving to Gitlab by clscott on Wednesday October 28, 2020 @12:20PM (#24920606)
No, robots.txt is for well behaving bots like bing bot and google bot, not bots that will spam your forums (and Git repo apparently).
Re:Wikimedia is moving to Gitlab by detaro on Wednesday October 28, 2020 @12:28PM (#24920719)
It doesn't stop the spam from being created, but it does stop the spam from ruining your site reputation in search results was the suggestion I guess.
Re:Wikimedia is moving to Gitlab by _ikke_ on Wednesday October 28, 2020 @04:46PM (#24923942)
But the goal is to prevent spam in the first place. I don't think these bots will verify robots.txt to see if the spamming is effective. They just spam anything they can get their hands on.
Re:Wikimedia is moving to Gitlab by remram on Wednesday October 28, 2020 @06:09PM (#24924669)
They probably don't have general code that spam any form, it's more likely that they have code specific to GitLab CE instances that knows to post snippets. If GitLab changes their default configuration so that those snippets are no longer indexed by Google, the spammer are likely to stop using that GitLab CE spamming script, after a while.
Re:Wikimedia is moving to Gitlab by marcinzm on Wednesday October 28, 2020 @12:27PM (#24920705)
But if the issue is SEO bots then robots.txt would block the search engines thus meaning the spam content is of no importance (it's effectively private) and doesn't cause issues for the main sites SEO (nor help the spammers).
Re:Wikimedia is moving to Gitlab by jancsika on Wednesday October 28, 2020 @12:43PM (#24920927)
To be honest, I'm not certain of the purpose of the spam.

Some portion of it would end up in the search results, sure.

But I don't know if there's some secondary benefit to, say, a casino showing a link coming from my site even if my site has a robots.txt saying that the address for that link isn't to be directly indexed.

Is there such a benefit? If not then I'll just set up the robots.txt and observe whether that does indeed solve the problem. But I'd much prefer to just set up the permissions I know I want on my own running instance than spend time making inferences about the reasons bots are abusing my instance's inputs.

Re:Wikimedia is moving to Gitlab by jancsika on Wednesday October 28, 2020 @12:33PM (#24920800)
That's right. I'm talking about SEO spam. Basically anything that has a url where the content includes input from the user will be spammed.

I'm fine with, say, the spammers hammering the main repo's merge requests and issue tracker. Those are things any healthy project will check regularly-- I'm even fine just pruning the spam there by hand (and historically I haven't gotten a lot there anyway).

But I don't regularly look at the global view of snippets, and I don't want to regularly prune the global user list for SEO spam in the user profiles. There's no good reason most FOSS projects need those things to be publicly viewable, anyway. But AFAICT Gitlab's admin settings only have a single setting that affects all these things across the board. So if you make snippets viewable only to logged in users, then nobody can clone from the main repo without logging in.

It's quite frustrating, and Gitlab shows no interest in disabling or hiding features like snippets and user profiles.

Re:Wikimedia is moving to Gitlab by tylersmith on Wednesday October 28, 2020 @12:26PM (#24920691)
robots.txt is nothing more than a request. It's essentially a "no dog on lawn" sign in the yard of a vacation home.
Re:Wikimedia is moving to Gitlab by tyteen4a03 on Wednesday October 28, 2020 @02:42PM (#24922541)
I wonder how effective QuestyCaptcha would be on GitLab.
Re:Wikimedia is moving to Gitlab by bawolff on Wednesday October 28, 2020 @08:55PM (#24925930)
Generally speaking questycaptcha works great if you fly under the radar enough that nobody is putting any effort in, and its just automated bots. It tends to fall apart the more high profile you are.
Re:Wikimedia is moving to Gitlab by eznzt on Wednesday October 28, 2020 @03:15PM (#24922947)
Can't you just put the gitlab instance behind an nginx proxy to achieve this? Like, if you are requesting ^/user/$, check for a cookie; if invalid, return 403
Re:Wikimedia is moving to Gitlab by 908B64B197 on Wednesday October 28, 2020 @12:24PM (#24920659)
I honestly can't see why someone would go through the trouble of making sure their instance is correctly configured and available when there are solutions (like GitHub) that just work out of the box.
Re:Wikimedia is moving to Gitlab by paulryanrogers on Wednesday October 28, 2020 @12:33PM (#24920794)
Control. Github can change the behavior, pricing, availability, or security of their offerings at any time. If they get hacked then you could suffer. GH is also closed source.

For many a self hosted solution is better, despite the costs.

Re:Wikimedia is moving to Gitlab by modoc on Wednesday October 28, 2020 @02:30PM (#24922392)
It's not a lot of trouble. To be fair my GitLab instance isn't setup for large numbers of public contributors. I have a somewhat limited network connection and work on projects that often have large-ish codebases, building Docker images, etc.... and I do 90% of that on my home network (local servers, storage, etc....). So running GitLab locally allows me (and a few other folks) to get all those nice features without relying on the world facing internet connection and without having lots of delays moving large files up and down...
Re:Wikimedia is moving to Gitlab by 908B64B197 on Wednesday October 28, 2020 @04:50PM (#24923989)
That's the use-case I can understand (if someone has a large number of machines or a fast internal network behind a VPN). My comment was more aimed at public facing projects that are open and accept contributions from anyone.