News for hackers. Stuff that matters.
Re:Gitlab Project Hit by Spambot by jitbit on Monday February 08, 2021 @10:55AM (#26066164)
We are a small SaaS vendor and fighting spam is, like, 20% of the technical work that we do. Bad actors register trial accounts and then reverse engineer your front-end and forge millions of requests. Just 2 days ago we had a very similar issue [1]

We have all sorts of protection: rate limiting, crippling non-paid accounts, detecting if a trial signup comes from a VPN/Tor (+5 to "suspicious" score!) etc... But they still manage to find ways. It's a never-ending battle. And the saddest part is - all this hard work is completely invisible to our existing paying customers :(

If your app has an email-sending module of some sort it WILL be abused. Even a trivial "reset password" form is a target.

My support goes to the GitLab team. Good luck and no hard feelings.

[1] https://www.jitbit.com/news/5354-spammer-attack-post-mortem/

Re:Gitlab Project Hit by Spambot by rectang on Monday February 08, 2021 @11:57AM (#26067044)
Imagine a world where spammers could be identified, prosecuted, and convicted reliably. The amount of economic energy unleashed would be staggering.

We are still in the early days of the internet. It is a frontier territory where crime is rampant.

Re:Gitlab Project Hit by Spambot by sneak on Tuesday February 09, 2021 @07:41AM (#26076783)
Do I have a minority opinion for believing that spam, even coordinated, ddos-level, business-destroying professional-grade stuff is not a severe enough problem to send men with guns to a place to lock a human being into a cage?

I don't want the government prosecuting people for speech, even mass, automated speech. There are much better solutions than resorting to violence.

Re:Gitlab Project Hit by Spambot by rectang on Tuesday February 09, 2021 @09:08AM (#26077584)
Yes, "spammers should not be punished by the government for destroying businesses" is bound to be a minority opinion, even if some among us are sympathetic.

Protocols and system designs which make abuse impractical and thus prevent spam before it starts are surely desirable, although they often come with their own tradeoffs (user friction, privacy, false positives, maintenance costs, etc.). If spammers can be deterred by improving along other axes all the better. But the general problem of spam is that computing resources allow for massive amplification of malicious actions, and that's never going away — it will be an arms race between spammers and their victims forever.

Re:Gitlab Project Hit by Spambot by sneak on Tuesday February 09, 2021 @09:09AM (#26077594)
I didn't claim they shouldn't be punished; I claimed that spamming is not sufficiently damaging for a reasonable response to it to include violence, which is what arrest/prosecution necessarily entails.
Re:Gitlab Project Hit by Spambot by rectang on Tuesday February 09, 2021 @09:41AM (#26078008)
It all hinges on whether or not you can protect the populace and effectively prevent spammers from causing harm. If you can't protect people, you will lose their faith.

As someone who battles spam on the small business front lines, I hate it. It fills me with fear and a sense of loss-of-control, that spammers may be commandeering our systems at any moment, possibly causing us to lose money, time, and reputation, cutting us off from vital services.

It takes hard work for me to overcome that and to recognize the humanity of spammers. Even now, I'm wondering, "Is sneak a spammer?" Rationally I don't believe it, but that's the kind of resentment that builds when people are left unprotected.

Re:Gitlab Project Hit by Spambot by sneak on Tuesday February 09, 2021 @09:45AM (#26078051)
Quite the opposite: I've an opt-in email list of thousands that I've never sent a single mail to because I haven't yet crafted a draft email that I consider to be sufficiently valuable to warrant appearing in someone's mailbox and requesting a few seconds of their time. I've also run my own mailserver since 1997, and have lost dozens of days in my life over the decades to battling spam.

I hate spam; I just hate violence more.

I don't consider solutions to problems that involve violence to be solutions. Violence is, after all, the last refuge of the incompetent.

Re:Gitlab Project Hit by Spambot by ZWoz on Monday February 08, 2021 @12:38PM (#26067580)
Thats about initiatives. Many daily drivers break some laws, like speeding or not using turn signals. Does that make traffic frontier territory? :) Try something serious, like operating black drug market or treatening high level politician (in internet): you going to be surprised, how well different countries co-operate. For example operation involving 30 countries: https://www.europol.europa.eu/newsroom/news/%E2%80%98avalanc...
Re:Gitlab Project Hit by Spambot by malinens on Monday February 08, 2021 @12:01PM (#26067095)
I work for an e-mail provider. This is very painful part of our work. Part of our services are free and ad supported. We are disabling, queueing thousands of mailboxes per day. Captchas and user scoring does not help that much. There are very cheap services were people manually will solve recaptchas/hcaptchas. Even required phone numbers (normal users are pissed when we ask for a phone number to use account fully but we don't have much choice...). There are phone number farms which are used for account verification also
Re:Gitlab Project Hit by Spambot by jdsalaro on Monday February 08, 2021 @01:21PM (#26068099)
> This is very painful part of our work

GitLabSec team-member here, ^ this 100%.

Our Trust and Safety team works really hard to keep up with spam. For those interested in how they do so, you can read more here[1]. We are also working on making our spam detection engine more effective at mitigating high-volume spam incidents such as this.

For current, working product improvements to detect and mitigate spam you and others can check out MRs labeled "spam fighting"[2]. We've had quite a few internal conversations about strategies and best-practices. For more information on how to contact our Trust & Safety Team, including tips on how to deal with abuse in your own instance or steps to suggest spam/abuse related features, see our handbook[3]

[1] https://about.gitlab.com/blog/2020/10/29/how-we-work-to-dete...

[2] https://gitlab.com/gitlab-org/gitlab/-/merge_requests?scope=...

[3] https://about.gitlab.com/handbook/engineering/security/secur....

Re:Gitlab Project Hit by Spambot by zoomablemind on Monday February 08, 2021 @05:20PM (#26071021)
Maybe introduce a Moderator role to the multi-user projects?

That would require the project's moderator to clear the newly posted issues (whatever is possibly a spamming route there) before they propagate. As this is an additional friction point, some bad actors may be intercepted there (say, trying to clear a million issues at once or other equally ridiculous actions).

Of course, by itself it won't stop someone from clearing a single issue to a million of "users". But may still help in id'ing such actors.

Re:Gitlab Project Hit by Spambot by jarym on Monday February 08, 2021 @11:40AM (#26066798)
So these 'bad actors' are they just out to find a way to DoS your service or do they have some other motive for forging millions of requests. (I get that email sending is a target - that has been the case since forever - but I'm asking about 'what else'?)
Re:Gitlab Project Hit by Spambot by jitbit on Monday February 08, 2021 @12:36PM (#26067553)
Yes, they sure do have a motive - to send spam via a 3rd party service.

Like in this case (GitLab) - create a fake "issue" with spammy links in it and then "notify" thousands of users.

In our case - we are a helpdesk ticketing app, so bad actors sign up for a trial, then add "users" to their account and then, say, create "tickets" on users' behalf (customizing the email template by inserting spammy links).

Re:Gitlab Project Hit by Spambot by jdsalaro on Monday February 08, 2021 @11:50AM (#26066949)
It varies: generating inconvenience(for fun or hate towards a particular service), spreading scam URLs(leveraging existing trust relationships), SEO and improving discoverability of their projects (porn, streaming of pirated content, etc) and many more.
Re:Gitlab Project Hit by Spambot by nonbirithm on Monday February 08, 2021 @11:57AM (#26067038)
I wonder if effective DDoS protection today is inextricably tied to having to route your traffic through centralized third parties like Cloudflare.
Re:Gitlab Project Hit by Spambot by freyfogle on Monday February 08, 2021 @12:31PM (#26067487)
Can't upvote this comment enough. Have wasted years of my life on these invisible, but sadly necessary, features.
Re:Gitlab Project Hit by Spambot by gogopuppygogo on Monday February 08, 2021 @11:13AM (#26066396)
Sounds like something a lot of companies need. Could be a good lifestyle startup company.
Re:Gitlab Project Hit by Spambot by nathancahill on Monday February 08, 2021 @11:21AM (#26066496)
Require a credit card on signup.
Re:Gitlab Project Hit by Spambot by wu_187 on Monday February 08, 2021 @11:38AM (#26066756)
That won't work as the same bad actors typically dabble in CC fraud as well. They will typically already have thousands of CC numbers at their disposal.
Re:Gitlab Project Hit by Spambot by f430 on Monday February 08, 2021 @12:42PM (#26067625)
My solution was to block IP addresses from the following regions who were never our target customers anyways:

India, Midde East, Turkey, Russia, China, South East Asia (except Singapore), All of Eastern Europe, Africa and island nations.

It immediately reduced the amount of fraud and spam. Sure you might miss out on the <1% of your revenues but its a tradeoff I'm okay with.

Re:Gitlab Project Hit by Spambot by TheCapeGreek on Tuesday February 09, 2021 @12:08AM (#26074123)
Is the African and island nation traffic that significant and fraudulent?

Having worked in the South African tech scene and for a UK company, every spam/dos detection I've seen has been from everywhere you mention except Africa and island nations.

Re:Gitlab Project Hit by Spambot by f430 on Tuesday February 09, 2021 @02:18PM (#26081425)
not sure what you are trying to say here. Simply blocking all sources of traffic which we don't do business with has been beneficial in stopping spam and frauds.
Re:Gitlab Project Hit by Spambot by bigint on Monday February 08, 2021 @09:53PM (#26073335)
Rude!
Re:Gitlab Project Hit by Spambot by f430 on Tuesday February 09, 2021 @02:17PM (#26081409)
Please explain why you think its rude to block countries thats responsible for most of the spam and fraud.

or is this another one of your alt nick you periodically sign in from?

Re:Gitlab Project Hit by Spambot by sergiotapia on Monday February 08, 2021 @01:05PM (#26067891)
Yep, totally agree. We blocked China (a country we do zero business in) and saw a drastic double digit percentage decrease.
Re:Gitlab Project Hit by Spambot by f430 on Monday February 08, 2021 @08:46PM (#26072852)
weird. not sure why you and my comment is being downvoted.
Re:Gitlab Project Hit by Spambot by dec0dedab0de on Monday February 08, 2021 @11:53AM (#26066979)
Credit card and a delay.
Re:Gitlab Project Hit by Spambot by applecrazy on Monday February 08, 2021 @12:04PM (#26067143)
That would basically destroy user acquisition metrics. As a user, I don’t want to wait days to get access to something.
Re:Gitlab Project Hit by Spambot by dec0dedab0de on Wednesday February 10, 2021 @10:30AM (#26090448)
it doesn't have to be days, most credit card fraud is found within hours. You can also allow partial use of the application during the vetting period, just exclude the parts that are being abused. Though you should probably aim to build a product that people want to sign up for, and are willing to wait for.
Re:Gitlab Project Hit by Spambot by mro_name on Tuesday February 09, 2021 @02:50AM (#26074923)
> acquisition metrics

here's the trade-off. Make you decision.

Re:Gitlab Project Hit by Spambot by LunaSea on Monday February 08, 2021 @01:06PM (#26067915)
Have you tried differentiating ISP vs hosting provider traffic?

Also, you could try to detect headless browsers like Headless Chrome.

Re:Gitlab Project Hit by Spambot by walrus01 on Monday February 08, 2021 @01:49PM (#26068417)
google "residential proxies for sale" - there is a whole huge grey market in selling proxies from peoples' actual residential internet connections, usually to be driven by bots or people in click-farms somewhere in a low labor cost location.

If you want to appear to be a legitimate comcast, charter, centurylink, frontier end user on a last mile broadband connection in a house somewhere in the US48 states, that's a standard service that fraudulent organizations make use of now.

Usually using software that people have been tricked into installing on their computers.

Re:Gitlab Project Hit by Spambot by technion on Monday February 08, 2021 @04:10PM (#26070168)
I don't need to Google them. "Residential proxies for sale" is one of the most common Facebook ads I see. There was one on my screen just five minutes ago. It's staggering that the industry is so blatant.
Re:Gitlab Project Hit by Spambot by darkwater on Monday February 08, 2021 @03:36PM (#26069757)
Exactly, and this is why "in theory" Google reCaptcha, the that is just a checkbox, is a needed solution. If they only used it to just stop spam and not also to track legitimate users...
Re:Gitlab Project Hit by Spambot by pcarmichael on Monday February 08, 2021 @01:35PM (#26068276)
Lots of VPNs route through hosting providers. Some hosting providers are more prevalent with spammers than others. But in the end you can only really use it as a scoring measure. It's not nearly binary enough to straight filter using it.
Re:Gitlab Project Hit by Spambot by LunaSea on Tuesday February 09, 2021 @10:13AM (#26078362)
Agreed but I'd rather lose the small percentage of users on VPNs than have my site flooded with junk traffic and spam
Re:Gitlab Project Hit by Spambot by waihtis on Monday February 08, 2021 @01:44PM (#26068365)
What do the bad actors technically do? Map your api endpoints and spam them with traffic? Something else also?
Re:Gitlab Project Hit by Spambot by ruffrey on Monday February 08, 2021 @01:55PM (#26068508)
Small SaaS vendor checking in. This comment rings true and it’s a constant battle.
Re:Gitlab Project Hit by Spambot by stanislavb on Monday February 08, 2021 @03:19PM (#26069563)
I feel you. Fighting SPAM is eating a serious amount of my energy, too.
Re:Gitlab Project Hit by Spambot by tutfbhuf on Monday February 08, 2021 @11:53AM (#26066985)
Use hCaptcha for sign up and other sensitive parts of your application.
Re:Gitlab Project Hit by Spambot by meibo on Monday February 08, 2021 @12:06PM (#26067162)
I'd rather not use the service if it has hCaptcha OR ReCaptcha.

hCaptcha has bad UX, their dataset is way harder to decipher and commonly takes me three or four tries to get around on Cloudflare, if it doesn't keep failing with a server error which I've had happen on multiple, distinct occasions.

At least for Google, I was done in one turn when lucky, if I had no other choice (e.g. banking).

Re:Gitlab Project Hit by Spambot by contravariant on Monday February 08, 2021 @12:30PM (#26067482)
It seems to have gotten better but I still haven't forgiven ReCaptcha for regularly making me spend ages retrying the unending and excruciatingly slow loading tests.

At least hCaptcha has a decent chance of letting me through if I answer correctly.

Re:Gitlab Project Hit by Spambot by boleary-gl on Monday February 08, 2021 @09:37AM (#26065005)
GitLab team member here.

We're actively working on mitigating this spam attack: https://twitter.com/gitlabstatus/status/1358777539624198145

Re:Gitlab Project Hit by Spambot by xeeeeeeeeeeenu on Monday February 08, 2021 @12:59PM (#26067826)
It's interesting that the nickname of the spammer is "fixspam gitlab". It seems that their intention is to force gitlab to introduce anti-spam measures.
Re:Gitlab Project Hit by Spambot by viraptor on Monday February 08, 2021 @02:14PM (#26068779)
Could be... I've seen this before - people annoyed by continuous low effort spam and the service not fixing it, deciding to show what a targeted attack will look like to force a response. (Not saying that GL doesn't care)
Re:Gitlab Project Hit by Spambot by arusahni on Monday February 08, 2021 @09:37AM (#26064998)
Here's their incident status: https://status.io/pages/incident/5b36dc6502d06804c08349f7/60...

Looks like the immediate issue was resolved a few minutes ago.

Re:Gitlab Project Hit by Spambot by jancsika on Monday February 08, 2021 @02:45PM (#26069156)
I run my own instance of Gitlab which basically means that I handle spam manually. It is a pain.

Gitlab recently added a band-aid of a feature to require admin approval for new sign-ups, which comes with its own obvious problems. But even the UI for that is clunky-- you have to click a button to reveal some options, find the option to delete the spammer, type (read: paste) the users handle as a failsafe, after which Gitlab takes you back to the main user view requiring you to click again on the unverified users tab.

Last time I talked about this problem someone from Gitlab claimed they were working on a feature to send an email with a simple link to click to approve/deny a user. That feature still has not shipped.

They did fix an unrelated catastrophic bug of keeping around artifacts on each branch even if you set an expiry. On a gitlab instance with even moderate development rate and moderate-sized artifacts this fills up the hard-disk with artifacts that the admin must manually delete (or manually create their own script to delete them, which is still a pain). However, their fix was to add a checkbox that defaults to the buggy behavior. Per project. So for every new user's project I have to manually tell them to turn this anti-feature off to keep from filling up the hard-drive with artifact spam. Or else write a damned script to go in and turn off this feature periodically.

The community edition of Gitlab feels a lot like the early Gnome 3 touch-screen stuff. Great in theory, but one quickly gets the sense that no one is dogfooding-- if they did these kinds of problems wouldn't exist.

Re:Gitlab Project Hit by Spambot by xrd on Monday February 08, 2021 @11:35PM (#26073926)
Why isn't there an email provider that allows you to 1) whitelist the content and 2) manually register the to and from addresses and 3) specify how many emails you estimate you will be sending?

At first glance, #1 seems tricky. But, I could see this as an opportunity to bring my users back to the site by providing very specific content teasers.

#2 feels possible in certain circumstances. If you can register the addresses when the user registers, you could avoid problems like changing addresses.

#3 seems like it would be problematic for the email provider who probably doesn't want to build a feature that reduces the ability to bill you. But I would pay for it.

If you had all this, couldn't this prevent the problems described here and incentivize the spammers to find a different, easier target?

Re:Gitlab Project Hit by Spambot by sushikokk on Monday February 08, 2021 @11:01AM (#26066252)
Maybe they flood the mailbox to hide legit mails in between all the spam mails, like password changes or other important notifications?
Re:Gitlab Project Hit by Spambot by CatTheHacker on Monday February 08, 2021 @11:23AM (#26066530)
Happens on GitHub as well, fortunately not with such massive flood
Re:Gitlab Project Hit by Spambot by jeanlucas on Monday February 08, 2021 @10:17AM (#26065599)
Is it spam or a "mail ddos"?
Re:Gitlab Project Hit by Spambot by jdsalaro on Monday February 08, 2021 @10:28AM (#26065784)
Strictly speaking the former. You could, however, say it's both since submitted issues trigger an e-mail being sent to people following the project.
Re:Gitlab Project Hit by Spambot by boleary-gl on Monday February 08, 2021 @10:27AM (#26065771)
Both?
Re:Gitlab Project Hit by Spambot by saurabhnanda on Monday February 08, 2021 @10:38AM (#26065919)
What is this going to do to their Sendgrid/SES bill this month, I wonder!
Re:Gitlab Project Hit by Spambot by nielsole on Monday February 08, 2021 @10:44AM (#26065998)
I would be more concerned with deliverability. Gitlab email notification now has a Spam score of 6.4 (up from 0.0) in Fastmail
Re:Gitlab Project Hit by Spambot by oakesm9 on Monday February 08, 2021 @12:43PM (#26067642)
For context, the default settings are to move anything with a score above 5 to the spam folder.
Re:Gitlab Project Hit by Spambot by markmacardle on Monday February 08, 2021 @01:20PM (#26068091)
what is the motivation behind attacks like this? Like is whoever put in the effort to do this just trying to annoy Gitlab's customers? What do they gain from that?
Re:Gitlab Project Hit by Spambot by talhah on Monday February 08, 2021 @02:44PM (#26069149)
Judging from the example email the name of the spammer is "fixspam gitlab", I'm guessing they want gitlab to fix spam.
Re:Gitlab Project Hit by Spambot by bennyp101 on Monday February 08, 2021 @11:33AM (#26066687)
Interesting that both Gitlab and Github had issues today - coincidence? Maybe Bitbucket will be next!