Of course, that means you'll need to at least spot check your bans because you can't rely on legit users escalating to you.
The thing is, the people that weren't the ones posting the content. It appeared their computer was affected by some type of malicious file to be part of a bigger network. (botnet?)
I could see from the thousands of different IPs in different countries around the globe, that it could be affected personal computers.
Very few of them were computers from hosting companies, the rest were normal people computers.
I'm sure these machines were doing the job, someone else would have tests the result.
When we did the shadow banning it didn't made a dent in their effort.
The way they changed email, changed username, tried to be unique was completely prepared specifically for our platform (I would guess so)
Whenever we counter their attack, they would be silent for a while and then attack again. They would adjust.
Shadow ban is effective when the attacker themselves will not be aware it, in our case it was tricky to know who was the observer.
More shady but probably legal ones use JavaScript and Apps that typically hide the fact you are becoming an endpoint in the T&Vs.
Straight up illegal ones use compromised computers and routers like you'd think.
Very few of them were from AWS, OVH and other hosting providers, very very few.
We ran each IP towards black lists ips, paid IP reputation checkers. Majority of the IPs were clean.
Back then we had IP reputation check, but it was a headache to maintain, so we disabled it later, however that time very very few them got stopped at IP reputation checks.
Fighting against abuse at the level of IP address attributes seems like a losing game to me. Honestly, the best I saw at this (3-5 years ago at least) for traffic was Distil networks, where they put a proxy device in front and examine your traffic and captcha or block based on that.
Since you have content being submitted, there's a lot more you can use to classify, such as how you used ML, so that's good. Part of me worries that this is all sort of reminiscent of infections and antibiotics though. The continual back-and-forth of you finding a block them finding a workaround feels kind of like you were training the spammers (even if you were training yourself at the same time). At some point maybe we'll find that most the forum spam is ML generated low information content posts that also happen to be astroturfing that is hard to distinguish from real people's opinions.
1: Fun fact, to my knowledge anonymous mobile IPs are provided by a bunch of apps opting into an SDK (like an advertising/metrics SDK) which while their app is open (at least I hope that's a requirement) registers itself to the proxying service so it can be handed out for use by paying proxy customers. Think about that next time you play your free "ad-supported" mobile game.
It took me a while to realize that ramping up the frustration level is, itself, a helpful deterrent.
The decision from sircmpwn was, at the end, to charge money for the service. Charging money and KnowYourCustomer will kill most exploits dead.
In this sense, this is turning the frustration level to 11. You can use the service to a certain extent, without frustration, but if you want to get serious then you're going to have to jump through some hoops.
Dedicated people will still find a way through, but you've cut off 95% of the flow and killed the low-effort attempts. Now, you can focus on the serious shit.
The risk of having their payment identifier/address banned from services before significant use makes it very risky for them to use such a thing even if tiny micropayments a worth it to the spammer.
It certainly could have other problems with people getting banned from such a system for things other than: spam and other use detrimental to the service provider. There is also the issue of how the initial buy in fee is distributed.
But a high buy in for a system that many online service providers use would very strongly discourage use detrimental to the services providers (I think; this is only for discussion, as this is posted by someone with little knowledge on this. Micropayment systems have been talked about a lot, but I don't remember high buy-in mentioned).
Edit: Forgot to mention that the idea of this is that service providers can offer their services at lower cost because the risk to them from a account/address with a high buy in is lower than from an account/address with no buy in.
Frankly, I'm shocked the other major free providers (GitHub/Lab) haven't done this by default. GitHub's current default is free for public branches, and a small fee for private.
I could see a flipped setup working: by default a very small fee (1-5 cents per x number of batches) that most companies wouldn't notice, and a path for FOSS projects to apply for credits.
I can't find the direct link, but I remember someone on HN pointing out that because CI tools are turing complete, GitHub actions is the cheapest serverless cloud product in the world right now -- you just need to figure out how to game the system.
I'm sure they've built very sophisticated filtering tools, but imagine someone slips through the cracks and gets a cryptominer working. Get that action registered in enough projects (by, say embedding it in an Actions library or generator tool) and that could be significant.
It's been done, multiple times. Here's a handful (96) of documented cases which are somewhat recent. [0][1]
It seems to be surprisingly easy to abuse the process, and GitHub are continually playing catch up.
[0] https://dev.to/thibaultduponchelle/the-github-action-mining-...
[1] https://www.bleepingcomputer.com/news/security/github-action...
If possible, it can help to hold back new systems and release a bunch of orthogonal anti abuse systems at once. Then the attackers need to find multiple tweaks instead of just evading one new system.
Just present bisected realities to spam watching groups and identify those who trigger a reaction.
If resources are free then you could even actually deploy their app and either whitelist it for their own IP or only allow very few requests before taking it down.
This would be even more frustrating and could ruin whatever they plan to do with their abusive app in the first place. Let's say they deploy their malware/phishing page, test it a couple of times (possibly from a different IP) and it works. They then start spamming the malicious link and waste decent amounts of time/money/processing power, not realizing that the link was dead after the first 10 hits.
We also get the less resource intensive, but still harmful abusive apps that port scan the internet. Those are relatively easy to detect. We generally don't want to be a source of port scans so we shut them off pretty quickly.
For example, I quickly searched the ToS for port scanning and didn't find it banned, yet you admit to intentionally frustrate such users.
I am not Fly.io customer at this time but I do port-scan my own services as a realtime security check.
As otherwise great service, you may open yourself for very angry user reviews if the tactic is "frustrate the user".
The idea of running UAT for spammers/scammers/troll-farms is pretty depressing.
Then there are others where its only an hour or so before rates get adjusted to our threshold and/or new IPs start emitting the same requests.
https://github.com/wanderingstan/Confusables
E.g. "𝓗℮𝐥1೦" would match "Hello"
Maybe somebody else on here will see it and learn about it before they need it, and at least you still have a new tool to reach for in the future.
>>> import unicodedata
>>> unicodedata.normalize('NFKD', '𝓗℮𝐥1೦𝗵𝗲𝗹𝗹𝗼')
'H℮l1೦hello'
NFKC/NFKD will handle "this is another form of the Latin letter A" type stuff but not "Cyrillic A looks like Latin A."
Back in 2015 I did some work just using simple bitmap rendering plus OCR to find text that looked like a small selection of known words. It was actually reasonably effective.
Unicode Character “;” (U+037E) is a beaut.
The initial problem wasn't those symbols but the content itself, the symbols and special characters came into the problem later.
Later on as mentioned in my original comment, that they would use positive content from other blog posts that were published/passed the moderation to mix up their bad content.
Probably could use a different method, but at that time needed something quick and fast and it worked and still works with very little tweaking.
Although we don't have massive amount of threats or abusers anymore to exactly know the effect, but again, so far it works.
That time, they would coming several thousands per minute, IP blocking, range blocking, USER AGENT, captcha or anything such didn't work on them.
I guess I can use that next time time to work on the data cleaning for that model.
Thanks.
I built a Python library for finding strings obfuscated this way. Was critical when moderating our telegram channel before an ICO. https://github.com/wanderingstan/Confusables E.g. "𝓗℮𝐥1೦" would match "Hello"
I think that depends on the users. People copying and pasting bits of text that was in English or another common language— think documentation, code, news articles, tweets, etc.— with a different character set could be problematic.
Also, 𝒮ℴ𝓂ℯ 𝒜𝓅𝓅𝓈 marketed as "𝔽𝕠𝕟𝕥𝕤 𝕗𝕠𝕣 𝕤𝕠𝕔𝕒𝕝 𝕞𝕖𝕕𝕚𝕒" would be ℭ𝔞𝔲𝔤𝔥𝔱 𝔲𝔭 𝔦𝔫 𝔱𝔥𝔦𝔰. (math symbols) A user base with young people getting bounced or shadow banned for trying to express themselves or distinguish themselves from their peers would be like ಠ_ಠ (Kannada letter ttha)
I think targeting the language they're using is a better bet.
¯\_(ツ)_/¯ (Hirigana letter tsu)
Minor nitpick, but ツ is the katakana tsu.
If they use many (maybe three? four? or more) character sets in the same post, or different character sets in any single word, then that'd be highly suspicious?
Whilst still letting people copy paste from another language
Special case needed for the shoulder shrug with an Hirigana letter tsu I mean katakana tsu
Sounds like you live in a filter bubble.
(╯°□°)╯︵ ┻━┻
(ﾉ◕ヮ◕)ﾉ*:･ﾟ
They kinda do. Check out the shrug "emoji", table flip, and so forth. Then there's the meme of adding text above and below by abusing Unicode's "super" and "sub" modifications.
You could block it to only ever represent ASCII, but then you've knocked out the ability to expand internationally.
I wonder if you can train a ML model using text as images. For example, taken as strings, "porn" and "p0rn" are not very similar, but visually they are.
https://www.theregister.com/2013/08/06/xerox_copier_flaw_mea...
Nothing fancy tbh.
For the first several deployment while taking care of edge case and debugging, all manually on my own laptop and shot into cluster as a docker image.
Later, when starting to classify more content on the platform itself:
- Webhook will trigger the CI to train the model with new ham and spam content. - A new docker image was created and deployed to container.
The above would fail if success rate in validation/testing would be below 95%.
Still when classifying bad content, the whole process happens automated.
Special thanks to Gitlab CI, shell scripts, Python and Docker Swarm.
If I'm a real human and trying to post a "good" post, but the model classifies it as bad and automatically blocks it, how do I appeal that decision? Can I? Or is my post totally blocked with no recourse?
Thanks for clarification.
When a post gets published, it will be send to machine learning image via REST.
If bad, the post will be kept as Draft.
A new record gets created in another database table to keep track them, the accuracy rate was recorded as well.
This was made to make sure no irreversible action was done on the good content.
Blogs with more than 1 year of history would not go through moderation but no action was being taken, just recording the accuracy for future reference.
Later, someone from our team (me usually) would check them by eye and pull trigger on them, they would go into make the training better.
If something would pass the moderation but it was indeed spam, would go into another iteration.
We had to do this for over a month, through the time, the success was around 99%, no blogs would be wiped by machine classification from our database unless confirmed by someone.
That time the whole model was trained for that specific content. Later it get into other type of spams. Which we trained different models.
Overall, the the machine actions were logged, content/users/blogs would get labeled and bad marks on them.
They would be displayed in a report page, until someone make the final decision, through the whole time, the user would be shadow banned (shadow banning didn't help though) and their content would not be published.
And there's a lot of overlap between how that system acted and what you're describing. It makes we wonder if there's space for a company that offers this sort of model training + content tagging + review tooling capability as a service, or if there's too many variation on what "good" and "bad" input is to make it generalizable.
Yes.
They would create this posts and get quickly on search results (The platform is pretty good for making SEO optimization out of the box) and they would write good quality posts as well.
They would also share this posts on some other websites, especially social media accounts.
We don't have Google analytics or such to see where exactly they would come from. I noticed huge traffic to such pages by looking at the logs.
Our nginx log parser was alerting us about sudden spike on certain blogs and pre-defined list of words we have.
That's when we noticed something is going on.
Didn't take more than couple of hours (while working on the model) that we receive email from data center people about hosting phishing content, again didn't take much longer we received emails from some of those companies as well.
> Were they embedding malware into your blogging platform, or getting ad revenue out of this somehow?
No. On the blogging platform, we have everything bleached out, nothing would go in without passing through sanitizers.
They would simply had people convinced to call those US numbers.
I actually called one of those numbers and yeah, it was one of those customer supports some other part of the planet earth and definitely not from the company he was pretending to be and very quickly asked me to install team viewer on my machine. I really wanted to let them access it via the windows on my virtual box and have some fun with them, but well, someone had to fix the moderation issue :D
We banned, shadow banned, deleted, recorded and many other things to them to make sure they're not reaching their goal.
The problem was, it wasn't a single user to deal with. thousands and thousands with no stop.
Flooding the platform like zombie apocalypse.
Were these all using your free plans?
Never ending.
We still keep the free plans even through there are abusers, but that won't be a reason to retire it. So many people using for legitimate reasons and keeping their personal writing there.
It's unfortunate, but well, it happens.
Is it because these scammers were non technical?
Or maybe the anonymity provided by your platform?
1. Highly technical, because the flow was scripted to work with our website. Bypassing captcha, email verification by using many different domains, email accounts and also highly distributed via many ips.
2. Non technical. Where they paid some ppl to do it manually, which doesn't seems to be due to the way they walk through many steps like piece of cake.
However, our platform was/is a target due to several reasons:
1. Easy to register and start blogging. 2. Free plan with no hard limit. 3. Quick rankings due to SEO implementation out of the box. 4. Absence of any moderation before such attack.
And probably some other possible reasons that made their job easier and us a better target.
Here's the old source I have on my computer that did the training
https://gist.github.com/Alir3z4/6b26353928633f7db59f40f71c8f...
This was doing the early work and later changed more to fit other cases.
Pretty basic stuff.
Basically, pulled the database into CSV file and anything that was published before the bad content was classified as HAM.
We had content that were OK, so marked as HAM and then our new bad content all marked as SPAM.
When deployed to production for some hours HAM content got wrongly marked and model got trained on them as well which made so many confusion but the problem was taken care of once the model got properly tuned and safer to let it be automated.
but don't think it will be useful for everyone, each usage and requirements is different and needs different solutions.
On the front FastAPI (behind uvicorn) to accept calls via REST API.
Deployed via docker.
To be honest, tensorflow and scikit-learn may not be the right fit for everything.
Every situation needs different approach and different solution.
Worth nothing, the most time consuming part was dealing with data itself and not model training or machine learning.
In couple of hours you'd notice you're starting at charts and tuning parameters.
Become a data scientist they said. Yay, artificial intelligence...
It's not that the content will be popped up to everyone when someone posted something.
There's a Feed page where you can read what others you follow have published.
There's a Explore page where latest content without any filter or categorization would be visible. This is where such content would appear, but only blogs older than 7 days would appear there (we have removed that delay in recent versions).
Basically no one noticed them.
Although we disclosed the issue we were dealing with some of the old users of platform when they complained about their posts not getting published. That was the first issue in first 15 minutes of the machine learning model classifying wrongly due to being fed mixed content (where bad content was mixed with good content from those exact blogs by spammers.)
Other than several bloggers reporting their posts wouldn't go through as expected, no one else got effected and I hope no people were lured with those scammer while their content was published on our platform.
The IP blocking was in place on application level and later into Cloudflare blacklist. Still they would flood in with different IPs and browsers.
Rolled out a machine learning model and trained it on the database. 99% of them vanished.
Next day, the machine didn't work and success rate was around 5%.
Found out, they have learned the trick and now using symbols from different languages to make it look like English.
Trained again, success rate went up again.
Next hour, success rate fallen.
This time, they mixed their content with other valid content of our own blogging platform. They would use content from our own blog or other people posts and mix it to fool the machine learning.
Trained it again and was success.
Once a while such content appear and machine model fails to catch them.
It only takes couple of minutes to mark the bad posts and have the model get trained and redeployed and then boom, bad content is gone.
The text extraction, slicing through good content and bad content, finding out symbols vs sane alphabet and many other thing was at first challenging, but overall pretty excited to make it happen.
Through this we didn't use any platform to do the job, the whole thing was built by ourselves, little bit of Tensorflow, Keras, Scikit-learn and some other spices.
Worth noting, it was all text and no images or videos. Once we got hit with that we'll deal with it.
[0]: https://www.gonevis.com
edit: Here's the training code that made the initial work https://gist.github.com/Alir3z4/6b26353928633f7db59f40f71c8f... it's pretty basic stuff. Later changed to cover more edge cases and it got even simpler and easier. Contrary to the belief, the better it got, the simpler it became :shrug