News for hackers. Stuff that matters.
Re:Cloudflare took down our site after trying to force us to... by pclmulqdq on Sunday May 26, 2024 @10:37AM (#40482505)
As far as I can tell, the issue with this is:

OP runs a casino/gambling site. Gambling is a regulatory mess (I have spent far too long dealing with this as an RNG supplier), and so it's very hard to comply with every jurisdiction, and each one needs you to prove compliance to operate in that jurisdiction.* Gaming companies spend a lot on compliance and tracking, but since the internet is the internet, it's pretty hard to enforce perfectly, so some countries and ISPs take this into their own hands.

Due to that, IPs hosting gambling and gaming sites often get regionally blocked by internet providers or otherwise flagged as hosting illegal content. Those regional blocks consequently affect the reputation score of the IP, and if you are a traffic aggregator like Cloudflare, can cause other customers to have issues. One of the most aggressive and annoying regulatory environments for gambling companies is the US, so it's very possible Cloudflare has had some trouble due to gambling use of their IPs in states in the USA.

Cloudflare wanted them to use the BYOIP features of the enterprise plan, and did not want them on Cloudflare's IPs. The solution was to aggressively sell the Enterprise plan, and in a stunning failure of corporate communication, not tell the customer what the problem was at all. The message from Cloudflare should have been "Enterprise plan + BYOIP or ban, and maybe we'll work with you on price" but it was instead "you would really like the Enterprise plan."

*As an aside - we're lucky in that respect being a tech supplier with relatively uniform rules, but our customers (the gaming companies) get the short end of the stick here.

Re:Cloudflare took down our site after trying to force us to... by nolverostae on Sunday May 26, 2024 @11:45AM (#40483063)
BYOIP is reasonable, though I doubt anyone actually does legislation blocks by IP. Since like half of companies on the internet use Cloudflare or other multi-tenant infrastructure everyone is aware that you can't block an IP address and hit one target. The only thing I've seen is DNS blocks (both DNS protocol directly and based on TLS SNI).

FYI, we also fully block users from the US (due to regulations).

My problem here is mainly the unprofessional communication and huge mess of mixing "compliance" with sales, without giving any clear information or options. And then the removal of our account without warning while we were still talking to them.

Re:Cloudflare took down our site after trying to force us to... by pclmulqdq on Sunday May 26, 2024 @12:13PM (#40483268)
You would be surprised how big of a hammer ISPs will use when they are told to hit something. They live in a very different world than many modern web software companies - they are the plumbers for lots of things you take for granted, and look at the world the way a plumber does. Thanks to TLS, the plumbers can't see the HTTP headers to figure out what's actually flowing, so they sort of end up whacking all of it.

Generally, low-reputation IP addresses are associated with scams, spammers, and other similar things. Gaming somehow gets lumped into this bucket in some jurisdictions, but that hurts you worldwide (similar with other "sin businesses" like porn). These blacklists get published (I think there's some parts of BGP that make this happen, but I'm not quite sure what the mechanism is), and being on any one of them hurts your traffic everywhere because it becomes suspect.

I agree with you that this mix of compliance, engineering, and sales is gross. If this was the issue, they should have just told the OP.

Re:Cloudflare took down our site after trying to force us to... by nolverostae on Monday May 27, 2024 @11:54AM (#40491937)
It will be interesting to see. Just for completeness, Fastly is not requiring us to BYOIP or anything unless it causes them actual problems, which so far it hasn't. I'm sure they also have other similar businesses to ours so they should have some experience.

I guess I'll see in a while if this was also just a sales tactic from Cloudflare or not.

Re:Cloudflare took down our site after trying to force us to... by pclmulqdq on Monday May 27, 2024 @10:53PM (#40496910)
Yeah, I also assume that any sane CDN of this size has enough IPs that they can reserve a /16 or so for their "risky" customers (each deployment needs a /24, usually, so /16 gets you 255 regional sites). If Fastly has no problems or can otherwise quarantine you, there's no good reason for the BYOIP demand.
Re:Cloudflare took down our site after trying to force us to... by thrdbndndn on Sunday May 26, 2024 @02:43PM (#40484403)
While they absolutely shouldn't ban IP for reasons you said, some do that anyway.

The most (in)famous case is China's GFW which banes IPs all the time. Yes, other websites often get accidentally blocked, but they don't care. Moreover, you can't even communicate with them because there are no official legal regulations. This is something what any CDN or cloud providers have to deal all the time.

Re:Cloudflare took down our site after trying to force us to... by orhmeh09 on Sunday May 26, 2024 @04:24PM (#40485100)
There are numerous official legal regulations and decisionmaking bodies pertaining to the GFW. What do you mean?
Re:Cloudflare took down our site after trying to force us to... by bitnasty on Monday May 27, 2024 @08:48AM (#40490260)
CCP does what they want, the “regulations” are meaningless.
Re:Cloudflare took down our site after trying to force us to... by zbentley on Sunday May 26, 2024 @11:07AM (#40482734)
I also wonder if the company in the article didn’t know that (either by reading between the lines as you did or via other correspondence they didn’t mention) and weighed that in their decision to go with Fastly.

BYOIP isn’t just expensive—if your content is bad for IP reputation the time-to-flagging of your IPs is going to be way shorter on BYOIP than on shared IPs due to there being less dilution. And that’s without getting into the challenges around rotating/renting IPs on a continual basis.

I do agree that CF did not communicate that well or professionally—if the sales emails are the only comms that happened here.

Re:Cloudflare took down our site after trying to force us to... by pclmulqdq on Sunday May 26, 2024 @11:19AM (#40482812)
I think it is possible that the company posting this didn't realize that this might be the issue, but you are right that they may know. It may have been a small company, even doing that much bandwidth. Online gambling sites tend to push an entire video game when you are playing on their site.

Many gambling companies are fine just doing BYOIP or running dedicated hosting infrastructure that is on providers who are explicitly running hosting for that industry (although they are moving to cloud). There is a good reason this separate infrastructure exists. In general, I would not assume they are rotating IPs: this is not a scam, it's a business, and they are largely fine with being blocked in places where they can't legally operate.

Re:Cloudflare took down our site after trying to force us to... by jimbobthrowawy on Sunday May 26, 2024 @03:33PM (#40484712)
>I have spent far too long dealing with this as an RNG supplier

Is there much of a market for that? I thought random.org had it all sewn up.

Re:Cloudflare took down our site after trying to force us to... by dist-epoch on Sunday May 26, 2024 @03:51PM (#40484835)
Gaming machines are highly regulated, almost like medical devices.

There are seals on the hardware, any modification must be approved, you must certify that the payout is the expected one, ...

Re:Cloudflare took down our site after trying to force us to... by mschuster91 on Sunday May 26, 2024 @04:17PM (#40485042)
I'd say worse than medical. As long as it works and doesn't fail, no one will give a fuck about medical hardware or the condition it's in. Just look at your average GP's ultrasound, it's probably older than the GP themselves is.

Gambling however, you constantly have government auditors and the tax office crawling up your literal arse to make sure you don't cheat the gamblers, or worse, the government out of their money. And in some cases, add the mob or other criminals on top who also want their cut.

Re:Cloudflare took down our site after trying to force us to... by jamesfinlayson on Sunday May 26, 2024 @09:08PM (#40486728)
Yep, governments can change, and the new one can decide to crank up the regulation too.
Re:Cloudflare took down our site after trying to force us to... by jasonm23 on Monday May 27, 2024 @07:13AM (#40489679)
Pretty sure it's your figurative arse but potato, pot-ahh-toh
Re:Cloudflare took down our site after trying to force us to... by PawgerZ on Wednesday May 29, 2024 @01:03PM (#40514233)
Literal. Gotta make sure you're not hiding any numbers in there.
Re:Cloudflare took down our site after trying to force us to... by bitnasty on Monday May 27, 2024 @09:00AM (#40490336)
Gambling is shady to begin with. It’s easy to say that all businesses only care about money, but gambling is pure greed, from both the customer’s and proprietor’s perspectives. Sure, there is greed in medical but gambling is ONLY greed.
Re:Cloudflare took down our site after trying to force us to... by pclmulqdq on Sunday May 26, 2024 @05:51PM (#40485717)
It's worse than medical devices, at least for the final machines and software.

For components in the path of money flow (payment processors, RNGs, hosting, etc.), it's similar.

Re:Cloudflare took down our site after trying to force us to... by pclmulqdq on Sunday May 26, 2024 @05:52PM (#40485724)
No, random.org isn't in that market at all, aside from doing some drawings local to them and unofficial games.

I think we are only one of ~3 TRNG suppliers who have been audited. Many games don't use a TRNG, though.

Since it uses atmospheric noise, you can also influence the numbers from random.org by transmitting radio waves in the area nearby - the operator of random.org has mentioned that there's so much RF activity that he is concerned about whether the bits are still random. A final issue is that they are also so low-volume that they probably can't get enough test data for the required audits (which can be a lot of data).

To underscore the volume question: Random.org used to have a running count of bits generated. The counter wasn't monotonic (before it broke in ~2015-2019), but the peak value I saw when I checked archive.org was about 250 GiB total since 1998 (that was in 2015). That is one quarter of the size of our "light" qualification test ("heavy" is 16 TiB). The RNG auditors also take O(100) megabytes for each audit, which would be a significant fraction of random.org's output.

Re:Cloudflare took down our site after trying to force us to... by Southclaws on Wednesday May 29, 2024 @08:58AM (#40511496)
honestly this is fascinating to me, I was curious too and upon searching "RNG Supplier" I couldn't find anything, 3 supplies in the whole world is a crazy supply-side industry!

I was just curious to see what a landing page of a RNG supplier looked like, how do you even do sales for such a thing? With 3 players I guess it's just something you know in the industry and those partnerships are likely long-lived, right?

niches like this fascinate me for some reason!

Re:Cloudflare took down our site after trying to force us to... by jamesfinlayson on Sunday May 26, 2024 @09:14PM (#40486761)
> Is there much of a market for that?

I don't think it's a huge market, but state-run lotteries around the world need good random number generators for games without physical balls (like Keno for example).

I've talked with people that have created RNGs (rather than buying off-the-shelf solutions) and it sounds like soul-crushing work - mostly due to dealing with the government regulators that need to give final approval before the RNG can actually be put into production.

Re:Cloudflare took down our site after trying to force us to... by Havoc on Sunday May 26, 2024 @06:24PM (#40485958)
>The solution was to aggressively sell the Enterprise plan

A demand for 120k upfront or else bad stuff happens is by no reasonable definition "selling", aggressive or otherwise

Re:Cloudflare took down our site after trying to force us to... by DoesntMatter22 on Tuesday May 28, 2024 @05:33PM (#40505857)
Sounds like something Sammy The Bull would be sent to do, not the sales team.
Re:Cloudflare took down our site after trying to force us to... by Aeolun on Sunday May 26, 2024 @09:22PM (#40486808)
It almost sounds like you are excusing them. Asking people to switch to the enterprise plan and bring your own IP is reasonable, but not on a timeline of 24h, and trashing their account when they tell you they are talking to a competitor makes me feel like I should flee Cloudflare services with all haste.
Re:Cloudflare took down our site after trying to force us to... by robocat on Wednesday May 29, 2024 @05:31AM (#40510098)
> and trashing their account when they tell you they are talking to a competitor

That is just a story they have made up. They don't know why Cloudflare shut their account down. I reckon the Fastly "reason" is likey a red herring.

Re:Cloudflare took down our site after trying to force us to... by ExoticPearTree on Thursday May 30, 2024 @03:36PM (#40527762)
It does not really matter, the 24h deadline is incredibly unprofessional. And deleting their account even more so.

I think the lesson here is to be as provider agnostic as possible and have a backup plan in case your current provider decides they don’t like you anymore or they just delete all your data just because they can.

Re:Cloudflare took down our site after trying to force us to... by pianom4n on Monday May 27, 2024 @02:50AM (#40488317)
Maybe the first couple of weeks there was a misunderstanding, but by May 7 it was clear what the situation was. They was told they need BYOIP through and enterprise plan; they just didn't want to pay for it.
Re:Cloudflare took down our site after trying to force us to... by bitwize on Monday May 27, 2024 @11:30AM (#40491675)
This is just how people communicate now in the business world. It's up to you to read between the lines.

One thing I've learned to be wary of on the job is "do you need help?" That phrase is often code for "You are not performing up to our expectations. This is your first and only warning. Get in shape or get out."

Re:Cloudflare took down our site after trying to force us to... by phendrenad2 on Thursday May 30, 2024 @01:18AM (#40520322)
Poor communication is worse than no communication.
Re:Cloudflare took down our site after trying to force us to... by bedatadriven on Monday May 27, 2024 @03:32PM (#40493835)
Ok, I have to ask: are you a Random Number Generator (RNG) supplier, or a Renewable Natural Gas (RNG) supplier, or some other kind of supplier??
Re:Cloudflare took down our site after trying to force us to... by OkGoDoIt on Monday May 27, 2024 @04:12PM (#40494207)
Considering they mentioned working with gambling/casinos, I would assume random number generator. Which may seem somewhat trivial to build a business around, except if you’re in a highly regulated industry like gambling that regulates the implementation of randomness (and probably requires auditing and other complicated things like that). I would love to read a blog post on all the complexities at here.