Even worse are undocumented quotas. Probably not quite the same thing, but I've had such an issue with a SaaS API for extracting data for a cloud system. It has an unpublished limit on queries. Writing a library of code to use it, I was fine. All tests passed. Running a full day's data pull & integration had sporadic failure: It would run w/o issue for half the job, then have seeming random errors on some requests.

I kill the job, start again, and the problem is there from the very beginning. Kill again, review my code for an hour or two (python, grequest) tweak some parameters, start the job again and it seems fine, problem solved? Nope, halfway through the same issue occurs.

More testing showed even that "halfway through" wasn't consistent.

It's the hidden quota, and it wasn't even consistent: Sometimes it would go 10%, sometimes 70%. I don't know if they even had static limits or some type of dynamic system. Then you get throttled, only requests fail without any indicative error message except the "failed" or "unavailable" (I forget which)

I determined the limit was tied to the API key. Given that error messages & the unknown limit made working within their hidden limits difficult, and support was neither sympathetic or forthcoming with details necessary to work within their limits, I simply created multiple API keys & rotated between them on requests. Probably not the best behavior for a tenant of a shared system, but I didn't see much choice when the alternative was to not use an essential advertised feature.

There are some "fun" horror stories of people hitting the Azure Resource Manager API quota limits. At least one of them was linked to some sort of shared service principal for the client org, so if anyone hit it, the whole organisation could get locked out.

It would first start to throw HTTP 429 codes occasionally, then you would get locked out for exponentially increasing times, up to two weeks or something absurd like that.

Even if you called support, there was not a lot they could do, because the rate limiter is a low-level thing built into an internal load balancer somewhere, and it is difficult to override it for a single account.

The worst part was that you could hit the limit while not actually taking any action! If you just had certain Azure Portal screens open, the JavaScript would refresh things in the background, constantly consuming the API rate limit quota. Some screens do so many calls that you'll hit the rate limit in a matter of minutes if you leave your browser on those. If you fail to close all browser sessions of all admins right away, you can lock yourself out for days.

Yikes. That sounds nightmarish.

My team struggled with this for two years. We ended up helping MSFT add a caching layer to the Kubernetes Azure cloud controller and creating a separate Azure subscription for each Kubernetes cluster.

Ooh boy. Undocumented limitations caused major problems with us with AWS SQS. It is now in the documentation (maybe because I complained to our AWS rep), but SQS has pool of 20k messages that it will pull from when serving requests. If you are using a FIFO queue and you have 20k messages with the same message group ID in that buffer, then you are unable to process any other message regardless of how large the queue is. It caused multiple severe degradations until we rearchitected how messages are grouped. If only it had been in the documentation from day 1.

Gotta love it!

Pretty scary that it just breaks without any error messages, and without an easy way to clean up the damage.

Queue overflows are not uncommon.

"For FIFO queues, there can be a maximum of 20,000 inflight messages (received from a queue by a consumer, but not yet deleted from the queue). If you reach this quota, Amazon SQS returns no error messages. If your queue has a large backlog of 20,000 or more messages with the same message group ID, FIFO queues might be unable to return the messages that have a different message group ID but were sent to the queue at a later time until you successfully consume the messages from the backlog."

Link to the documentation: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...

I only figured out what was happening by trial and error while trying to fix the production issue where I was moving 1000 messages at a time into a 2nd queue. Once I had dequeued 20k, everything starting pumping like normal. That's when I fired off an email to our AWS rep asking for confirmation from the SQS dev team. Within a couple weeks, the docs you just quoted were added.

Wow, that's a scary story.

Yea, we had a queue backlog of 3 million (that were bound by an SLA) by the time I figured out what was going on.

I had a similar experience with a SaaS. They let us run fine for months, then started degrading our service. When we contacted them, they tried to convince us we had a bug in our implementation. No, we confirmed we were using their service exactly as we intended to. They seemed to have designed their service with a particular use case in mind, and ours wasn't it. We tried multiple times to explain our use case and why their service was useful for us, because we didn't want them to think we were using the service in bad faith, but if they understood, they never let on. After much grumbling, they would lift the restrictions on us, but apparently not far, because we'd run into them again weeks or months later as our traffic increased. They were obviously unhappy with the way we were using their service and determined to cause issues for us, so we gave up and stopped using them.

Google Maps^W Places API. Official limit is 100qps. Except that if your query has more than 20 results, the "next page" link doesn't work for between 1-2 seconds.

Yeah, same story here with Google Drive API. My app is getting rate limited, even though I am nowhere near the maximum requests per 100s. Just requesting things in bursts makes it trip. When you chat with support (not easy given that it's Google...), they usually refuse to make your quota higher because they don't see you hitting the current limit even remotely...

Around 2 years ago in S3 there was an interesting quote issue - official documentation stated quotas per S3 prefix (kind of like folder path), but in practice it turned out only first 20+ characters of a prefix were taken into account for quotas, so our burst computations loading thousands of objects were mostly timing out. We had to add random prefixes at the beginning just to get around that. AWS updated their quota management to per-full-prefix, finally though.

S3 has an interesting limit of about 60 LIST requests per bucket, depending on the number of prefixes/size of bucket.

They used to advertise 100 per second, but then they pushed a big update that doubled the read QPS limit, and conveniently left out the reduction in LIST qps from the docs.

My unexpected AWS quota story: AWS Batch. Using managed instances, just asking AWS to provision “suitable and available”. Daily jobs (docker containers) ran fine for the first 6 months. Eventually started stalling and failing with DockerTimeout. They would fail in batches of 100s and then at some point start working again.

It took several hours and some back and forth with support to realize that the burst IOPS quota of the provisioned underlying EBS disks on the EC2 instances forming the ECS cluster had been depleted so disk performance completely tanked to the point the docker agent couldn’t be reached for 4 minutes.

And here I thought the whole point was that these things would be abstracted away...

All these weird quotas and complicated pricing were the reason why we switched to digitalocean once they had managed kubernetes. It's such a pleasure to not have to worry about it and also much cheaper.

Even more so when you factor in dev time.

I've been bit by the same issue on ECS.

Some stream processing applications restarted and they perform a lot of reads at startup to recover their in-memory state. All other containers on that instance eventually also restarted and got migrated to another EC2 instance which also got IOPS depleted soon enough.

And the cycle continues. The issue was there was no proper monitoring set up and getting a timeout from Docker isn't very helpful error message.

Since then I've made sure to build in checks to prevent bouncing all services on a machine at once and spreading out applications that use disk across machines instead of binpacking.

The icing on the cake here is that those IOPS from the docker agent is outside of your control. Before having to dive deeper into it, I would have assumed that only IOPS stemming from the workloads themselves would count against quotas.

Using these abstractions of abstractions of abstractions that all end up leaking fatal failure modes you have to deal with yourself makes me start questioning the fundamental value proposal. The one major thing you get away from is setup costs, but the total time investment gets amortized.

We got bitten by this. Worst thing is that even with the help of an account specialist, we couldn't get a precise number for how much IOPS we needed, but had to look at the burst balance graph and calc^Wguesstimate the slope.

And this is why it’s not easy to use any of the cloud platforms parttime. They’re all full of these kinds of gotchas and your example is even a minor one since it really is quite clear from the documentation. There’s even a choice of Provisioned Throughput if you’re clicking this together in the Console, which should probably make you suspicious about what happens if you don’t use that.

But a certified AWS Architect, especially at the higher levels will easily spot the big gotchas from just your architecture diagram. Consider getting certified yourself, if you want to be more confident around these in the future. But there really is no replacement for just putting in a lot of hours.

What value is the certification for the cloud? It's such an incredibly rapidly changing thing, anything you learn goes out-of-date in a matter of months. The two things I complained about in this thread are no longer issues, they got fixed.

The project I'm working on right now is using a PaaS stack is a bit of a "moving target". I feel like those people running to catch a train that's still rolling along...

Does the certification cover all products? They seem to be releasing products more and more frequently.. it has to be tough to keep up to speed on everything to a degree where you don't need Google to explore options and plan projects.

This is why you should never use anyone else's server's for your software.

The linear IOP density model seems clever and logical but is a huge source of headaches because it includes a patently false assumption that IOPs scale in proportion to growth in object size. Performance quotas should be assigned at the object level (block device, file system, bucket) regardless of size.

It’s not because the underlying storage is 1 TiB disk, so you’re getting a fixed allocation that’s shared among many other 50mb/s clients (ie assuming flash and overall transfer rates of ~2gigabytes/, that lets them coex roughly 40 customers on 1 machine without over subscribing)?

Isn’t that kind of pricing mode about the only one that would be feasible to implement to make this cost effective? How are you thinking it should work? Fixed cost per block transferred?

The pedantic answer would be to allow customers to specify both minimum storage space and minimum IOPS. Then charge them for whichever is the larger portion of a drive and provide the space and IOPS that they are then paying for.

This way, if you need 10MB but also 25,000 IOPS, then you'll pay for ~10% of a drive. You'll get your minimum required 25,000 IOPS....and also 200GB or whatever share of the drive is required to get you those IOPS.

At the end of the day, I'm not entirely sure it matters whether the cloud provider breaks it out like this as long as it at least has a little gray UI element under the specified storage space slider that reads out the IOPS to you.

It would be exactly the same as the current situation where customers do this manually. So probably not particularly necessary - why make customers fill in additional fields they might not need to?

I do think cloud providers should make it clear during requisition and read back how many IOPS you're getting, just for clarity.

It does seem like having temporarily high quotas and then throttling back seems to break developer experience. Good deeds and punishment - but in this case there's a solid underlying reason why it has detrimental effects.

For Amazon EBS you have some kind of IOPS slider. This is possible even on normal EBS (gp3) volumes, not just on those special high performance EBS volumes (io2).

> General Purpose SSD (gp3) - IOPS 3,000 IOPS free and $0.006/provisioned IOPS-month over 3,000

> General Purpose SSD (gp3) - Throughput 125 MB/s free and $0.0476/provisioned MB/s-month over 125

https://aws.amazon.com/ebs/pricing/

It would make more sense to me to charge for the size of the provisioned share and performance profile, like EBS, instead of this weird performance tiering. It basically makes EFS useless for the bulk of use-cases I can personally imagine.

They do that as well - see "provisioned throughput"

See the section "Specifying Throughput with Provisioned Mode" here: https://docs.aws.amazon.com/efs/latest/ug/performance.html

Fixed cost per IOPS allocated. Essentially, the same thing as before, but without the necessity of you storing large blank objects.

How would scheduling of concurrent I/O workloads work? If I’m paying some price per 1k IOPs and my service gets a spike, won’t I greedily take out the other services running on the same machine rather than getting throttled? Doesn’t this also penalize workloads that do lots of small I/Os rather than a few big ones even if the amount transferred is the same?

1) By ensuring capacity for max allocated concurrent IOPS

2) Wrong relationship; you're paying for IOPS to the block store, so you'd be trampling on other accesses to the same block store.

3) This penalizes them less - in that workloads that do lots of small IO on small files will actually be able to request the IOPS they need, instead of IOPS being (wrongly) dynamically allocated.

Can't this be addressed by provisioned throughput on EFS?

I remember having something similar years ago. Ended up just creating a 100GB file, or something, just to get the IOPS I needed until we could migrate off EFS.

I ran into this same issue with GCP while using the boot disk for some caches. In this case the grace period is a few minutes before they throttle. It was quite a pain to track down.

The "pain to track down" is important because nothing failed.

I didn't get an alert.

There was nothing in the logs.

There wasn't anything in the portal to indicate that something had changed.

Everything was up and responding, just really, really slowly.

The application wasn't even timing out, because the EFS share itself was responding to TCP ACKs instantly, and even the timeouts at the NFS protocol layer weren't being hit. I was just getting one... file... at... a... time.

After about 2-3 hours of troubleshooting I opened a service ticket with AWS, and it took them another few hours to figure out what was going on.

Like other people in this thread suggested, I first copied a 10 GB empty file into the volume to speed it up. Later they made the fixed IOPS SKU available and I switched to that.

To be fair, there is a cloud watch metric you could have set an alert on: "PercentIOLimit"

Ahhh... the joy of enterprise monitoring systems that do exactly nothing by default, and are very helpful in avoiding any further recurrences of one-time issues. At a nominal fee, of course.

The golden rule of both backups and monitoring is: There are no time machines.

It's not helpful to find out after the fact that a default-off alert or metric threshold alarm could have avoided the issue. It's not helpful to blame the user for not knowing every one of thousands of metrics they "should" be monitoring. How would they know until they get burnt at least once?

Even if they do get burnt, how would they know which of the metrics they weren't capturing could have been useful it was was captured?

That's not a rhetorical question!

Fundamentally the issue is this: Practically no enterprise monitoring system stores data efficiently enough to capture all metrics, so instead they simply... don't.

Instead, these "solutions" trade a moderately difficult storage compression problem at the service provider end for a physically impossible time travel problem on the consumer end.

Just blame the user for not knowing ahead what disasters they will face! Job done! No need to figure out columnar compression, that would take actual engineering work for a couple of guys. But why bother when it's soooo much easier to just dump some JSON into a storage account or S3 bucket and bill the customer for every metric. Mmm... dollars per metric per month. That's the ticket to a nice robust revenue stream!

Apologies if I sound salty, but I've traced the root cause of outage after outage back to lazy vendors writing MVP monitoring systems that do literally nothing useful out of the box. These vendors simply refuse to store data efficiently enough to capture all relevant metrics to that I can check what happened without needing a TARDIS. Why would they when monitoring is a revenue stream that they measure in gigabytes?

PS: An ordinary Windows desktop has on the order of 10,000 to 50,000 performance counter metrics that it tracks. However, with even light compression, that's barely a few gigabytes for a year of logging every metric every second. I've written code to do this personally. Name me a cloud vendor that can approach this within an order of magnitude without an eye-watering bill every month.

I'll be the fly in the ointment here, the additional sand in your shorts, and say "This is why bare metal is better".

Bare metal doesn't have to be VM/containerless. Roll your own. But at least in this case, you're dealing with your own issues, with things not hidden and abstracted away, and at literally 1/100th to 1/10000th the cost of AWS.

And yes, that's with the hardware investment and wage costs rolled in.

I feel like AWS and others created some sort of one ring, and just reeled admins in, pulling them from the wild, so that with a shortage of greybeards, people can't find those with the 'sysadmin temperament', and are therefore stuck with cloud.

And it is indeed a specific temperament, to create stability, which means creating some constraint, yet to forge that constraint in the most effective way, whilst doing one's best to enable devs to work most effectively.

AWS has constraints, but because it's a wall of "this is the way external corp does it", it gets far less flack than the guy that smells like onions, when he says "we can't do this safely" or "this policy must be followed".

Yes, OK, this is a bit of a rant. Sorry.

Hardware has the exact same monitoring issues, worse even.

The dinosaurs of the on-prem hardware world like Dell, HPE and IBM make the most atrocious systems management software that I have ever seen. Bargain basement quality at best.

The cloud is eating their lunch for a reason.

Everyone wants a single pane of glass, not a hundred unique and special vendor-specific consoles to manage one app.

Everyone wants unified logging built-in, not Splunk on top of fifty different log formats, none of which can be easily correlated with each other.

Everyone wants a unified IAM system, nobody likes to deal with expired SAML or LDAP certificates in the middle of a change.

Etc, etc...

Compared to having to deal with 10 different teams just to spin up a single VM and 15+ teams for a moderately complex application with HA/DR and monitoring, being able to simply click through a cloud portal is an absolute joy.

It's the same thing that made VMware so popular. Nobody liked to have to get finance involved and order new kit six weeks ahead just to be able to spin up a tiny web server.

With bare metal, you just have to roll your own monitoring.

Primarily, only buy what you can work with easily. Mainly, esure raid monitoring can be scripted, and you really need not depend upon the manufacturer's horrid software (which I agree is just that). A bit of IPMI for failed power supplies, or what not. If a box dies, it dies, that's what failover is for.

I don't your logging complain, you need it no matter what you do. Just use an rsyslog server, logcheck or other app, monitor for important events, done. I've never used anything off the shelf, and yet all I hear are complaints from those that do.

I get how people might want to eschew the above for simple stuff, but once you start needing heavy monitoring, it's all the same ball of wax.

FWIW these issues in ops are referred to as a “brown out”. Everything “works”, but at a huge reduced performance. Usually without alerts because nothing has completely failed.

Yep this bit me in the ass too. When I first used it I was expecting something like an NFS share from a NetApp or ZFS Filer. I see they offer provisioned iops now but still, why not offer a version without all the rigmarole?

>""The baseline rate is 50 MiB/s per TiB of storage (equivalently, 50 KiB/s per GiB of storage)."

Does the mean that baseline is only realized when at least a Tib of storage is actually being used then? In other words there was a distinction between how much you storage you were actually using vs how much storage you provisioned? Are there other services that use this same model?

I've seen this same iops story play out 2 different times at two different companies both with the same story on production systems.

Both were solved by the same thing,

Provisioning a few terabytes of storage space with nothing on it.