What the ever loving heck... seriously?! Why wouldn't this be a control plane API that reconfigures a data plane?!
I think they did fix it eventually, because I know that multiple people on my team complained to Microsoft about it. Very short-sighted decision on their end, but to be fair it was a brand new product.
What's so surprising about that? If your CRUD operations take longer to do, and you're doing those to drive a GUI, of course the GUI will lumber along.
The reverse also applies; by separating them you can have issues with your control plane but not have the database go down.
A couple of years ago I stumbled upon a Azure project which started off using the old timey Cosmos DB. Looking at the repository history from those days, I saw a bunch of Entity Framework configurations and navigations and arcane wizardry that would take an engineer months to really figure out.
Then there was an update to CosmosSDK, and all that EF noise was replaced by single CRUD operations that took the unserialized object, id and partition key as input. That's it.
Worlds of difference, and a simple key-value query takes ~10ms to do.
Yes, it's worlds of difference.
Unless that query goes over the Internet to another continent, that's a really long time isn't it?
It is an absolute eternity though. A KV lookup is fractions of a microsecond in a managed language like c#. A http request is in the microseconds range on localhost and a smidge more on a performant local network. A poorly behaved local network (busy wifi on an ISP router) is 1-2ms, and I can do a round trip to my nearest AWS region in 10-15ms from my home network.
It’s an absolute eternity, and when thinking about this stuff and scaling, think “is it worth slowing down the normal case by 1000x to introduce an external service”
ConcurrentDictionary<K, V> read latency is going to be around 7-15ns for the data hot in the cache, scaling with key length if it is string-based, and anywhere between 75ns and 150ns for reading that out of RAM. Alternate implementations like NonBlocking.ConcurrentDictionary can reach down to 3.5-5ns for integer-based keys given all data is in L1 and the branch predictor is fully primed, on modern x86_64 cores.
Redis is cool but people use it too willy nilly. It's expensive to run, and you have to pay a cost of (de)serialization, and network latency. Sometimes that's necessary, especially if you need to share stuff across multiple nodes and you're afraid of hammering the database too much, but a lot of the time, I'd say even most of the time, you can get away with just a big ol' global ConcurrentDictionary (or whatever the equivalent is in your language of choice).
Basically any thread-safe dictionary is fine, because no matter what kind of locking strategy they're using, it will almost certainly be faster than anything that has to hit the network, by several orders of magnitude, and you have one less thing to manage in your application. You can figure out the best way to invalidate old entries, or find a library to do it (e.g. something like a Guava cache), and you'll likely much better performance than arbitrarily farming to Redis.
That's fine if all you're doing is caching limited data independently on each node and you have no requirement for consistency or even durability.
Most of the usecases you stumble upon do not fit that scenario. That's why "a lot of the time" suggesting a dictionary is plain wrong.
You've assumed you have multiple nodes here. Our point is that you can scale _significantly_ more with a ConcurrentDictionary (or equivalent) than you think - far more than a single node backed by redis will go. You have a point about durability, but in my experience, most apps that lean on Redis don't actually solve that problem. If the app goes down mid request, it will leave broken state behind. The data itself in redis might be valid, but the app can send partial data. The solution to this is the same as handling durability in a non-redis case too.
Also, runnign Redis on the same machine for persistance is an option for durability.
> Most of the usecases you stumble upon do not fit that scenario. That's why "a lot of the time" suggesting a dictionary is plain wrong.
I disagree. Most of the usecases _do_ fit this scenario, and Redis is (over) used as a tool to allow for horizontal scaling caused by an unwillingness to scale vertically. At the level of Meta/X yes you absolutely need it. But for a web app with about 10k concurrent users, you can handle it on a single node.
I think you need to revise your assumptions. Even optimized DynamoDB setups shows average response times in the 15ms range and P99s at the 20ms range.
One of the mistakes you seem to be making is assuming that to do a query on a globally distributed database with multiple partitions, you'd only need to hit one single box and no data exchange would take place. That bears no resemblance with reality.
If you're hosting a service in a cloud provider and you implement your services so that your call to the cloud provider's database goes over the internet to another continent, you have serious problems but none of them are caused by the cloud provider.
Also, CosmosDB is globally distributed.
I don't think anyone making that sort of claim knows anything about cloud services. A single roundtrip of a no-op request within the same data center takes 0.5ms. Add querying across multiple partitions and data seeking, and you don't find cloud providers doing better.
To frame how oblivious that claim is, DynamoDB is lauded for it's response times being sub-20ms.
https://stackoverflow.com/questions/34552625/how-to-get-sub-...
It was interesting seeing the biweekly status updates, they basically all started with “This is how Jet.com broke Azure core services this week”.
As much as it sucks, this was a deliberate strategy all the way from Satya - every employee knew Azure was a joke, but the only want to actually fix shit was to get internet scale customers to break it daily and weekly.
I don't get it. There's lots of distributed systems theory that could provide a more robust, analytical approach to a scalable architecture. If a system is regularly breaking like this, it sounds like it should be a "back to the drawing board" moment.
The only thing that seems to get the fiefdoms to work on an issue is if a big enough customer calls enough of an attention to the issue so that no leader/fiefdom wants to be blamed in such a "high profile" issue that everyone got their work in gear for. This leads to organizational dysfunction since now everyone understands that there's no point in trying to fix anything until there is enough attention on it, and that is mainly achieved when a big customer complains.
I've even had projects that took months of engineering work that would resolve many user complaints about a lack of really basic functionality, and different elements of leadership would block the launch. I can only guess that some fiefdom did not like change and would not state their objection publicly, so I only got stonewalled, and I never got an explanation to throw away months of work. I am as certain as I can be that if a big customer complained loudly enough about the problem that such basic functionality was missing, the organization would demand that the fix be launched.
For some values of "better", I guess. Performance is still terrible, their data visualization/inspection tools are shameful, their SQL dialect is finicky and has no error reporting beyond "something is wrong with the input value", and their official Python SDK has a race condition bug that can silently clear out your documents when under heavy load.
I used to work at a Cosmos-heavy house and I would utter "fucking Cosmos" around 15 times a day.
For the most part it was just fine, until we started using CosmosDB (then called DocumentDB).
DocumentDB, in its first incarnation, was utterly terrible. The pricing was extremely hard to predict, so we would end up with ridiculous bills at the end of the week, the provided .NET SDK for it was buggy and horrible, but the very worst part was the WebUI appeared to be directly tied to your particular instance of CosmosDB.
Why is this bad? Because if you under-provisioned stuff for your database, it might start going slow, and it would actually lag the web interface you would use to increase the resources! We got into situations where we had to turn off the entire application, just to bump up the resources for Cosmos. It felt like it was a complete amateur hour from Microsoft.
My understanding is that Cosmos has gotten a lot better, but man that left a sour taste in my mouth. If I end up getting some free credits or something, maybe I'll give Azure another go but I would definitely not recommend it right now.