The only solution that doesn't require spinning up a VM, reading all the data and writing it back again is the Azure Data Factory [0]. Which according to Azure's own benchmarks [1] manages to copy only 2MBps between two CosmosDb instances each with 100,000 RUs ("Resource Units"). That's more than half a day for a small 100GB collection.
And in case you're wondering whether 100K RUs is a lot, consider the cost: 5840$/Month for single-region writes. 11680$/Month for multi-region writes [2].
Oh, and as a bonus - can you guess the official procedure for restoring a collection from a backup? [3]:
> If you have accidentally deleted or corrupted your data, you should contact Azure support within 8 hours so that the Azure Cosmos DB team can help you restore the data from the backups.
And yeah, it's 8 hours because they backup the database every 4 hours and keep the last two backup copies [3]
[0]: https://docs.microsoft.com/en-us/azure/data-factory/introduc...
[1]: https://docs.microsoft.com/en-us/azure/data-factory/copy-act...
[2]: https://azure.microsoft.com/en-us/pricing/details/cosmos-db/
[3]: https://docs.microsoft.com/en-us/azure/cosmos-db/online-back...
0. https://docs.microsoft.com/bg-bg/azure/cosmos-db/optimize-co...
We are in the process of updating the Azure pricing calculator to provide various options (e.g., # of regions, single vs. multi-master, etc.). The updates to the calculator are coming shortly.
In the meantime, please feel free to ping us at [email protected], if you need any help with cost estimates or have any questions.
You can also use the changefeed system which is stored as an event-hub stream and then save it to another collection using Azure Functions.
Hopefully the unresponsiveness issue was addressed quickly after it was reported by you.
Proof: https://imgur.com/a/asem3d9
Maybe at very large scale AWS shines through, but I would choose Azure over AWS any day.
I'd personally trade my first born with no second thoughts to use AWS instead.
Microsoft is just better at selling the whole package to enterprise where your primary function isn’t tech.
They were first to incorporate EU legislation, and are frankly still the only of the three big cloud suppliers that can show us exactly where our data has been or passed through at any time.
Their support is also excellent. We pay Microsoft a lot of money, much more than the cost of Azure because of licensing, but with it comes support. If something breaks down, I can call Seattle directly after which we will receive hourly updates by phone until the matter is resolved.
With google you get to fill a form or chat with a bot. AWS is better, but again mostly to their big customers, and since we aren’t buying office 365 or licenses from them, we’re just not big enough to qualify for the support we need.
If I need really high load/throughput, I'd probably reach for Cassandra on DO for best bang for the buck. YMMV though, and not too big on Cassandra unless you really need it.
Thanks for the feedback.
Between AWS and GCP even though AWS come out on top, GCP wins in quite a few services.
I'm not disputing the outcome, just feel like it would be useful for others in a similar situation.
1. Many unexpected behaviours which you would not encounter in any other software - for example when you're blocked by its firewall, it would let you connect on all 7 OSI layers, then kick you out with "Command failed with error 13: 'Not Authenticated' on server xyz:10255'". By the way even Azure Portal itself doesn't know how to handle this behaviour correctly and when not included in the firewall whitelist just blinks like crazy.
2. Many seemingly basic features commonly found on other database systems are for some reason missing here - it doesn't support users as such, just issuing resource tokens[1] (which you must implement yourself). Thus, most CDB users just go straight with the master key even on production.
3. It is not fully (nor at least nearly fully) compatible with any of the advertised APIs - SQL doesn't support joins, MongoDB misses quite a few features and adds a requirement for the shardkey to be part of the query. Which is not supported by Spring Data, so you must do low-level queries for everything but select (think updates field by field with null checks).
4. The tooling is horrible - as it is not compatible with neither mongodb nor traditional sql, you would need to include the "CosmosDB Emulator" in your build in order to have at least some predictability. The problem? It only runs on docker4win and a Windows VM would need to be running by all developers on the team and workers of the CI/CD. As of April 2019
5. It is CRAZY expensive and the pricing is not really transparent - in our case we had 20 MB of data, 4 1000RUs collections and multi-master write in Western and Northern Europe. The monthly bill? 300 EUR even though Azure Portal was displaying 1/6 of that cost - apparently for multi-master you multiply by the number of regions + 1 (2+1 = 3) and for geo-redundancy only by the number of regions (2), thus the 6x higher cost. The result of that was that we were paying several times more to manage 20MB of data than for all the remaining data - hundreds of GB.
Maybe if you really, really need the planetary scale, the master-master replication and the <10ms SLAs it would make sense and the hassle and cost would be worth it (though I seriously doubt it), but in any other case it is a massive mistake to go this route.
1. https://docs.microsoft.com/en-us/azure/cosmos-db/secure-acce...
1. This occurs when, after the connection has been established, CRUD commands are sent before the authentication command. Note that some MongoDB drivers have a bug where they are not authenticating the connection in rare cases and sending CRUD commands. Could you please check and upgrade to the latest driver of Mongoose/NodeJS-Native/NodeJS-Core driver, which has a rewrite of their client-side auth stack and fixes a lot of bugs. One of such issues is described in https://jira.mongodb.org/browse/NODE-1798 which was fixed by https://github.com/mongodb-js/mongodb-core/commit/7ffb4bbe05...
2. Resource tokens provide fine grained access control but it does require you to implement a custom token management service. We are working on support for Azure AD integration and support for custom roles and policies. The feature is code complete and should be made available soon.
3. Cosmos DB’s SQL query language is supported on a non-relational data model - which maps cleanly to document/column-family/graph. It does not support joins since (unlike a relational model), a single Cosmos container can contain items belonging to any arbitrary schema.
4. Re. API for MongoDB requiring a shard key, we are going to relax this restriction before 7/30/2019. As you noted, we do support an emulator for Windows. We are working on a Linux emulator and should be released later this year.
5. The portal was displaying incorrect cost estimate in certain cases. We addressed it and the fix is being rolled out.
Thank you for the feedback!
I get your point and they should definitely fix either the cost calculation, or paste a disclaimer for multi-region pricing. But I want to (sincerely) ask how you could assume that multi region would _not_ cost more than single region.
Thank you for the feedback!
We have published a number of documents on pricing transparency and cost optimization of your Cosmos DB cost that we want to make sure everyone is aware of:
[1] Pricing model - https://docs.microsoft.com/en-us/azure/cosmos-db/how-pricing...
[2] Optimize cost for your dev/test workloads - https://docs.microsoft.com/en-us/azure/cosmos-db/optimize-de...
[3] Total Cost of Ownership (TCO) - https://docs.microsoft.com/en-us/azure/cosmos-db/total-cost-...
[4] Understand your bill - https://docs.microsoft.com/en-us/azure/cosmos-db/understand-...
[5] Optimize provisioned throughput cost - https://docs.microsoft.com/en-us/azure/cosmos-db/optimize-co...
[6] Optimize query cost - https://docs.microsoft.com/en-us/azure/cosmos-db/optimize-co...
[7] Optimize storage cost - https://docs.microsoft.com/en-us/azure/cosmos-db/optimize-co...
[8] Optimize reads and writes cost - https://docs.microsoft.com/en-us/azure/cosmos-db/optimize-co...
[9] Optimize multi-regions cost - https://docs.microsoft.com/en-us/azure/cosmos-db/optimize-co...
[10] Optimize with reserved capacity - https://docs.microsoft.com/en-us/azure/cosmos-db/cosmos-db-r...
If you have any questions regarding pricing, please feel free to reach out to us directly at [email protected] (direct line to the product team).
We do really appreciate the feedback.
Thank you.
The partitionkey is required so that CosmosDB knows which partition to query. You can actually enable broadcast queries which will ask every partition if you don't want to include the partition key, but this will be slower and require more RUs.
For development (before going into production) with Azure Cosmos DB, we suggest a number of options (also documented here: https://docs.microsoft.com/en-us/azure/cosmos-db/optimize-de...):
[1] Cosmos DB emulator (Cosmos DB Local version) - https://docs.microsoft.com/en-us/azure/cosmos-db/local-emula.... The Azure Cosmos DB Emulator provides a local environment that emulates the Azure Cosmos DB service for development purposes. Using the Azure Cosmos DB Emulator, you can develop and test your application locally, without creating an Azure subscription or incurring any costs. When you're satisfied with how your application is working in the Azure Cosmos DB Emulator, you can switch to using an Azure Cosmos DB account in the cloud. Support for Cosmos DB emulator on Linux is coming.
[2] We provide Try Cosmos DB for free experience - https://azure.microsoft.com/try/cosmosdb/. There is Try Azure Cosmos DB for free experience which gives you a time-limited full-service experience with Azure Cosmos DB. It is designed for trying out Cosmos DB, doing a tutorial, a demo, doing a quick start, a lab without requiring an Azure account or a credit card.
[3] We provide a free tier (Cosmos DB is a part of free account) - https://azure.microsoft.com/free. In Azure, we support a free tier with 12 month access to all popular services in Azure. As a part of free tier specifically for Cosmos DB you get 5GB in storage and 400RUs.
Thanks so much!
>Random failures
>Largely undocumented
>Painful interop with non-azure proprietary offerings
>Almost zero developer community
>Most of the coverage ... isn't by developers, it's by various MS affiliated blogs such as this one, which are more advertisements than resources.
Welcome to Azure!
I'm not sure I feel much different across Azure Cosmos and Aws Dynamo. It seems super-easy to get locked into a relatively expensive proprietary offering in each case.
There's no way to add indexes to this data model, it's all done by just writing your data in another table using the key you want and the value pointing to the original table's row. Perhaps Azure could offer secondary indexes like AWS DynamoDB does but it's probably too complex of a feature to add at this point, which is why it took DynamoDB several years to get to working GSI.
10TB of Table storage is 655$ 10TB of CosmosDb is 2500$
Please feel free to send us the details of your workload that suffered from poor performance to [email protected] (it is a direct way to reach the engineering team and it does not cost anything), and we can help determine why and provide with the recommendations.
For best practices and performance optimizations, the following links will help:
https://docs.microsoft.com/en-us/azure/cosmos-db/sql-api-que...
https://docs.microsoft.com/en-us/azure/cosmos-db/performance...
https://docs.microsoft.com/en-us/azure/cosmos-db/performance...
https://docs.microsoft.com/en-us/azure/cosmos-db/performance...
Also, the following conceptual docs will help:
https://docs.microsoft.com/en-us/azure/cosmos-db/partitionin...
https://docs.microsoft.com/en-us/azure/cosmos-db/partition-d...
https://docs.microsoft.com/en-us/azure/cosmos-db/request-uni...
We are doing a full refresh of our docs (rewriting a lot of the older content and adding new technical content). We welcome feedback on how we can make the docs better and invite everyone to contribute via public PR/Github. If there are any topics that are missing in our docs or are not easily discoverable, please let us know. We will make sure to address it.
We are committed to building a great community. As a relatively young database, we have ton of work ahead, but we are super committed to making sure we make developers successful.
Imagine my shock when I started at Google a year later and discovered Dremel (now BigQuery) where I could just run a query and get the results back in seconds over tens of terabytes of data, with no quotas or approvals.
I think having a system like Dremel is a huge competitive advantage, because you don't have to guess blindly about things, you can just query them and get results immediately. If Cosmos is still the best MS has internally, they need to roll their own Dremel asap. Lots of great people from SQL Server team work at Google now, but MS still has plenty of people who could do a competent petascale columnar DB with one hand tied behind their back.
Dremel now uses standard SQL and unlike Kusto it supports multiway joins and a bunch of other things expected of a more "general purpose" analytical DB.
Ultimately MS efforts will be stymied by the fact that their underlying storage story is nowhere near as good as Google's Colossus.
I won't disclose the details, but that's another thing that blew my mind: I could do linear reads (which is what columnar DBs do nearly all of the time) faster than I could process the data, from hundreds, or even thousands of workers at the same time.
The IO throughput there is truly immense, which is especially impressive given that all of the storage is remote.
It's like saying BigQuery and BigTable share something because they both have "Big" in their name.
@llcoolv has responded and the response is correct.
For scenarios where you create a lot of collections/tables, we recommend to provision throughput on a database and share the provisioned throughput among all the containers – to save on costs. You can change the throughput at database exactly like how you will change the throughput at container level. It is the same API, it is supported via CLI, SDK, Rest API and Portal as well.
The entry point for database level throughout is 400RUs $24 per month and you can add any number of collections/tables that can share this throughput. You can scale up and down in the increments of 100RUs (or $6/month).
Please see the following links for details:
[1] https://docs.microsoft.com/en-us/azure/cosmos-db/set-through...
[2] https://docs.microsoft.com/en-us/azure/cosmos-db/how-to-prov...
[3] https://docs.microsoft.com/en-us/azure/cosmos-db/how-to-prov...
Please see the sample here for moving data:
[4] https://github.com/Azure/azure-documentdb-dotnet/tree/master...
Thanks!
I am aware that one can provision per database, but as another poster mentioned, you can't change that once a database is created, and also you need to specify something that's called partition keys and that can't be done using MongoDB driver. If I recall correctly, you don't even have examples available for that in CosmosDB Python API.
I mean yeah - it's possible. But it is also a pain in the ass.
Yes, for shared throughput database, you will need to create a new database.
- You can create MongoDB API collections with shard keys defined. It is fully supported by MongoDB driver
- Support MongoDB collections without shard key specified is coming in May
Thank you for the feedback on the docs and the missing code samples. We will add those in the docs.
Thanks!
Feel free to reach out to us at [email protected] any time with any questions.
Shockingly poor perf. Is CosmosDB really that bad or did we have it misconfigured? We're not sure, the docs didn't help us understand.
Random failures when connecting. Random errors (or worse, no errors but unexpected results) when querying.
Largely undocumented.
Painful interop with non-azure proprietary offerings.
Almost zero developer community. Most of the coverage of CosmosDB isn't by developers, it's by various MS affiliated blogs such as this one, which are more advertisements than resources.
Unfortunately CosmosDB seems to fit the Azure MO of investing more into sales/marketing than into engineering/support. I don't know about the general atmosphere, but in the circles where I work, Azure and its various proprietary components are losing developer goodwill at a staggering pace.