News for hackers. Stuff that matters.
Re:Evolving a NoSQL Database Schema by librasteve on Monday November 11, 2024 @03:11AM (#42105328)
Very informative read … however, I am left not really understanding what the Entity Manager is … perhaps a migration tool to move data from NoSQL to RDBMS, perhaps a front end helper service? Is it FOSS, a PAYG service? How can I deploy it? Perhaps this article long enough already and is intended to whet our appetite.
Re:Evolving a NoSQL Database Schema by karmaniverous on Monday November 11, 2024 @03:42AM (#42105472)
I'm the author, and thanks for asking! The article is really just background, submitted by a friend. Still building the demo & documentation, so my apologies for the confusion.

Entity Manager is a framework for defining, managing, and most importantly QUERYING an entity model with DynamoDB. It's actually platform-generic, so the DynamoDB-specific machinery is implemented at https://github.com/karmaniverous/entity-client-dynamodb

Entity Manager's most important feature is that it permits a simple, scheduled partition sharding configuration and then transparent, multi-index querying of data across shards with a very compact, fluent query API.

This resolves the biggest challenge of using DynamoDB at scale, which is that very large data sets MUST be sharded, and a given query can ONLY operate against a single shard. If you're querying on the basis of a related record, you won't know which shard your results will be on so you must query ALL shards.

Entity Manager reduces this to an effortless operation: once you've defined your sharding strategy for a given entity, you can forget sharding is even a thing.

For some more color on Entity Manager within the context of SQL vs NoSQL databases, please review this (much shorter!) article: https://karmanivero.us/projects/entity-manager/sql-vs-nosql

Re:Evolving a NoSQL Database Schema by librasteve on Monday November 11, 2024 @05:47AM (#42106127)
thanks … that’s really helpful
Re:Evolving a NoSQL Database Schema by vdvsvwvwvwvwv on Monday November 11, 2024 @07:37AM (#42106697)
That makes me want to learn more. I will read this.

Like most been burned and frustrated by this sort of sharded DB and unperformant queries if you dont or cant use the partition key. Usually you need a second table amirite :) And seen others burned.

Especially when your intro is a Jira ticket learning on the job!

Re:Evolving a NoSQL Database Schema by karmaniverous on Monday November 11, 2024 @07:49AM (#42106744)
Very cool of you to say so!

I've actually been using the JS version of EM in production for over a year. It's been working flawlessly.

The TS version is a complete rewrite that factors in a BUNCH of lessons learned and is completely--maybe obsessively lol--type-safe. The query builder got a LOT of attention, and the fluent API reduces even complex queries to a super-compact, declarative coding experience.

I'm pushing a big update tonight and will then resume my focus on the demo & docs, basically the companion stuff to this one. Should be ready for use in a couple of weeks.

Thanks for the interest, it really means a lot to me!

Re:Evolving a NoSQL Database Schema by mustime on Wednesday November 13, 2024 @05:16AM (#42124725)
If you're looking for tools to simplify query creation, especially as you're exploring different DB schemas, https://ai2sql.io/ could be useful. It uses AI to quickly generate queries based on plain language, which might help when you're experimenting with different setups.