> It's just odd to me that I had to make this change in the first place. I wonder how many millions of dollars AWS is raking in just because devs are unaware of things like that.

AWS documentation very clearly outlines the behavior of a query and a scan. The internet is full of guides [1][2] on how to model your data so you utilize queries vs. scans.

If the developers at your company misuse a product, and aren't aware of these features when does personal responsibility come in? How is AWS as a platform suppose to discern that the trade-off with a GSI is what you want, and how is this scenario not generalized to every product? People misuse SQL databases all the time, for example.

1 - https://www.alexdebrie.com/posts/dynamodb-single-table/

2 - https://www.sensedeep.com/blog/posts/2021/dynamodb-singletab...

There are two sides to this arrangment: devs who mostly don't have a choice over the tools they use vs. the trillion-dollar company that is building and documenting them. The notion that the devs have just as much culpability in how things end up as AWS is absurd.

> devs who mostly don't have a choice over the tools they use

I'm not going to assume that DynamoDB was forced upon them. I don't know. What I'm more sure about:

Developers have a choice in if they want to invest the time in understanding the tools they use, and learning about the tool's capabilities. In this specific case, we're not talking about a lot of time. 2-3 hours to read the documentation for a new database you're going to be building your business on top of.

What is your argument exactly?

I've personally seen developers "forced" into using DynamoDB because some AWS professional services consultant suggested it. You could argue the developers should've known better, but they had relatively little real world experience. The costs of Dynamo are relatively small compared to the development costs of using it when it's not a good fit.

I think its hard to draw a fine line on where this responsibility lies.

For example: for years, MongoDB took a lot of shit because by default it spins up unauthenticated. Hypothetically; devs take on the responsibility of adding a password. Realistically, shodan was able to index hundreds, maybe thousands, of databases across the internet with no authentication, some containing highly sensitive data.

Firestore is a great analogue to DynamoDB. Not perfect, but they solve a lot of the same problems in a lot of the same ways. Firestore automatically indexes every field in every document in every collection. Flat out. Everything is on an index. That single decision has reverberated to save tons of hours in compute time across all of their customers.

Is this expensive to do? I have no doubt; and its reflected in the cost of the two services. But believe it or not: Firebase isn't that much more expensive than DynamoDB ($0.25/million reads, versus $0.60/million).

> I think its hard to draw a fine line on where this responsibility lies.

I don't disagree, but I think there's consensus on what we consider footguns and what we consider features. Default on security footguns (like unauthenticated access to S3 buckets) should be changed. AWS has changed this. Default on performance features that cost customers money, and may be unused? We should limit those.

> That single decision has reverberated to save tons of hours in compute time across all of their customers.

It's hard to make a statement like this without data. How many of those fields are wasted disk space on a server somewhere under utilized? How much compute does indexing cost on those under utilized fields that should have been omitted?

You make the trade-offs on if you use Firestore or DynamoDB. They're different products, with different features.

I use Firestore and it won't let your perform multi-field filter and order operations without an explicit index.

When you try to do so, it spits out an error that tells you this operation requires an index and then provides a link to create the specific index.

I know it is a different database paradigm, but I think this is great "UX" design.

Firebase has a bunch of additional great features in this vein. It allows definition of min/max instances for Functions and if you set the min instances in the code, it will force a confirmation of price changes to prevent any unexpected charges. (GitHub Actions will fail, for example, until the price increase is confirmed via CLI.)

So i get OP. Yes, it is the responsibility of the engineer to know these things and make decisions, but the vendor can provide much better UX around these constraints.

The equivalent in a relational database would be writing a query without a where statement, and then implementing the where in application code. AWS has a lot of unimpressive bits, but DynamoDB is not one of them, you just have to know how to use it properly (like any database!), and design your schema for the database (which isn't always easy). In general, using scan means you aren't using Dynamo correctly.

Caveat: I used to work at Amazon (not in AWS), but don't anymore.

> The equivalent in a relational database would be writing a query without a where statement,

Sort of, though I'd say scan operations are typically more expensive in any RDMS.

The probably becomes when you want/need something more than the secondary index, then you gotta get more creative.

Our entire industry is plagued by people rushing forward while having no idea how the tools at their disposal work. It's immensely wasteful.

I can't even claim to be immune from this, though I do try to know what's available and how to use it. I'm damn near 40 but it's not unusual for me to still, today, stumble on some tool older than me that does exactly what some New Shiny crap does—but, not infrequently, better, at least for some common use-cases, plus it'll have all the kinks ironed out, or at least documented. Good luck promoting any of that in the workplace, though. For all we go on and on about the importance of re-use and avoiding NIH and all that, we sure are quick to disregard solutions that we can't npm-install.

No clue what the fix for this is, or if it even makes sense, all things considered, to try to fix it.

Symptom of a rapidly growing industry. With N(developers) doubling every 5 years or so, chasing newness at all times and not knowing about good old things is the norm.

I'd rather have that than an industry where any tech newer than 10 years is radical and scary.

Us old farts can remind the kids that old solutions exist when appropriate. It is much harder to try introducing something new into an org that doesn't want it, but needs it.

Yeah I liken it to someone who picks up a blacksmiths hammer and then goes and drives some nails with it. It's like 'sure I guess it works, but man there are way better options that will work better'. Happens with all our industry tools, but perhaps there is no better example than database choice.

Engineers often see a database simply as a place to store things, it's just a box.. they work on a product for awhile that's using mongo, they are more likely to choose it later when they are developing a solution because they've used it before and know it 'works' and more importantly they know something about it. Databases as a topic of study are complicated, there are tons of tradeoffs happening all over the place and until you hit any given pothole you may not really understand those tradeoffs.

I guess what I'm saying is that in my life I'm lucky enough to have an old friend that is REALLY into databses. You should find this friend or become this friend.

Honestly at the end of the day this is all human nature. The hammer worked for you before why do the research to find out that there is a better hammer for you to use that would get your job done quicker and easier. You've gotta get these boards nailed down RIGHT FUCKING NOW or your bosses are going to be pissed.

> Engineers often see a database simply as a place to store things, it's just a box..

Oh man, I had an edit right after posting where I almost attached an addendum about this specifically, but then decided against it to avoid distracting from my core point. There's a totally bizarre allergy to actually using database features, for fear of "lock-in" or something, yet most programs are more likely to be re-written on top of the same database than to have the database swapped out from under it, unless someone made a really god-awful choice of DB, or load scales tremendously (which falls under "good problem to have, and you can afford the migration"). The result is that we as an industry waste an awful lot of time and money creating often-buggy application-layer solutions to things that could have been solved by leveraging features of, for example, PostgreSQL, with results that'd be cheaper, less-buggy, and better-performing. Drives me absolutely nuts.

Or, like, we have nginx in our stack and it could do something we need done, with a config change or a little Lua... but no, we'll spend 10-100x as long to make a worse-performing custom solution to this problem that a thing we're already using can trivially solve for us, but we'll do it in Javascript or Ruby or Python or whatever, as part of our "app". Ugh.

And don't get me started on system-level tools and daemons. Like, we're already using Linux or FreeBSD or whatever, which come with a stupid-large set of great, proven tools, so how about we actually use it rather than just using it as a fancy DOS and paying for SaaS to do things that our OS or distro can already do, probably more reliably? But no, we don't even bother to configure a recipient for root alert emails more often than not, and we just treat the whole thing as a dumb application runner that needs some complex external support system to keep it working right.

> Engineers often see a database simply as a place to store things, it's just a box.

As you said, it's just humans, really. How many self-help books about organisation are out there? Organising your files, organising your clothes, organising your tools -- it's not a solved problem, there are always tradeoffs, and it's very easy to just choose the same solution you've used before, because you already understand how to implement it.

TBH, I think oftentimes it's the right choice to just use any old tool as a hammer[0]. In many (perhaps not most?) cases, you're probably going to save more time implementing the organisational strategy you already know than you are implementing the technically-correct strategy. Though, you probably still need to do some quick maths on that tradeoff -- what's the cost of choosing the solution you know?

By way of example, when we were rebuilding our incident management stack at Dropbox, we spent a lot of time considering our database structure, because one of our key design goals was to reduce friction. Our existing tool had appalling load times, which was (in part) causing users to avoid declaring incidents. We knew we had to create a very low-friction experience to encourage users to declare incidents, and that this had long-term ramifications. So, myself and the other senior engineer on the team invested several days between us determining the correct design for our database. Conversely, when I was building a tool to do some trouble ticket automation, I just implemented things in a simple manner with questionable performance -- because the implementation only cost me 30 minutes, and the impact of a 5-second scan vs. a 0.5-second query was essentially nil (the absolute worst-case scenario was an automated process took a few extra seconds and cost a couple cents a day). No reason to overoptimise.

0. Adam Savage even titled his book "Every Tool's A Hammer"!

"I wonder how many devs don't know that hash tables are bad for range scans".

Fixed that for you.

I just learned DynamoDB this year, after I was forced into it...not a NoSQL fan.

In any event, while I find the structure and docs pretty confounding overall, they do make that behavior pretty clear. The libraries even separate out scans from queries, with the latter taking only a key.

I'd put the blame on the implementer in this case, not AWS.

I actually agree. This was probably only an issue because some dev I work with didn't read the docs or didn't think it mattered at the time because our userbase was so small the difference between a scan and a query was negligible at the time.

Part of me does wonder if there would be a benefit in DynamoDB providing a function that under the hood creates secondary indices and uses queries where that would be the most efficient choice automatically. But maybe that would be a better fit for an ORM of some sort.

That sounds like a query planner… but it’s fundamentally not really what dynamodb is about. Query planners are amazing but the tradeoff is unpredictable performance - dependent on traffic, data size etc. Dynamodb offers “predictable performance at any scale” precisely because it doesn’t offer this sort of abstraction.

That would have been a killer feature for me. Instead of reading about GSIs, LSIs, and even going so far as having to create a dummy field to do sorting properly[1], it would be pretty nice if you could somehow input the queries you want to do, and have Dynamo make suggestions.

1 - In a collection, I could find no way to easily do 'give me the latest record.', which is super easy in rdb. I ended up creating a key something static like 'x', with date as the range key to accomplish this. If anyone has a cleaner way, please share.

Well what are you doing is ideally lot many engineers should do. But in my experience devs, IT departments, companies are drunk on Cloud Mezcal. So any effort which need effort is "oops thats not our business lets increase instances in Cloud to improve performance" And I am not making this up, A service we are running on ten datacenter VMs, an equivalent service is running on 150 instances of same size in AWS.

The cloud service is crappy architected, developed Java application but instead of fixing it , solution is to double/triple or 10 times the instance using Kubernetes Autoscaler. One thing they have going is ever larger budget since it is next gen cloud service, whereas my serivce being plain old efficient Java app is legacy to be deprecated and replaced with new cloud version.

This has less to do with the cloud and more to do with the common mantra of "premature optimization".

To a lot of engineers there's always bigger fish to fry.

Their documentation is very clear about scans being expensive. Amazon isn't to blame for the poor engineering decision.

That was my initial impression reading the OP as well, if your implementation is slow because you're using scans, well, it's because you're using scans. That said, I honestly don't know enough about the details regarding OP's implementation and why they couldn't refactor to properly Query the PK, but it sounds similar to how you'd authenticate reads on Firebase so maybe it's habits from a different platform kicking in. That said, Amplify allowing users to just get into GraphQL whether or not they actually need it might be a legitimate noob trap.

Eh their documentation is pretty clear about all this stuff and all the new directives released a couple years back (like the PK and index stuff) clearly guide people towards using indexed queries and not scans: https://docs.amplify.aws/cli/graphql/data-modeling/

A lot of this is on the author IMO

Years ago after joining a new team, I was shocked that I just had a GSI to solve a performance problem that was somehow eluding a dozen engineers making around 200k each.

It's funny to read something like this contrast against all the statements about how leetcode interviews do not help and ds/a is something you can learn on the job

> It's just odd to me that I had to make this change in the first place. I wonder how many millions of dollars AWS is raking in just because devs are unaware of things like that.

What "things like that"? Do you mean the basics of querying data with DynamoDB?

https://docs.aws.amazon.com/amazondynamodb/latest/developerg...

This is literally the second topic in any intro to DynamoDB course, that comes right after creating your first DynamoDB table. Even that covers explicitly the basic requirements of running queries.

That's funny because reading SQL doesn't tell you whether the DB will scan or use indexes. With Dynamo DB it's a separate api request, it couldn't get more explicit.