>Overall, though, the product is very solid and it's one of the few database that you can say "just works" when it comes to scalability and reliability (as most AWS services are)
How those two can coexist?
source: currently being burned out on an adjacent aws team..
if you want to know why capitalism causes this, start a startup and prioritize quality, do not get to market, do not raise money, do not pass go, watch dumpster fires with millions of betrayed and angry users raise their series d
Alternative would be to attempt a near 'perfect' solution for the product requirements and that may either hit an impossibility wall or may require substantial long term effort that would impede product development cycles. So likely the former approach is the smarter choice.
Enjoy the sausage, but if you have a weak stomach, don’t watch how it’s made.
(I work for AWS but not on the DynamoDB team and I have no first-hand knowledge of the above claim. Opinions are my own and not those of my employer.)
This is true though there's only so much technical debt and internal process chaos you can create before it affects the outcome. It's a leading indicator, so by the time customers are feeling that pain you've got a lot of work in front of you before you can turn it around, if at all, and customers are not going to be happy for that duration.
Technical debt is not something to completely defeat or completely ignore, instead it's a tradeoff to manage.
One concrete problem with technical debt the article highlights is it that negatively impacts the time to deliver new features. Customers today usually expect not only a great initial feature set from a product, but also a steady stream of improvements and growth, along with responsiveness to feedback and pain points.
Additionally, the business cares about the outcome, not the internal process.
Ostensibly, the business should care about process but it actually doesn't matter as long as the product is just good enough to obtain/retain customers, and the people spending the money (managers) aren't incentivized to make costs any lower than previously promised (status quo).
This is designed to reduce the chances of eager employees going out and astro-turfing or otherwise acting in trust-damaging ways while thinking they're "helping".
When this is the case it’s often nice to state this conflict of interest, so others can take your appraisal in the appropriate context.
I’m not implying anything about the post, just stating what I assume to be the reason for the disclosure.
If the customers don't have any feedback or missed feature asks at launch, you waited too long to ship.
You know who has great internal code and test quality? Google. Which is why Google doesn't ship. They're a wealth distribution charity for talented engineers. And their competitive advantage is that they lure talented people away from other companies where they might actually ship something and compete with Google, to instead park them, distract them with toys, beer kegs, readability reviews, and monorepo upgrades.
To me the takeaway is large/interesting/challenging engineering projects are pretty close to disasters generally. Some time they do become disaster actually.
On the other hand if a project looks like straight up designed, neatly put into JIRA stories, and developers deliver code consistently week after week then it may be a successfully planned and delivered project. But it would mostly be doing stuff that has already been many times over and likely by same people on team.
At least this has been my experience while working on standardized / templated projects vs something new.
Some projects are run like the Doomsday Clock, and nobody can get anything done. Other ones increase and decrease on complexity all the time, and those tend to catch-up to the first set quite quickly.
Cassandra is too opinionated and its CAP behavior wasn't great for a service like this, so they built on top of Riak. (This also eliminated any thoughts I had about Erlang being some uber-language for distributed systems, as there were (are?) tons of bugs and missing edge cases in Riak)
Not Invented Here can run very deep in some branches of an organization. Depending on how engineering performance evaluations work, writing a homebrew database could totally be something that aligns with the company incentives. It might not make a single bit of sense from a business standpoint but hey, if the company rewards such behavior don't be surprised when engineers flush millions down the tube "innovating" a brand new wheel.
Questions were handwaved away, and the usual Amazon black box non-answers which always smells like they are hiding problems.
Any ideas how this is working? It seems bolt-on and not well thought out, and I doubt they'll ever pay for Aphyr to put it through his torture tests.
Consistency and conflict resolution
Any changes made to any item in any replica table are replicated to all the other replicas within the same global table. In a global table, a newly written item is usually propagated to all replica tables within a second. With a global table, each replica table stores the same set of data items. DynamoDB does not support partial replication of only some of the items. If applications update the same item in different Regions at about the same time, conflicts can arise. To help ensure eventual consistency, DynamoDB global tables use a last-writer-wins reconciliation between concurrent updates, in which DynamoDB makes a best effort to determine the last writer. With this conflict resolution mechanism, all replicas agree on the latest update and converge toward a state in which they all have identical data.
I've seen this kind of thing mentioned many times, pretty baffling TBH based on Dynamo's pretty good reputation in industry. Are these mostly to the stateless components of the product, or do they see data loss?
There are times when bad deployments happen and customers were impacted.
Engineering, however, was a disaster story. Code is horribly written and very few tests are maintained to make sure deployments go without issues. There was too much emphasis on deployment and getting fixes/features out over making sure it won't break anything else. It was a common scenario to release a new feature and put duct tape all around it to make sure it "works". And way too many operational issues. There are a lot of ways to break DynamoDB :)
Overall, though, the product is very solid and it's one of the few database that you can say "just works" when it comes to scalability and reliability (as most AWS services are)
I worked at DynamoDB for over 2 years.