quizotic

Thank you for the pointer to DB fiddle!

Bridgewater is famous for valuing a culture of almost brutal honesty and fierce internecine warring over ideas. It argues this combination is vital to reaching the deep understanding that is critical to the success of its investments. While Bridgewater has experimented with leadership transitions multiple times, changing directions and reversing course seems consistent and faithful to its internal DNA. Life doesn't always go in neat straight lines. Forcing a simple story line isn't always best.

This seems so much more clear to read, with less typing.

Do you do that by memory? Or do you have some text indexing tool that helps?

This is Dan Abadi we're talking about. More importantly, the points he makes are important, clear and true. If that causes heartburn for Vendor A and pride for Vendor B, that's secondary. The primary goal is to help users of distributed database systems understand the kind of trouble they can encounter with anything less than strict serializability.

NB: there are important applications where correctness in specific situations is not paramount. Double-click and ad-serving companies in general are more concerned about speed and throughput and are generally willing to have approximate correctness. Strict serializability isn't a universal value, but it's good to know about when you DO care about the kinds of anomalies Dan illustrates.

Does anyone remember Larry Harris and "Natural" back in the 1980s?

The only problem with singular nouns for table names is that they occasionally conflict with SQL keywords. The canonical example of this is from the TPC-H benchmark which has tables named REGION, NATION, SUPPLIER, CUSTOMER, PART, PARTSUPP, LINEITEM ... AND ORDERS <- plural because ORDER is a SQL keyword.

I like the singular guidance for all the reasons given. I like the consistency guidance. But the real world sometimes gets in the way.

So true. And so deliciously snarky. Confession: I'm guilty of far greater stupidity. I've asked "Ship of Theseus" questions when interviewing candidates for object database companies, and conservation puzzles, like tessellating a chessboard missing diagonal corners with dominoes. Because "Thank goodness I hired coders who can discuss philosophical identity problems". At least people who can reverse a b tree on a whiteboard can code an algorithm.

Don't C++ 20 modules solve this problem?

Do you have any insight into the reason for the unequal access? Is it that poorer or uninsured patients cannot afford non-emergency healthcare? Is it that they are too far away? Is it education/knowledge/social - belief that they shouldn't go to a doctor for "nothing". Are they "too busy".

Also, are there any studies to show that populations in other countries use their medical systems more frequently across the board, or that they use more frequent preventative visits?

You mention the problem with obesity and its co-morbidities. Are there any studies or plots that show health outcomes (lifespan, infant mortality, maternal mortality) as a function of % of population who are obese? I wonder if poor health is linearly correlated with obesity, without regard to healthcare spending...

Agree you need all of these things and more... but if you could do this above the microservice level, as a piece of K8S or whatever the MSA, then you might have an advance over RDBMS.

The result would be the same features/guaranteed you have with current DBMS, but you get independent deployability, elastic scalability, redundant availability, and other advantages of microservices that are missing with traditional DBMS.

If you never need to coordinate two or more microservices, then sure. But that means you're dealing with a monolith. In the original article, you could combine Orders and Users into a single microservice. That would resolve all the issues that are raised ... except that you might want reference the Users in a different context. At that point, you either have to split Users into their own microservice, or duplicate the User information. Either way, you're backed into consistency issues.

Part of the promise of microservice is that they're small modular independent components that you can connect and combine into higher-level services.

If you need to read information and write information to two or more microservices, then you have transaction issues. If you need to relate information across two microservices, you have reference integrity issues. Just comes with the terrain.

There's a bit of an emperor's clothes problem with microservices. If microservices are never combined together, they are essentially monoliths under a cooler name. But as soon as you combine them, you run into the same problems that are solved by traditional shared database systems. Here are two: cross-microservice referential integrity, and cross-microservice transaction coordination, but there are plenty more.

Take the Orders/Customers example of the article. Assume the Customers microservice has some notion of identity, and that the Orders microservice contains a "foreign" reference to Customers identity. If the microservices are truly independent and autonomous, then the Customer microservice should be able to change its scheme for Customer identities, or a delete a Customer. But then what happens to the Orders that reference that now old and outdated Customer identity? If changing the Customer microservice breaks the Orders microservice, then what's the point of the separate encapsulation? Traditional shared database systems have ways to deal with this kind of referential integrity. As far as I can tell, this issue is ignored by microservice architectures. As is the larger issue of cross-microservice constraints (of which referential integrity is just one instance).

The thing that really gets my goat is the lack of cross-microservice transaction coordination. In its place, we get all sorts of hand-waving about "eventual consistency" and "compensating transactions". Growl. Eventual consistency has a meaning that's related to distributed/replicated storage and the CAP theorem. Maybe its principles can apply to microservices, but the onus is on the microservice proponents to actually connect those dots. And most importantly, eventual consistency is implemented and supported by the DBMS, not by application developers. The thought that each microservice will independently implement the transactional guarantees of consistency and isolation should fill everyone with dread. That job should belong to the overarching system, not to individual microservices. It's too hard to get right.

So for now, shared database in microservicew is not an anti-pattern. It may be the only workable pattern. When microservice frameworks grow up and offer the capabilities of shared database systems to microservice developers, then we can talk about anti-patterns.

Is there any protection to be had by a corporate shell game? Suppose I put my technology assets in one corporation, my revenue and monetary assets in a 2nd corporation, and my customer facing presence in a 3rd corporation. The patent trolls descend on the customer facing presence, which has no substantial technology or monetary assets. If it goes into bankruptcy, I lose brand, but can restart elsewhere and preserve my tech and monetary assets ?

Not getting this at all! How does it help find competitive websites?

So I just had an experience of being forced into "vertical slice" development (not quite the same as architecture, but still). This seemed initially as crazy as erecting a multi-story building in vertical slices. But it turned out to have some advantages, and I might do it again by choice.

At the risk of stating the obvious, at the end of the day, it's the use cases and user stories that matter, and in my experience, these almost always evolve over the life of a project. Designing from the ground up often ends up supporting potential requirements that never become real. As the project grows and matures, it becomes hard to trace whether some capability in a lower layer is critical or unneeded, so the tendency is to keep everything. This can become an albatross.

The virtue of vertical development, is that it's easier to do all the work and only the work that is required.

Theoretically, the flip side is that it's easy to avoid thinking up front about the big picture, and easy to promote stove-piping with all of its disadvantages.

My guess is that either the traditional or the vertical approach involves eventual refactoring, and that the first leads to bloat in unused capability while the second leads to bloat in duplicated capability. But the bloat comes later in the project with the vertical approach. In the early phases of a project, the vertical approach may lead to leaner, more malleable systems, with faster delivery and user interaction.

Bottom line for me, is that vertical development didn't fail (I thought it would), and had some surprising benefits.

Glad to hear about the FDWs ... but wouldn't foreign keys be your friend here? What else would you need, really, to layer this on an existing DB?

Well ... I kinda like the query syntax's ability to traverse relationships, nest objects/sub-headings, and apply filters at the level of projected expressions!

This has been done before of course, but I'm not sure I've seen this combination of syntax on an RDB before. It's certainly easier to read and write that a bunch of outer joins.

What would be nice to know is whether this can work on top of an existing PG database, providing easier syntax.

While their last example makes _sense_:

  open_prs := User.<assignee[IS PullRequest] {  
      title
  } FILTER .status = 'Open'

I don't find it easy to read/understand. If the [bracketed expression] acts as a filter, why not:

  open_prs := User.<assignee[IS PullRequest AND 
                            .status='open'].title

which preserves the nice dot-chaining for link traversal.

404?

Beautiful. Data Flow + Functional. Love the idea and can't wait to try it out!

Heck of a title. Wish the article was better at explaining what was actually done.

The article argues that conservative views are correlated with fear/concern over physical safety, and that increasing the feeling of physical safety moves conservative views toward more liberal views.

If true, it makes me wonder whether the balance of conservative and liberal populations confers survival value to the whole population. Surely there are times when there is more or less risk to our physical safety. Perhaps the liberals tend to be more blind to it when risk is more probable; and perhaps the conservatives overestimate it when it is less probable. Might we need the presence of both views to keep us safe when we're at risk and to loosen the reins for exploration when we're in less danger?

+1 for the idea of tuning query optimization based on ML. I don't know of any DBMSs that take this advice, and that's remarkable in this day and age.

The prescription of changing sequential_page_cost to equal random_page_cost is certainly reasonable for SSD, but I wonder if the underlying issues aren't somewhat deeper and more interesting. One difference between a sequential scan and an index scan is the amount of data being scanned. PostgreSQL stores information horizontally as rows and a sequential scan will have to read in all column values of all rows. An index scan will read through all values of a _single_ column. The 50x performance difference _might_ be just that the whole row is 50x wider than the width of the indexed join column.

An interesting second factor relates to the nature of the SSD storage. With SSDs a read request will pull back a 4K page, even if the read request was smaller. So it's not quite right to say that a sequential read and a random read cost the same on SSD, particularly if the same 4K page must be read multiple times. I suspect that the particular index technique used by PostgreSQL tends to organize data such that successive indexed values reside in the same 4K SSD page. IOW, it's not so much that the cost of random SSD access is the same as sequential SSD access (though that's true), as it is that the PostgreSQL index mechanism doesn't require multiple reads of the same 4K page.

if a Hash-based index was used instead of a Btree-based index, and if the table width was narrower, the sequential scan might have outperformed the index scan.

Jim Starkey (founder of NuoDB) is rumoured to have said that any adjective before the word "consistency" is equivalent to the prefix "in" as in "eventual consistency is inconsistency". A bit harsh, and not completely true, but a good warning to a world that increasingly believes eventual consistency is the best way to handle the CAP theorem.

Yeah, another case of "blame the person" instead of "blame the lack of systems". A while back, there was a thread here on how Amazon handled their s3 outage, caused by a devops typo. They didn't blame the DevOp guy, and instead beefed up their tooling.

I wonder whether that single difference - blame the person vs fix the system/tools predicts the failure or success of an enterprise?

While it's good that you know about this optimization, the key point is look at the evidence first. It's surprisingly hard to temporarily ignore what you know, and just use your eyes, but it's so important.

In the early 1990s, I'd co-founded an object database company, with a standard "east-coast-style" non-compete, which among other things, granted us injunctive releif. Our top developer left to work for our main competitor. We sued, and the courts ruled basically that there is no slavery in the US and our developer had every right to earn a living doing what he knew how to do. Maybe laws have changed, and maybe it varies by industry, but my experience is that noncompetes are meaningless. BTW, I don't particularly wish they had teeth, and my company was probably not significantly harmed by the outcome. Just saying I wouldn't sweat too much about signing a noncompete.

Gosh I like this! Eliminates the artificiality. Eliminates the competition and the anxiety. Replaces all that with something natural.

Why is this approach so rare? What's the downside?