sgarland
If there’s another option I’m unaware of, I’d love to hear it. Also to be fair, I didn’t evaluate Sid for this, only Stable.
I suspect that the average Cisco customer expects a bit more stability than $RANDOM_SAAS.
I don’t know how people who’ve worked with Python and been frustrated with its performance don’t know about Numba.
In contrast (I know this is apples to oranges), Python’s stdlib is incredibly broad. colorsys lets you convert between color spaces. plistlib lets you read and write Apple’s property list files. graphlib has a topological sorter, which can lead to hilariously short interview questions when the interviewer isn’t familiar with Python’s stdlib.
> Additionally, think resource constrained devices like mobile phones.
SQLite will saturate the phone’s I/O before you saturate SQLite.
> Or an aggressive cost reduction mentality where each cpu cycle shouldn't be unnecessarily wasted without feeling guilty.
> Our database Dip stores raw binary as well as json as first class citizens.
These two statements are in conflict. If you want to minimize CPU cycles, parsing JSON is a terrible idea, and you should be pushing everything into bitfields, operating on it with bitwise manipulations.
> If you come from a systems engineering background like that, you wouldn't be much impressed with many traditional databases at all. I am generally not much interested in SQL databases anyway.
I would love to see a Jepsen analysis of your DB. Then a benchmark of it vs. Postgres, or MySQL, or SQLite.
IMO, if you read the later post on Docstore [0], you can see that they spent an enormous amount of time building back everything that they gave up with Schemaless - the enforcement of a schema (the irony...), transactions, serializability, etc.
They needed to create secondary indices as tables because they were storing everything in JSON blobs, which is difficult to directly index now, let alone in the 2010s. You can of course argue that they needed to use JSON blobs because they kept needing schema changes, and online DDL back then was difficult. I would counter that if you keep needing schema changes, you haven't done a good job modeling.
Everywhere I have worked that has DBs they claim are too large, I have been able to prove (typically by cloning one and remodeling it) that the reason they're suffering from performance issues is because they've made a series of decisions which were fighting against their RDBMS: UUIDv4 PKs, strings for low-cardinality data, excessive secondary indices, poor query plans from the ORM, massive JSON blobs that's duplicating already-present data, etc. At my last job, I showed a 30% speedup purely from changing to a PK that made sense for the table's usage.
I am not asking for hand-holding, I’m asking for one single publicly-available example of this being done, because I cannot fathom it being necessary or beneficial, and you have not provided concrete proof.
You should, of course, test it on your own data before deciding.
Same with locks / contention: non-locking reads don’t cause contention, or at least, none that wouldn’t also occur with a separate table acting as the index.
Do you have any examples of this being used at scale, preferably with an accompanying blog post or other form of documentation?
Clustered indices would be so, so nice for Postgres to have. OrioleDB can’t come quickly enough.
> However, this stamp is applied only to the row itself, not to any indexes referencing the row.
Indices also hold the hint bit LP_DEAD, which is set by any query that follows the index to the heap and finds that it’s invisible to everyone.
I suspect that a combined query that uses pagination rather than LIMIT … OFFSET would be more performant here, and could skip the dead tuple problem.
DELETE FROM step_outputs
WHERE ctid = ANY(ARRAY(
SELECT s.ctid
FROM step_outputs s
WHERE (s.created_at, s.id) > (?, ?)
AND s.created_at < ?
AND NOT EXISTS (
SELECT 1 FROM workflow_status w
WHERE w.workflow_uuid = s.workflow_uuid
)
ORDER BY s.created_at, s.id
LIMIT ?
))
RETURNING created_at, id;
Are you suggesting that one should recreate B+trees in code?
Then I moved to one in North Carolina that was god-awful. Dues started at $65/month and went up every year. They’d send out neighborhood reminders that a pressure-washing service would be in the neighborhood next week if anyone wanted to hire them; anyone who didn’t got notices a few weeks later that they needed to pressure-wash their driveway and/or house. I also had to buy a specific brand and sheen of paint to repaint my mailbox; of course, the lettering was only available from a single supplier.
I now live on an acre lot that sits between two neighborhoods. I have no HOA, and it is glorious.
Personally, I like regexes, in the same way that I like shell string manipulation: it looks utterly inscrutable, and performs magic. If I write one in code meant for others, I usually use extended mode with each line commented.
Fantastic analogy, and I assume you already know where I'm going with this.
The Titan submersible [0] implosion was by all accounts due to studiously ignoring best practices and test results. They shipped fast.
In contrast, James Cameron's Deepsea Challenger [1] was a slow and methodical build, and has heavily redundant backup systems for everything. For example, the ballast release system operates by demagnetizing coils that hold the ballast; that way if they have a system failure or power loss, it fails open, and they ascent. If that somehow fails, the supports for the ballast will corrode from sea water before the oxygen runs out, so in the worst case, it's sitting in the dark for a few hours before heading back up.
Or, let's look at the U.S. Navy submarine program. I was a propulsion plant operator (reactor operator + throttleman) on a Virginia-class sub. Though I can't go into specifics, "redundant" doesn't begin to describe it. FFS, the San Francisco (688-class sub) ran into an underwater mountain at flank [2] and survived. The SUBSAFE program [3] was a direct result of the loss of the Thresher [4], and to date, no SUBSAFE-certified submarine has been lost.
> Then look at software. When is the last time you encountered a bug? Have you ever heard of bug free software?
You're conflating "free of defects" with "capable of gracefully handling defects." NASA writes as close to bug-free as I think you'll find, but even theirs isn't perfect [5]. The difference is the ability for the software to work through or around those problems. Submarine / submersible design and construction is the same. Are the engineers, welders, machinists, electricians, etc. attempting to do perfect work? Of course. They also are aware that perfection is impossible, and so the design accounts for that.
0: https://en.wikipedia.org/wiki/Titan_(submersible)
1: https://en.wikipedia.org/wiki/Deepsea_Challenger
2: https://en.wikipedia.org/wiki/USS_San_Francisco_(SSN-711)#Gr...
3: https://en.wikipedia.org/wiki/SUBSAFE
4: https://en.wikipedia.org/wiki/USS_Thresher_(SSN-593)#Disaste...
Rows store all columns together
Not guaranteed. The DYNAMIC (default) and COMPRESSED row formats [0] can shift larger values of variable-length columns in off-page records. In the example given, this wouldn’t occur, but it isn’t a guarantee per se. This is roughly equivalent to Postgres’ TOAST, though Postgres’ default threshold for TOAST is 1/4 of a page (2 KiB with its default 8 KiB page size), whereas InnoDB is 1/2 of a page (8 KiB with its default 16 KiB page size).
0: https://dev.mysql.com/doc/refman/9.7/en/innodb-row-format.ht...