ndriscoll

That doesn't do anything to help with the issue being discussed, which is index fragmentation and random IO affecting performance. It just means now you have two indexes to maintain.

I suppose that's fair, though in my experience nobody seems to think it's crazy if the computer takes over 1s to respond to a human, so 200 ms is kind of small potatoes, and for most cases you can have a local read replica, so only writes need be slow.

Yeah, I've never used Claude, and generally use the codex CLI with whatever base prompt that implies, and I've found that it's always been very clear and concise without goofy "-isms." It just comes across as an intelligent colleague. It even makes effective use of bold and italics for emphasis. I'm a little mystified reading all of these comments.

On the other hand, one modern server is 10,000x faster than a 1990s server with 10,000x the capacity, and modern RDBMSs are better at executing queries and give you more options for index types, so the 1990s single-server approach goes very, very far.

I suppose that over time we'll also see more data sovereignty laws, which gives a pretty natural partition key (i.e. for most data you needn't have a global serial because putting them in one database where you'd be worried about conflicts would be illegal anyway, and if you do have some place where you can do those joins, you can adjoin a partition id).

Yes, you cannot be in one order and also a different order at the same time. You can make another index, but that's something else entirely.

How would it internally sort your index by (timestamp, ID) without you later having to provide the timestamp to locate the ID?

It's been a while since I've touched db stuff, but pg still has to write the indexes before commit completes, no? So it's still having to go load and write a gazillion pages. And then unless your filesystem has atomic page writes like ZFS, it also writes the whole page to WAL to ensure that it doesn't do torn writes. Writes to the same page within a checkpoint don't need full page write, so benefit.

Higher density for indexes you use for joins. Also you can then use UUIDv4 to hide creation times if that's sensitive too.

Yeah but for nearly everyone, they'd get better performance from not using a distributed database, and benefit from "hotspots" i.e. data locality.

That's a bluray rip. AFAIK every streaming service is way lower bitrate than that, even at maximum quality.

There's an 18 TB refurb from serverpartdeals right now. Or there are a bunch of smaller new ones in the $20s on Amazon. The analysis doesn't really change even if you're looking at e.g. $30/TB. It's still hundreds or thousands of items for a year of subscription cost; more than you could reasonably watch. Even going full 8 TB SSDs at $150/TB is still thrift store cost per movie when you amortize it ($2.25 at 15GB, or $0.30 at 2GB).

Of course then as another comment here said, you're more likely to realize you don't actually want any of it. And then when you realize it's not worth pennies to store, you also realize it's not worth your time to watch.

There's probably something to be said for being inspired by people living their dreams. Like it was no doubt inspirational for many to watch Jordan living his dream of being a baseball player no matter what people expected of him.

Aren't even the "high quality" 4k streams like 15GB/movie? So at $20/TB that's ~$0.30/movie at streaming quality? And you could still have probably better than DVD quality that people used to pay $20 for for like ~2GB, or $0.04.

> The manufacturer typically uses anticompetitive business practices, so they should be allowed to use more anticompetitive business practices

Or we could say that product tying is illegal, which it already is. Why do we care about specific monopolistic manufacturers' business strategies for mid tier commodity computers? It should be plainly illegal for Nintendo to try to stop people from running a Nintendo game on a Sony or a Dell computer, or to stop Sony from writing a game for a Nintendo computer. Or for Sony to stop Nintendo from doing those things. There's literally no reason to support this "you need a special X brand computer to run X games." If they can't sell hardware on their own merits, it's because they're not good at hardware, and you were being forced to buy it for the software you wanted.

Or if the jury acquits, they are asked whether it was reasonable for the state to prosecute the case for each of the charges. If ≥50% say no on any charge, criminal charges are automatically brought against the prosecuting attorney and judge for malicious prosecution. Conviction carries mandatory disbarment plus prison time.

Many corporations (their actual customers they're broadcasting to when they say how smart their models are) have requirements (externally imposed, so not something they can just change) that all code is reviewed, so they need it to be understandable, which is basically the same problem.

The vulnerability also required you to log arbitrary user provided strings and maybe some other precondition that I can't remember. Nothing at my workplace at the time was affected despite using log4j since user provided data was either parsed (e.g. a uuid) or private/not something we would log.

Other vulnerabilities are possible, but still much easier to rule out in a language with a decent type system and macros (e.g. Rust or Scala).

F=ma is slightly important for physics, no? And deriving the equations is kind of the point; it's the understanding of "why" something is true.

e.g. I can tell you things act like a spring near (stable) equilibrium, and you can believe me, but why on Earth would that be true? Well, Taylor expand the potential at equilibrium. Define V(0)=0, and equilibrium means V'(0)=0, so V(x) ~ V''(0)/2 x^2, so F = -dV/dx = -V''(0)x, which is a spring with spring constant k=V''(0).

That's all a very conceptual explanation; the point is near an equilibrium, zoom in and you have a parabolic potential! What is the potential for a spring? A parabola! This is basic Calc BC stuff, but it also justifies why we might care about about simple harmonic motion. Calculus is the whole business of "zooming in."

"World expert" might not even be that bad of a characterization. It's familiar with the ins-and-outs and every major distribution, every DE, is fluent in debugging at any level (knows every network utility, container technology, systemd option, sysctl, how various subsystems plug together and how to use them in practice, etc.), knows how to look up what it doesn't know if it's not up to date to within the literal last month or two, and if it came down to it, could patch/fix things, including reverse engineering hardware drivers.

I wouldn't be surprised if it could better deal with Linux issues than Linus.

If you `git blame` the software you're interested in, I can't imagine it should take too long to find the last commit that had the version you want. Then you could either use that, or for the first commit with the new version, use commit~. Or anywhere in between depending on whether dependencies are important to you (that's at the limit of my git-fu, but I imagine you could do some kind of restricted blame to then find the dependency version you care about if you do care, and recursively do that until you have exactly what you want).

If you have a CI system that's building containers (e.g. using flakes), you can have the nixpkgs version you're using pinned in your own config, so you can just look at your own (smaller) history.

This should all be way easier than trying to do it on any other distribution.

Tor sounds like just about the worst possible way to hide what you're doing from your employer. Why would you not proxy through... your house?

You can install a single package from a specific nixpkgs commit and it will pull in whatever dependencies it needs to make that work, with everything as of the time of that commit. You may have duplicated libraries but they can coexist fine, and you'd have that with docker as well (probably even worse unless you are very intentional about building layers).

I imagine people do this somewhat regularly at least to e.g. occasionally pull in a package or two from unstable when they normally run a stable system.

The point of batching is that high requests/business "transactions" don't necessitate high concurrent database transactions. The thing that matters here is actual workload. How that turns into transactions is something you can tweak. And once you have a decent workload, you can build batches with dozens of items in a couple milliseconds, so you can still keep single digit millisecond latency with batching.

Yes, if you're architecting around slow disks instead of just using fast ones (which cost a fraction of the development cost of what you're saying), and if you do things like give up ACID in service to that, you're crazy and incompetent.

It's not magic, but for 99.999% of people, magic is not required. So again, like I asked twice, what is "extreme scale"? Weirdly, no one who brags about "scale" is ever willing to say what that means, and everyone saying they've never encountered this "scale" is willing to put concrete numbers behind it like 100s of TB at ~1M QPS or whatever.

At 100M rows in a table pg can do just fine with single digit ms latency with 5 digit request rates. Is that your "scale"?

Well it would hardly be blood money to destroy free society and replace it with a surveillance hellscape if you could make just as much elsewhere.

Okay, again, what kind of scale are we talking about here? 100ks of writes/second? Millions? For data that can't do reads from a replica (presumably not since you're saying use eventual consistency)? Always absent from these kinds of blogs are concrete numbers for what kind of volume they're discussing, making them completely useless drivel. Also that blog is saying they were running into trouble in early 2014 and started this project; okay, the first NVMe drive was mid 2013. Did they do that first? They totally could've been using racks of 10k RPM disks around that time with maybe SSDs for the indexes. You'd be way better off going all NVMe today as your first step to "scaling" (in fact you are insane if you are using anything else for an OLTP database), and just that is likely to immediately solve whatever problem you have already.

If postgres can already do partitioning, batching, different tablespaces on different disks, etc. except do it correctly with ACID, then what exactly is the benefit of manually implementing what it already does well?

Also how does going "schemaless" help with scaling at all? That's just asking for data integrity issues for no benefit. Peak mid-2010s money-furnace resume-driven "engineering."

Because they've been through this process as part of learning math. e.g. you read your analysis book, and come to:

> If a set E in R^k has one of the following three properties, then it has the other two:

  (a) E is closed and bounded.
  (b) E is compact.
  (c) Every infinite subset of E has a limit point in E.

You stop reading, and you figure out the proof. Once you're satisfied, you continue reading the book and compare with its proof. The people who managed to get far enough that they were doing research have almost certainly gone through this exact process. They've self-selected as individuals who are able to motivate themselves to learn things even when they're already known, and to be able to figure things out without being given the solution. There's an insane amount of math to get through to get anywhere near the frontier, and it all builds on itself such that you can't just skim and skip, so they must have already learned to be able to do this.

Likewise, when given an exercise of the form "Prove P" or even "Prove or disprove P," you can be pretty certain 1. it is doable and 2. it is doable using techniques/theorems discussed in the course/book you're working through. But people are perfectly happy to do this for several years.

What is extreme scale here? Like you've been able to fit over 1 PB of NVMe in 1U for years now (and before the AI inflation, prices were low enough that you could do it for like $100k IIRC). Are we talking 10s+ of PB? Who actually needs this scale?

At mere 100s of TB of data, I think you'll find that a normal database is far more likely to scale better. Most developers are... not as good as database developers at maintaining a database, and will not do a good job of it when they write their own inside of their application. They're often not even that good at using a database.

e.g.

> You also get to have your indexes in dedicated shards on dedicated nvme hardware. With batching, you can reduce the contention on indexes even further

You can just do this with postgres. It works great. Most people don't even get that right. And they're going to suddenly get it right and implement efficient indexes?

If that were the case, then people would have unmet needs, and so there would be work to do. You need to keep your proposed scenario internally consistent here.

The sites that are there for ad impressions already worked this way though. The incentives were geared toward poor human discussion as well (c.f. reddit as a best case, and e.g. instagram/tiktok/youtube as what they'd like to be). So the problem here is ads, a desire to grow to include the entire world to show them ads, and a desire to do whatever it takes to keep their attention to show them ads.