zbentley
Computer systems, though, can have that distinction. If you read data supplied by the user, and then process it via behavior in code, and process that result against administratively-set data in the database, the separation between user-supplied data, administratively-configured data, and instructions (code) is extremely clear. Even if you are doing something that could be nondeterministic, e.g. incorporating randomness or doing sentiment analysis or whatnot.
I don't think the oft-repeated "AI is fallible just like humans are" applies here. Computer systems are an area where that separation can be achieved. We know this because we've achieved it a lot. LLMs, powerful as they are, are a step back in that regard.
That's not because investors are idiots. A little bit of it is because a certain amount of credulity/faith is a necessary skill in that business. But most of it is just because humans are fallible and trickable in general. Advice from an expert can only help with that so much.
And bugs like “WHERE 4 = 5”, or selecting from nonexistent columns, or selects permanently modifying table behavior, or any of a dozen others that Jepsen found … didn’t happen in released databases. Even young ones.
Prior to the age of LLMs, that level of bugginess wasn’t “everything has growing pains”. That’s the kind of thing previously associated with poor undergraduate CS homework or “the CEO’s kid wrote our database” corporate horrorshows.
> with no QA
That’s not immaturity. That is a choice. A “the car we are selling does not have airbags, seat belts, or brakes” kind of choice.
> Maybe you don't know too much about engineering
Save it. You have no idea what you are talking about.
I agree with GP, that vendors should be somehow required or very strongly encouraged to participate in such audits. Not because they want to (as you say, most wouldn't like their dirty laundry aired thus), but because consumers deserve to know. U.S. government software groups don't like FISMA/RMF audits either, but they have to do them anyway.
More cynically, I suspect that many low-quality database projects succeed or fail based more on marketing and hype than robustness. A dishonest product could simply take the line "engineers at Orkes were able to produce a highly sophisticated database system" from the report out of context, quote it on their homepage attributed to aphyr/Kyle/Jepsen, and reap short-term rewards and popularity. I hope I'm wrong.
That’s not to say you can’t have operations experts or ops-only roles. You can. But if the developers don’t participate in operations at all, they are doing themselves and the business a disservice.
It’s exactly like having a chef who doesn’t taste food, or a car mechanic that doesn’t drive. Some savants exist who can do good work even then, but they’re vanishingly rare.
However, statements around the internet (and in these comments) are less than definitive as to whether a) that proxy has a cache that was compromised; b) that proxy has select extensions which can be published directly to it by hand (one of which was compromised); c) other.
There also might be a need for "downgrading" cross-compilation profiles a la "don't compile/AOT-cache this app for the current CPU, compile/cache it for <the kind of CPU that I have 10k EC2 instances running>". You'd think that would be easier to achieve by just compiling on a production-like machine, and you'd be disappointed by the number of shops running Java apps hand-built on a Windows laptop across millions of cloud Linux cores.
Server-Side/Desktop JavaScript, I think, has a much better/more polished deployment experience than Java...at the cost of having a much worse build (ES module hell, bundler hell...) and maintenance (dependency hell, security hell, constantly changing APIs hell...) experience than Java, which excels at both of those.
Pop quiz: across all kinds of FDs (including block files, dirfds, sockets, signalfds, eventfds, lock descriptors, /proc files, tmpfs files, timerfds, and so on), which features of {select/epoll, io_uring, O_NOBLOCK, inotify/fanotify, etc.} are supported?
...or people who had a test/deploy/debug cycle on environments that necessarily had no layer cache (ephemeral VMs, one-shot kubernetes clusters, OpenWRT routers that lost the layer cache on reboot). Or people with network-constrained traffic from registries or mirrors thereof.
Or people in environments that took container security findings (perhaps too) seriously and were tired of triaging endless reports for random dependencies in "FROM ubuntu" containers, and wanted a default required-by-policy base image that didn't come with the world.
Or people who needed a relatively full-featured build environment against musl without having to do bizarre incantations in e.g. a Fedora image to switch away from libc while still being able to link against other system libraries.
Most of the time, though, you're right. Take it.
If you do all that on the client side (not in a sidecar, ideally: actually in your client code when deciding what to transmit) you have a fairly nice way to temporarily turn on e.g. rich trace recording to hunt a bug, or start spraying event bodies to a local file when debugging a troublesome service instance, and so on. The rest of the time, things can transmit a few big payloads and a lot of small metrics, and things stay cheap.
In other words, it's easy to go from "recorder.record_rich_typed_event(...)" to capturing any or all of events, traces, metrics, or logs. It's not easy to go from e.g. "metrics.record_integer_metric("somemetric", 15)" to recording a richer, more complex event in a pinch.
> create new account
I don't think "need to create an account to figure out that you can't contribute with a given forge" is a real problem. They're just as publicly browsable sans-login as GitHub, for the same reasons: people want to share their stuff. And when you do need to make an account, most software forges have SSO with the big/common providers.
> find it archived, google for new repo, find that one archived, go to new repo
You needed to RTFM and do a bit of research to figure out where/how to contribute for GitHub projects too. They got archived/moved around/invisibly-forked just as often as people move forges today. How many times have you seen. "FizzBuzz is archived and no longer accepting contributions; an actively maintained rebuild of this project is at https://github.com/somebody/FizzBuzz-ng"?
> find they now take patches via email instead of PRs
Plenty of GitHub repos either don't accept contributions via GitHub PR (Linux) or require lots of nonstandard hoop-jumping to get a contribution approved (Python).
> 2-3 different protocols
Git is the protocol. I don't think the move away from GitHub is making anyone move away from Git. hg/jj/svn are still minority tools for the same reasons they always were.
> one index
It's Git. The index is folders on your hard drive. List their origins if you forget the URLs of their forges. Make a browser bookmark if you don't want to keep the checkout around.
Under the most charitable interpretation of your complaints that I can come up with, contributing in a not-universally-GitHub world is like ... 1% harder per repo contributed to. At most. To the tune of, at the very absolute worst, like 2 minutes of extra work when making your very first contribution to a project. I don't think it's that hard to stop googling "ProjectName github" and start googling "ProjectName git source" or "ProjectName contribute code".
The only alternative I can think of is "own and operate all of your site's hardware on physical premises to which you have sole access", and even that's not perfect.
Are you arguing against hosting/cloud/datacenter services in general? Or is there a specific technique you would prefer people use to secure their keys in a non-owned environment?
- If a vulnerable dependency is in your environment as a transitive of something that's not at risk (e.g. code you don't use, code that doesn't talk to untrusted sites), code that shouldn't use it will pick up that transitive.
- Autodetection of optional stuff is often silent. E.g. libxml vs. pure-python XML parsing doesn't really act any different at load time or most of runtime, besides being faster, so it's non-obvious what's being used when.
- Probing for packages that aren't listed as a concrete dependency by importable name risks loading unintended code due to name conflicts (malicious, or accidental/caused by local files with the same name or weird PYTHONPATH).
- There are plenty of ways to get an ImportError after running a lot of Python code in a dependency before giving up! That code might mutate global state/change behavior, or do insecure things, etc. Malicious actors could take advantage of ImportError being raiseable by user code to conceal naughty behavior at import time.
I really want Python to come up with a saner way to concretely identify (avoiding name conflicts) and probe for optional dependencies at import time. try/except-ImportError ain't it.
Basically once you leave the realm of the 5-10 most popular building blocks in an area (which OS packagers do distribute), you're still left with plenty of packages that have 10ks+ of users, sane maintainer/security/ownership models, and code that would be very difficult to just copy-paste a la stackoverflow (e.g. native/compiled deps with complex wheel build tooling).
What?
Step 1: go to project's manpage/readme or website.
Step 2: find the URL of their software forge (git/svn/hg site).
Step 3: clone and contribute using that website.
That process has been largely the same for a lot more than 20 years. Most people were not using GitHub as a discovery/random-repo-browsing system.
Great for professionally administered servers or expert power users. Not for much else. Until recently, esoteric use-cases like playing audio were catastrophically unsolved. And then those got better around when screen sharing, accessibility devices, and remote-assistance tools all broke because of another OSS schism. Installing and running an ordinary program requires choosing among a handful of ways of doing that (container systems, package managers) even on the same distro. And then if a graphical program wants to do crazy stuff like "open a file", well, the permissioning situation is "fragmented" to put it generously.
Gaming? Gaming works great on Linux now! ...because a corporation with centralized planning, KPIs, and incentives spent hundreds of millions of dollars on it.
I like Linux. It's been my daily driver for years, on and off. I use it for a living, I use it for a hobby. But that doesn't mean it's not a perfect example of every single one of the OSS flaws that GP listed.
Cooperative concurrency: it depends. It's sometimes the right move, but has drawbacks. It's usually implemented to enable efficiency or partial parallelism of kernel-space code (IO). There are plenty of platforms that absolutely need it for that purpose: c10k, nodejs's rise to fame, all that. It behaves poorly when mixed with non-cooperative (blocking) code, and is poorly understood by many folks who expect userspace code to be parallelized as well. A common claim is that cooperative concurrency is harder to use than pre-emptive, but that's not true: some common things are harder, but some less common, but more dangerous things like race conditions tend to be much easier. Tokio and other cooperative systems that schedule onto fixed kernel threads can help with the hard parts a bit, but it's a partial mitigation at best.
Preemptive userspace concurrency: firmly disagree with your original point. Green threads are usually combined with userspace-code parallelism (e.g. goroutines scheduled onto multiple threads) and efficient kernelspace interactions (IO multiplexing). In the popular implementations of green threads I know well (Erlang and Go), this is an unalloyed blessing, in that it both removes many of the dangerous edges of threaded programming, facilitates increased efficiency, and lets users semantically model concurrency/parallelism with much less fear of e.g. creating too many threads, torn reads, etc. That's a big benefit: now we can model things like actors or CSP as subprograms written in procedural, synchronous, ordinary-looking code without getting Ph.Ds in operating system internals or having to carefully reason about complex interlocking layers of callbacks and futures to minimize risk of blocking or deadlock. In many ways, this brings the good parts of single-responsibility/what folks call "the UNIX philosophy" into a single process with minimal pain.
Rare cases where the userspace-concurrent runtime fights with your ability to interact directly with the OS, as with Docker's fork/thread/process-global-state challenges, do emerge. But I don't think those are primarily because of userspace concurrency; rather, they're because it's hard to do low-level things in a high-level, managed runtime. You can fork in the JVM, too, and you'll have problems. Different ones, because the runtime's different (e.g. endless RSS burn at idle due to the child GC moving parent objects), but problems all the same. Same for Python: forking in the presence of regular OS threads and/or a running event loop is equivalent to arming a minefield; doable but troublesome, just like it was for the Docker folks with Go.
To be fair, some of the same pains as the cooperative model appear with pre-emptive, too, when userspace concurrency systems are retrofitted onto runtimes that do a lot of other kinds of concurrency as well--but that's because of the hybrid/retrofit nature of the product, and not because of green threading as a concept. Java's rocky adoption of virtual threads is an example of that.
For Go: you're likely right that OS thread overhead was the main reason for Goroutines, but I don't think it was the only one. There is also a ton of ergonomic efficiency to be had via green threads; like async/await, they're kind of a syntactic unlock that allows you to very liberally use goroutines/channels as a means of semantically modelling actors/state/data flow in your program, where doing that with regular OS threads is both dangerous and (given the start/stop overhead of threads on lots of other platforms) feels like kind of using a howitzer for a flyswatter. Golang saying "yeah, create a million of these, we don't care, it's 4 machine instructions per to start one" was pretty liberating for a lot of people.
The goroutine architecture also unlocked some nice performance at the OS level: GC latency improvements became easier, and some IO multiplexing efficiency a la epoll became almost free to implement due to the ability to cheaply move and suspend goroutines.
> rule broken by almost every language in common use
I don't think so. Go/Erlang/Java do break it (Java very recently). C/C++/Rust/C#/JavaScript/Ruby/Python/Perl/TCL don't break it. PHP ... sort of breaks it, (IMO they just hide cooperative nonparallel concurrency in a nice API, but I'm not well-informed on php)?
And for most people, that's a bad idea. Most people don't have to know what RPATH, ldd, multiversion/simlink library discovery, etc are. And even if they did, Dockerfiles give you an easy way to do something you couldn't easily do before: multi-stage builds and a highly robust way to say "drop the parts of my transitive hull that were only relevant at build time". Dockerfile semantics kind of descend naturally from needing the behavior of stages, parallelization, cache keying for image layers, semantic layering (FROM), and making simple use-cases relatively at-a-glance/contained.
Q: Why not have container builds and runs be invoking-user-specific by default? A: As the adjacent commenter pointed out, this would be an overlay on fundamentally rootful behavior. It would also remove a lot of "works on my machine, works anywhere", since different user accounts running/building containers could be restricted in different ways on different computers, so my "docker build" and your "docker build" might both work, but do different things or fail in different ways. Note that this isn't the same as saying "rootful dockerd is good"; it's not (usually)! That's why rootless docker (https://docs.docker.com/engine/security/rootless/) is recommended in many environments. But even in rootless, the run-user of the container and build are still uniform with the rootful behavior (albeit they require more participation from the daemon-owning user account WRT UID mapping/user namespaces/privileged actions), which I think is a good thing. Rootless docker, last I used it, behaved sensibly re: vs. user container images (kept per-user) and globally-cached layers (removable by anyone), but that may have been a specific configuration where I was using it.
Q: Why aren't docker images user-browsable files? A: I can't say for sure; I didn't build Docker, but I hear the guy who did is in the neighborhood :). I suspect it's a combination of caching (since build stanzas/Dockerfile lines are intended to be proxies for the content address of a layer, there's nothing to gain by making layer caches per-user), implementation convenience, and an influence from the Golang platform wherein (after the age of vendoring wound down) the developer tooling has many convenience layers that de-emphasize user visibility of build/download artifacts a la "go run". Docker goes further, though. And yes, I know that you can get at OCI images/layers as files. But most people don't.