News for hackers. Stuff that matters.
Re:Why SQLite Does Not Use Git by bane on Saturday July 22, 2023 @09:50PM (#36831832)
I have no idea why Richard focuses on such things but:

I'm old enough to remember when developers spent time making sure they could plow all of their build assets into a single binary distributable. It often had kind of a zest to it and when you dealt with software had directories full of stuff it looked both "corporate" and "sloppy".

I've never quite gotten over the feeling that the piles of dynamically linked libraries hasn't helped things. I know objectively that there's a notion that you can update libraries instead of applications, but it feels like it makes updating applications fragile and you inevitably end up in some kind of dependency hell. "But breaking insecure applications is a feature!" I mean, okay, I guess. But I still need something to work and don't want to spend all day fixing broken layers of dependent stuff. If I have to do that, I may as well just update a single binary.

Go seems to have come back around to this style of thinking, and in a sense container images and JAR files are often trying to replicate what it's like to just download a binary, chmod +x it, and execute it.

Re:Why SQLite Does Not Use Git by beebmam on Saturday July 22, 2023 @10:01PM (#36831900)
If you're using TLS/crypto in an app, it's scary to statically bundle those libraries.
Re:Why SQLite Does Not Use Git by jjnoakes on Saturday July 22, 2023 @10:24PM (#36832017)
It depends. Sometimes you have no choice but to deploy to an old machine with back-leveled tls libraries, and static linking can save the day.

Of course that tends to be the exception (hopefully).

Re:Why SQLite Does Not Use Git by andersa on Saturday July 22, 2023 @11:50PM (#36832399)
The situation on windows is fascinating: everyone links these libraries dynamically, and yet, there are about two hundred of them on my system, every application using its own uniquely outdated version.
Re:Why SQLite Does Not Use Git by kibwen on Sunday July 23, 2023 @12:12AM (#36832506)
On a closed-source system, sure. On a system where you have the source, you have the option to rebuild your binaries.
Re:Why SQLite Does Not Use Git by bborud on Sunday July 23, 2023 @07:00AM (#36834245)
In my practical experience the set of things that can go wrong if you link apps dynamically is much larger than the problems that arise when they are statically linked.

For one, it is more complicate to keep track of which of the many shared libraries on a typical system are used by which application. It is common that the same library occurs multiple times in different versions, built by different people/organizations and residing in different directories.

Quick, without looking: which TLS library do your network exposed subsystems use, which directories are they in and where did you install them from. When you do go to look: did you find what you expected?

Have a look at all the other shared libraries on your system. Do you know which binaries use them? Do you know which versions of which libraries work with which binaries? Do you trust the information your package manager has about version requirements? Does it even have that information?

Then there's the problem of what happens when you upgrade. The servers you run might have a rigorous battery of tests. But now you are running them with libraries they were not tested against. Sure, most of the time it'll work. But you don't know that. And you have no way of knowing that without downloading, building and running the tests. Or have someone else do that.

I've been in the situation where someone inadvertently updated a library in production and everything came crashing down. Not only did it take down the site, but it took a while to figure out what happened. Both because the person who did it wasn't aware of what they'd done. And the problem didn't manifest itself in a way that made the root cause obvious.

The clearest risk with statically linked binaries is if they are not updated when there is, for instance a security problem. But in practice I find that easier to deal with since I know what I'm running, and for anything important, I'm usually aware of what version it is or when I last checked for updates/problems.

Re:Why SQLite Does Not Use Git by nwallin on Sunday July 23, 2023 @10:52AM (#36835876)
> For one, it is more complicate to keep track of which of the many shared libraries on a typical system are used by which application. It is common that the same library occurs multiple times in different versions, built by different people/organizations and residing in different directories.

That's not common at all, man. I strongly recommend you don't do that.

> Quick, without looking: which TLS library do your network exposed subsystems use, which directories are they in and where did you install them from.

Openssl 3.x.y. It's /usr/lib64/openssl.so or similar. They are installed from my distro's repository.

> When you do go to look: did you find what you expected?

Yes. Openssl 3.1.1-r2. The OpenSSL binaries are actually named /usr/lib64/libssl.so and /usr/lib64/libcrypto.so. Upstream version is 3.1.2. There have been two low priority CVEs since 3.1.1 (never change openssl...) and my distro has backported the fixes for both of them into 3.1.1-r2.

> Do you know which versions of which libraries work with which binaries?

What do you mean "which versions of which libraries"? There's only one version of each library. If the package manager needs to keep an old version of a library around, it gives a loud warning about it so I can either fix the problem or ignore it at my own peril.

Those two .so files (libssl.so and libcrypto.so) as used by postfix, dovecot, and nginx. They are also linked by opendkim, spamassassin and cyrus-sasl, but those don't have open ports on the internet, so they don't really count. OpenSSH can optionally link to openssl; as it happens, my openssh does not link against a crypto library, openssl or otherwise. It just uses openssh's built in crypto schemes.

> Do you trust the information your package manager has about version requirements?

Yes.

> Does it even have that information?

... wat..? Of course it does?

> I've been in the situation where someone inadvertently updated a library in production and everything came crashing down. Not only did it take down the site, but it took a while to figure out what happened. Both because the person who did it wasn't aware of what they'd done. And the problem didn't manifest itself in a way that made the root cause obvious.

I've been in the situation where a security guard at my last job inadvertently discharged his service revolver into a Windows machine, and it crashed. That doesn't mean I stopped using Windows. (I mean, I did stop using Windows...)

That's genuinely just not a problem that I've had. Not since 2004 and all the C++ programs on my computer broke because I force upgraded from GCC-3.3 to GCC-3.4 and the ABI changed. Or that time in 2009 where I installed a 0.x version of Pulseaudio on my gaming machine. Or that time I replaced OpenSSL with LibreSSL on my personal computer. If your server takes a shit because somebody was fucking around doing stupid shit on prod, and you do root cause analysis and come up with a reason that it broke other than, "employee was fucking around and doing stupid shit on prod" and the recommendation is something other than "don't fuck around and do stupid shit on prod" I don't know what to tell you. Dynamic linking isn't going to stop a sufficiently determined idiot from bringing down your server. Neither will static linking.

Re:Why SQLite Does Not Use Git by bborud on Sunday July 23, 2023 @12:10PM (#36836673)
> What do you mean "which versions of which libraries"?

If you upgrade a shared library to fix a problem, how do you know that the application has been tested against the fixed version?

And no, your package manager won't know.

Congratulations on a) not having multiple installs of shared libraries on your system and b) for knowing which version you have. Knowing this isn't very common.

Re:Why SQLite Does Not Use Git by rstuart4133 on Sunday July 23, 2023 @09:12PM (#36841855)
> If you upgrade a shared library to fix a problem, how do you know that the application has been tested against the fixed version?

Distro's like Debian solve that problem by not upgrading. The only things deemed worthy of "fixing" are security issues, and they are fixed by backporting the fix (only) to the existing shared library. Thus no API's (of any sort - even unofficial ones like screen scraping) are upgraded or changed, so no testing is necessary.

And thus:

> And no, your package manager won't know.

It doesn't have to know, because the package manager can assume all releases for Debian stable are backward compatible with all the packages in that release.

A lot of the noise you see on HN comes from people using distro's on their desktops. To them a distro is a collection of pre-packaged software with all the latest shinies, which they upgrade regularly. But Linux's desktop usage is 3%, whereas it server usage is claimed to be over 95% (which eclipses Windows Desktop share). Consequently distros are largely shaped not by the noisy desktop users, but by the requirements of sysadmin's. They need a platform that is guaranteed both stable and secure for years. To keep it stable, they must solve the problem you describe, and for the most part they have.

Re:Why SQLite Does Not Use Git by bborud on Monday July 24, 2023 @11:53AM (#36849955)
> because the package manager can assume [...]

Right.

Re:Why SQLite Does Not Use Git by toast0 on Sunday July 23, 2023 @11:22AM (#36836187)
If you're linking to OpenSSL, it's scary to have that upgraded from under you. Maybe it got better in the 3 series, but I seem to recall pretty much all the important 1.0.1? releases would be something you'd need to mitigate a bit vulnerability, but would also have api changes that would break your application if you were trying to do newish things. Sometimes justified, but still a pita.
Re:Why SQLite Does Not Use Git by no_wizard on Saturday July 22, 2023 @10:05PM (#36831913)
Monolithic builds are great if you have no control over the deployed environment (IE desktop apps sans OS supplied libs). They’re worse if you do control the environment and how the upgrade paths get followed
Re:Why SQLite Does Not Use Git by bane on Saturday July 22, 2023 @10:19PM (#36831993)
Doesn't it seem that more and more people are just given access to some managed environment they have little control over anyway?

I feel like sometimes the dependency on dynamically linked stuff is akin to "well transistor radios are great if you don't care about soldering on fresh vacuum tubes like a real radio person would."

Re:Why SQLite Does Not Use Git by colordrops on Saturday July 22, 2023 @10:33PM (#36832055)
You'd love NixOS. Gives you the flexibility of dynamic libraries with the isolation and full dependency bundling per app and less janky than snap or flatpak.
Re:Why SQLite Does Not Use Git by osigurdson on Saturday July 22, 2023 @10:00PM (#36831891)
Well, it’s either a single executable in a container or a bunch of dlls/sos.
Re:Why SQLite Does Not Use Git by resonious on Saturday July 22, 2023 @10:49PM (#36832140)
Directories full of stuff is similar to websites with URL paths like "/site.php?page_id=18231238". Or even better when subdomains get involved and it looks like "secure3.action.domain.com/admin.php?page=123424". It technically works but is a bit ugly.

Also another web analogy might be dynamic linking being similar to microservices. People want to build and ship smaller components that can be swapped out independently. It works but does seem to make updating and testing fragile. You can test heavily at the boundaries but there's still kind of an "air gap" between the main app and the lib/microservice. If you want to be really sure there's no breakage, you have to test the whole thing, at which point you might as well just ship a monolith.

Re:Why SQLite Does Not Use Git by saagarjha on Saturday July 22, 2023 @10:51PM (#36832148)
Dynamic libraries make things nice to swap out for hacking or testing, though.
Re:Why SQLite Does Not Use Git by resonious on Saturday July 22, 2023 @11:10PM (#36832258)
Definitely true for hacking, but if we're talking about testing then it's really not so different from adding `if test_mode then ...` to your code.
Re:Why SQLite Does Not Use Git by csydas on Sunday July 23, 2023 @06:11AM (#36833974)
>Directories full of stuff is similar to websites with URL paths like "/site.php?page_id=18231238". Or even better when subdomains get involved and it looks like "secure3.action.domain.com/admin.php?page=123424". It technically works but is a bit ugly.

OOC, why does this stand out for you? Just to explain my curiosity, I've worked on Mac since I was a kid starting with System 6 and then going to OS X when it came out, so Apple's "your program is all in that file" just kind of made sense to me and it was really convenient to just drag a file to the trash and the app is _mostly_ gone, minus a few .plist and other config files in ~/Library.

But I _like_ the old forums and sites that still show the stuff like page_id=N; for the boards and forums I go to, it's very useful to just jump around long topics or you can play with it on your shitposting.

Plus most modern browsers truncate or hide the full URL anyways; I dislike this feature personally, but at least Safari's concise tabs are a good balance for someone like me.

Re:Why SQLite Does Not Use Git by resonious on Sunday July 23, 2023 @10:17AM (#36835510)
Fair enough, for message boards it's fine. I think I was mostly just thinking about old/sloppy WordPress sites where you might click on "about us" and it takes you to ?page_id=1234. Feels like a lack of attention to detail compared to /about-us. Similarly, a binary surrounded by a bunch of folders and dlls feels like a lack of attention to detail (and thus kind of "corporate" as the previous poster mentioned).
Re:Why SQLite Does Not Use Git by throwawaymobule on Monday July 24, 2023 @06:13PM (#36854961)
Bit of a tangent, but those .app 'files' are directories. I wish other OSs supported something like that.

Those APE executables (e.g. redbean) are the closest I've seen that work widely.

Re:Why SQLite Does Not Use Git by MuffinCups on Saturday July 22, 2023 @11:48PM (#36832395)
A dynamically linked library need only have one image of itself in memory.

If you are running a process that, for example, forks 128 of itself, do you want every library it uses to have a separate copy of that library in memory?

That's probably the biggest benefit. But it also speeds up load time if your executable doesn't have to load a huge memory image when it starts up, but can link to an already in-memory image of its library(s).

The only real downside is exporting your executable into another environment where the various dynamic library versions might cause a problem. For that we have Docker these days. Just ship the entire package.

Re:Why SQLite Does Not Use Git by anotherhue on Sunday July 23, 2023 @12:59AM (#36832663)
Wouldn't KSM help with that? If the security concerns aren't a factor.
Re:Why SQLite Does Not Use Git by never_inline on Sunday July 23, 2023 @01:03AM (#36832682)
> If you are running a process that, for example, forks 128 of itself, do you want every library it uses to have a separate copy of that library in memory?

Fork uses CoW, right?

Re:Why SQLite Does Not Use Git by Expurple on Sunday July 23, 2023 @03:15AM (#36833221)
> it also speeds up load time if your executable doesn't have to load a huge memory image when it starts up

I'm not sure about Windows and Mac, but Linux uses "demand paging" and only loads the used pages of the executable as needed. It doesn't load the entire executable on startup

Re:Why SQLite Does Not Use Git by kaba0 on Sunday July 23, 2023 @03:30AM (#36833294)
> For that we have Docker these days

Or just a proper package manager, like nix.

Re:Why SQLite Does Not Use Git by Terr_ on Sunday July 23, 2023 @01:51AM (#36832870)
Somehow this makes me think of games Back In The Day where you could simply replace your crosshair by editing a bitmap file, versus now where everything's so much more locked-down behind proprietary container formats and baked-in checksums, etc.
Re:Why SQLite Does Not Use Git by Shorel on Sunday July 23, 2023 @09:04AM (#36834939)
I agree with you so much.

Dynamic linking is the bane of backwards compatibility.

Now everything is containers, appimages, flatpacks, docker images and so on, and all they do is pack all the libraries a binary may need, in a more wasteful and inefficient format than static linking.

In that sense, we truly have the worst of both worlds.