julian-klode

> though I hope existing well tested rust libraries are used rather than NIHing them and introducing new bugs

I think that's much more likely to introduce bugs.

Think of it that way, a lot of the Rust libraries are rewriting existing copyleft libraries in permissive licenses, so they cannot look at the original code, dooming them to repeat the mistakes that were made in the original code and having to fix them all over again on their own (as both go from "oh this is simple" to "oh another corner case").

I just want to translate code 1:1 to Rust, reusing my existing knowledge, design decisions, and tests. It should behave _exactly_ the same as before, just memory safe.

Like the last one was/is the inability to comprehend safety of large buffers on ppc64el because the stack clash protector code generated by gcc isn't understood. The one before that was more problems of that sort on armhf where it also didn't understand the clash protector - in more cases.

It's quite surprising and it takes days to weeks to debug each of these, going down to the assembler level and verifying that by hand.

As far as I understand, Fil-C changes the ABI of the system, therefore it requires a new architecture in Debian terminology, e.g. amd64fil. And then you'd need to use multi-arch to pull in amd64fil binaries where that works.

We'll have to see how this plays out but it's not super plug and play.

It's certainly what we aim for in APT. We do have an overwrite of course, since we need to copy uninitiated data around: The cache file is allocated as a whole and written at the end, but not all parts of it are used, but it triggers stuff.

Don't want to introduce complex code to only copy the parts that are actually reachable would be silly and introduce bugs.

But keep in mind valgrind is super buggy and we spend quite a bunch of time working around valgrind false positives (outside of amd64)

It's not like I'm in a hurry to switch to Rust and will spend full steam on it. It's amongst the lowest priority items.

A lot of the Rust rewrites suffer a crucial issue: they want a different license than what they are rewriting and hence rewrite from scratch because they can't look at the code.

But here we're saying: Hey we have this crucial code, there may be bugs hidden in it (segfaults in it are a recurring source of joy), and we'll copy that code over from .cc to .rs and whack it as little as possible so it compiles there.

The problem is much more there on the configuration parser for example which does in a sense desparately need a clean rewrite, as it's way too sloppy, and it's making it hard to integrate.

In an optimal world I'd add annotations to my C++ code and have a tool that does the transliteration to Rust at the end; like when the Go compiler got translated from C to Go. It was glorious.

/me hides in shame

It's insanely complex, particularly you want _verified_ crypto. Last year (or two years ago?) I had to fix a tiny typo in OpenSSL's ARM assembly for example, it was breaking APT and Postgres left and right, but only got triggered on AWS :D

> > since it only supports amd64 at this time and is maintained by a single genius. > That's easily fixable.

as easily as fixing Rust to work on the remaining 4 architectures?

> > It also doesn't help you to attract new contributors. > I don't understand this point.

C++ doesn't attract a lot of developers, Rust attracts many more. I want more community, particularly _young_ community. I don't wanna work on this alone all the time :D

Do keep in mind that a lot of the people involved in these sorts of things are neurodiverse in some ways, and may have significant trouble dealing with change.

As teh64 helpfully pointed out in https://news.ycombinator.com/item?id=45784445 some hours ago, 4ish years ago my position on this was a total 360 and I'd have had the same reaction to now-me's proposal.

Isn't it also funny that all of these things are done by the same person?

In all seriousness though, let me assure you that I plan to take a very considerate approach to Rust in APT. A significant benefit of doing Rust in APT rather than rewriting APT from scratch in Rust means that we can avoid redoing all our past mistakes because we can look at our own code and translate it directly.

Alpha and PA Risc where the predecessors of Itanium fwiw if you want to feel old in an easy to understand way.

Keep in mind APT is using the GNU dialect of C++, particularly C++ 17, shortly C++ 23. And it always exploits the latest C++ features while at the same time still having workarounds in the code for a lack of namespaces in C++ compilers prior to the C++ standardization...

We have replaced GnuPG by Sequoia in the previous Debian release.

Actual cryptography code, the best path is formally verified implementations of the crypto algorithms; with parsers for wrapper formats like OpenPGP or PKCS#7 implemented in a memory safe language.

You don't want the core cryptography implemented in Rust for Rust's sake when there's a formally verified Assembler version next to it. Formally verified _always_ beats anything else.

There is no Perl code in APT itself, nor in dpkg on the installing side; Perl is used by dpkg-dev, i.e. when building packages.

If anyone sees that horrible mess of hacks around pre-STL C++'s lacks of namespace in combination with latest C++ features as part of the C++ community I'd be very surprised :D

If APT were a hardcore C++ project surely we'd have like adopted namespaces everywhere by now.

I wish, but I get new security bugs in those components like every year or so, not all are tracked with security updates to be fair, some we say it's your own fault if you use the library to parse untrusted code.

After all the library wasn't designed around safety, we assumed the .debs you pass to it are trusted in some way - because you publish them to your repository or you are about to install them so they have root maintainer scripts anyway.

But as stuff like hosting sites and PPAs came up, we have operators publishing debs for untrusted users, and hence suddenly there was a security boundary of sorts and these bugs became problematic.

Of course memory safety here is only one concern, if you have say one process publishing repos for multiple users, panics can also cause a denial of service, but it's a step forward from potential code execution exploits.

I anticipate the rewrites to be 1 to 1 as close as possible to avoid introducing bugs, but then adding actual unit tests to them.

I mean they can't in the first place, because ports only have unstable available

Fil-C is amazing but is much more problematic than Rust at this point since it only supports amd64 at this time and is maintained by a single genius.

It also doesn't help you to attract new contributors. With the changes we made over in Ubuntu to switch to rust-coreutils and sudo-rs, we have seen an incredible uptake in community contributions amongst other things, and it's very interesting to me to try to push APT more into the community space.

At this time, most of the work on APT is spent by me staying awake late, or during weekends and my 2 week Christmas break, the second largest chunk is the work I do during working hours but that's less cool and exciting stuff :D

Adding Rust into APT is one aspect; the other, possibly even more pressing need is rewriting all the APT documentation.

Currently the APT manual pages are split into apt-get and apt-cache and so on, with a summary in apt(8) - we should split them across apt install(8), apt upgrade (8) and so on. At the same time, DocBook XML is not very attractive to contributors and switching to reStructuredText with Sphinx hopefully attracts more people to contribute to it.

It's rather that GnuPG is ill-regarded for its code immaturity tbh. You don't even need to read the code base, just try to use it in a script:

It exits 0 when the verification failed, it exits 1 when it passed, and you have to ignore it all and parse the output of the status fd to find the truth.

It provides options to enforce various algorithmic constraints but they only work in some modes and are silently ignored in others.

Let me assure you it was my own decision. The final paragraph is my paraphrasing of a fellow Debian developer and CTTE member's stated opinion.

Ports are not part of Debian and particularly don't release with Debian, they only ship unstable.

Yes that's true and there's synergies but keep in mind I also have a personal mind

We're really talking about alpha, hppa, m68k and sh4

Yes sure

apt (3.1.7) unstable; urgency=medium . [ Julian Andres Klode ] * test-history: Adjust for as-installed testing . [ Simon Johnsson ] * Add history undo, redo, and rollback features

This is a very contentious topic, you can see my initial take on this in

https://blog.jak-linux.org/2023/10/10/a-case-for-different-u...

The problem is that's not really true. i.e. consider you get postfixed pulled in by postfix | mail-transport-agent and the new distro release changes it to exim4 | mail-transport-agent as we picked a new default MTA.

Does this mean we should, by default, replace your MTA and require you to have to configure a different one?

You will be able to do this (pick the same automatically installed packages given the same set of manually installed ones; installing missing Recommends, etc, by using --fix-policy and some more options perhaps).

As for configuration this is worse, files managed by ucf have 3-way merging but it's not automatic and most configuration isn't managed by ucf but by dpkg.

apt-mark showmanual shows you all manually installed packages and apt-mark auto marks stuff as automatically installed.

The list of manually installed packages is effectively the "world".

The new solver takes inspiration from apk in that it marks the world/manually installed packages the root of the dependency problem (hence why they can't be removed by the solver itself, only manually).

So hibernate is somewhat unreliable and prone to data loss, image you hibernate after having installed a new kernel, so the decision was made to disable it due to that IIRC, independent of secure boot.

With secure boot and lockdown, hibernate is no longer possible on an alternative reason: We need to ensure that the kernel memory has not been tampered with. If you hibernate, you could then go and modify the memory in the swap and bypass the lock down security guarantees.

To address that you'd need to authenticate the swap using the TPM somehow, but I don't know enough about TPMs to know if that's feasible. Usually people would seal some crypto key against the TPM but here it's somewhat the opposite way around.

They even should all be supporting efifb?