squid-deb-proxy can do most of this - it fetches and locally caches lists and packages. Clients install squid-deb-proxy-client which uses multicast-DNS to discover the local proxy.
Packages are fetched and cached by the proxy first time they're requested and thereafter served locally (subject to lifetime/space etc. squid configuration).
There are ways to pre-populate the cache but you're always going to have situations where package updates have to pull from remote archives.
apt-mirror is designed for a similar purpose. It allows creating a sub-set or entire mirror of one or more upstream archives.
Clients can then be pointed at the local mirror which could be on the same host or LAN.
Generally, 'packages' on github will be source-code only so you'd need a quite complex build server, or at least per-package specific links in your local cache rules for pre-built binaries.
https://blog.thelifeofkenneth.com/2018/01/off-grid-raspbian-...
One solution is to do it once only: grab the base image, install all the packages you need, and then make an image of it so you don't start from scratch again next time.
No.
What I want is something closer to Kiwix already on an SD card so even if everything is down, I can read about something I've never been interested in before. But for software.
Will those who receive this bother to connect it to networks? Should I even enable network/GUI for those who want to tinker easily? How should the password be handled?
I'm considering two extremes:
1. A raspbian lite headless system with all unnecessary peripherals disabled and no network features.
2. A full installation, GUI and all with HDMI and network enabled so it's easy for people to play with it if they want.
I'm leaning towards 2 because I think it would be nice to have a clock that auto updates leap seconds and the tzdata db. If I go for 1, that opens up the possibility of using a raspi zero (non-W) for a real BOM and power savings.
As for the password: I'm thinking of having something short and simple other than "raspberry" that the user can/should change. This seems like standard practice with many enterprise systems.
Easily arguable that RPi isn't made with such use cases in mind. I'm all in favor of removing unnecessary bloat if only a small percentage of user base is ever going use it, especially when the software is readily available.
>If those repos are inaccessable for any reason, I have a bunch of hardware that's very hard to do anything useful with.
How so? They are just Linux boxes, you can just download the source code and compile the binaries you need. Pre-built packages are not necessary for functional OS.
What about dependencies? If you have the internet access required to download the source code I'd say you'd be better off just using the repos.
a) define your entire system state and dependencies in a single, declarative file
b) prebake an image based on this file
It's all done and ready, available now, no hacking around required.
But a bigger question for the ergonomics of NixOS: Are NixOS and Nix prebuilding for ARM now?
None of them provide native, single-source-of-truth declarative configuration that is easy to reason about, pure, and guaranteed to deliver sane results every time (vs. something managed via a classic CM system). Oh, also one that it symmetrical to the way the distribution itself is built and managed.
> But a bigger question for the ergonomics of NixOS: Are NixOS and Nix prebuilding for ARM now?
Yes. https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi
Ports to other ARM devices are also very easy.
Firstly: the Nix language isn't pure. And neither are some base Nix library functions.
Secondly: You may find it easy to reason about. Many of us have not had that experience. Trying to do work as a developer, I felt it was absolutely miserable the instant I needed to depend on a new package or a new runtime. Every language had slightly different conventions and rules. You had to relearn how any specific package worked to integrate it with another, because there often wasn't sanity. And if you DID need to somehow interface with something outside of Nix (say, a vendor binary not in Nix) you had to use an unreliable environment hack.
And of course, the tutorials and docs didn't actually cover the majority of concerns on how to add new stuff folks will inevitably have, except for a trivial C executable.
In one case, after spending a week working out how to add a package to enable a Haskell binding to said package correctly, I submitted package updates that took MONTHS to propagate into the main repo, so I had to start pushing my fork of the nix repo from machine to machine via github on my own to manage multiple machines. It was pretty ridiculous and I regretted my choices.
I like the Nix philosophy. I respect a lot of the people on the project. But I am not a fan of the "it is all fine and well-baked and I'm sure you can use it too" approach a lot of Nix proponents decide to take.
You could absolutely arrive at a solid installable image for ANY major Linux distro.
> Yes. https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi
That's great! I'll have to try it again there if it has packages I want for SDR work that are not comically ancient.
The issue with NixOS for me was probably quality control. When I did an update and instead of fetching the files from the binary cache it starts to build stuff on its own, I could be reasonably sure that some build error happens. Coupled with sparse documentation, I was very often at a loss at how to fix it and just had to remove the package for a while and try again a couple of days later.
Another point is NixOSes path mangling... Do we really need that? Can we not try to use namespaces and OverlayFS etc. to let each process assume it has a normal FHS file-hierachy while in real its 'root' is cobbled together from multiple package installation directories? Instead of patching the paths of every package, letting the kernel do the path calculations seems to be less intrusive.
So in practice updating a Gentoo system is more reliable than updating NixOS.
(B) is probably accomplished with any OS.
As to caching, I tried setting one up and found out a few things.
I originally tried apt-cacher-ng:
https://www.unix-ag.uni-kl.de/~bloch/acng/
but had trouble (can't remember exactly what) and muddled through polipo instad.
I found out:
- it was a lot easier and faster than copying cached packages around
- when doing "apt-get update; apt-get upgrade" the cache really sped up multi-machine updates. The first machine was slow, the rest were fast.
- By far, the majority of cache traffic was for updates to base stuff. foo-1.2, foo-1.2.1 foo-1.2.2, etc.
- second was prerequisites for packages I needed.
- the few specialized packages I used were not updated as frequently.
Want to have an offline copy of software on Windows? Go download all of the installers from their respective sites. Want it to constantly update - that's gonna be about 100 lines of Python code. And better make sure they're the "offline" installers, not those idiotic stubs that just dl the latest version for a server. Also don't forget about all of the 20 different VC++ runtimes that are sometimes not packaged in the installer.
I have a feeling it would be even harder on macOS.
(all of this is of course ignoring the legal issues of distributing those files - which is generally not an issue on Linux)
All software is most definitely not in one repo. If it were, people wouldn't have to use PPAs or compile from source and deal with all the problems those things cause.
> better make sure they're the "offline" installers, not those idiotic stubs that just dl the latest version for a server
Oh, you mean the ones that aren't just acting like apt.
> I have a feeling it would be even harder on macOS.
Depends, but MacOS software that uses Application Bundles should be fine. Even a lot of Windows software works fine if you just copy the installer contents to a directory. Self-contained application directories (or single files) are an old as dirt concept that Linux communities never got behind, preferring instead to use overly complicated schemes like package management that come with a bunch of their own problems. So many, in fact, that people are now often distributing applications in Docker images.
The way I see it there are 2 different use cases here: 1. You want a few individual apps kept safe on a USB stick in case you need them offline - installers on Win, AppImage on Linux 2. You want a constantly updated cache of the programs you use in case of the apocalypse - pain in the ass on Win, trivial on most Linux
Windows only seems superior here because no. 2 is barely possible, so everyone has gotten good at no. 1.
Oh, if only there were more than a handful of applications on Linux distributed as AppImage, and any support given to them at all by file browsers, life would be so much simpler. Linux community hates desktop so much they completely ignored an embedded ELF icon standard and have consistently poo-pooed every non-repo way of dealing with applications ever.
> You want a constantly updated cache of the programs you use in case of the apocalypse
Why would I want that? So I could discover that something I rely on was broken by a recent update only after I have no ability to do anything about it? Well, lets assume so. On Windows this is relatively simple, just keep a copy of Program Files and most applications will still work fine. It's not ideal, but I'll take it over package managers not even being able to install an application to a different disk.
In some mythical parallel dimension where the Windows system directory, user profile directories and the registry don't exist, sure.
User profile and registry are for settings. You can easily keep backups of those as well, but it isn't really necessary unless you have highly tweaked configurations or something.
Have you not touched Windows since 1995?
Which OS handles this better, in your opinion?
I've built most of the base repository of Arch Linux on my RPi4 without cross compiling. I'm buying a few more so that I can go 'full Gentoo' and pretty much rebuild the world.
Have a recording of a container booting from a rebuilt userland: https://asciinema.org/a/283303
It might feel like the ARM packages are somehow special but really only the blobby bits like the RPi firmware is. Everything else is just your bog standard armv7/aarch64 ELF.
So yeah, back up a GCC binary or bootstrap, I guess? I can probably email you a tmux binary in a pinch, plus I have a full local mirror of the repo for the apocalypse? :P
They help you maintain a stable software stack over time. And generate a fully contained software image that can be flashed without needing the internet afterwards. (All downloads happen during compilation)
I'm pretty severely on the noob end of the scale when it comes to software, so I don't know if maintaining a new distro for myself would make sense, since all the tutorials I'm following assume I have Raspbian and all its built-ins available to me.
But maybe in the post-apocalypse, some hero with a prebuilt Rasp-yocto will rescue all our useless boards.
Personally I find it almost magical that I can install and update almost any software I need with a short command.
Imagine that you can build your rootfs is a docker container and export it to IMG
If those repos are inaccessable for any reason, I have a bunch of hardware that's very hard to do anything useful with.
I know there are such things as apt-caches and squid caches and stuff, but I could really use thing that goes through every apt-get I've ever done and the top 50,000 packages on github and stuffs 'em all onto an SD card and shows me how to use them from my commandline.
OP mentions this as a future direction for the project, but I think it's one of the most important.