News for hackers. Stuff that matters.
Re:Reverse engineering GitHub Actions cache to make it fast by junon on Wednesday July 23, 2025 @10:35AM (#44659693)
> iptables was already doing heavy lifting for other subsystems inside our environment, and with each VM adding or removing its own set of rules, things got messy fast, and extremely flakey

We saw the same thing at Vercel. Back when we were still doing docker-as-a-service we used k8s for both internal services as well as user deployments. The latter lead to master deadlocks and all sorts of SRE nightmares (literally).

So I was tasked to write a service scheduler from scratch that replaced k8s. When it got to the manhandling of IP address allocations, deep into the rabbit hole, we had already written our own redis-backed DHCP implementation and needed to insert those IPs into the firewall tables ourselves, since Docker couldn't really do much at all concurrently.

Iptables was VERY fragile. Aside from the fact it didn't even have a stable programmatic interface, it was also a race condition nightmare, rules were strictly ordered, had no composition or destruction-free system (name spacing, layering, etc), and was just all around the worst tool for the job.

Unfortunately not much else existed at the time, and given that we didn't have time to spend on implementing our own kernel modules for this system, and that Docker itself had a slew of ridiculous behavior, we ended up scratching the project.

Learned a lot though! We were almost done, until we weren't :)

Re:Reverse engineering GitHub Actions cache to make it fast by tsaifu on Wednesday July 23, 2025 @10:42AM (#44659772)
yeah our findings were similar. the issues we saw with iptables rules, especially at scale with ephemeral workloads, was starting to cause us a lot of operational toil. nftables ftw
Re:Reverse engineering GitHub Actions cache to make it fast by tekla on Wednesday July 23, 2025 @10:54AM (#44659893)
I've had this problem.

We ended up using Docker Swarm. Painless afterward

Re:Reverse engineering GitHub Actions cache to make it fast by theideaofcoffee on Wednesday July 23, 2025 @11:14AM (#44660155)
Programmatic interaction with iptables ... isn't difficult and has been a relative constant in all of the systems I've managed for nigh on two decades now. Hint: strict ordering of rules for a firewall is a feature, not a bug! Sure it's not as nice as pf(4), but it feels like nftables carries on some of the DNA that made pf nice to work with. If you're going that low-level you should probably just make use of the base netfilter primitives by the kernel and dispense with the userland, these having been available since -1999-. Better yet all of this could now live as a handful of bpf programs. But what do I know, I've only been doing this for more than half my life.

Maybe it's for the best that the project was scrapped, considering the bar for technical competence at Vercel has been embarrassingly low across my interactions, I'm not surprised that iptables and networking as a whole has been a challenge.

Re:Reverse engineering GitHub Actions cache to make it fast by junon on Wednesday July 23, 2025 @12:27PM (#44661016)
What a delightfully arrogant comment :) This was about ten years ago. At the time, given the constraints, iptables did not have a public, stable programmatic interface, no. Perhaps you used other firewall systems/frameworks. We had to interop with Docker. I don't remember specifics; I do remember going very deep down that rabbit hole.

For what it's worth, the development team, at least in the early ZEIT days, were the most competent engineers I've ever worked with on a team.

Re:Reverse engineering GitHub Actions cache to make it fast by dang on Wednesday July 23, 2025 @01:37PM (#44661836)
Please don't cross into being a jerk on HN. You may not feel you owe the person you're speaking with better, but you owe this community better if you're participating in it.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

Re:Reverse engineering GitHub Actions cache to make it fast by immibis on Wednesday July 23, 2025 @11:00AM (#44659975)
I think iptables compiles BPF filters; you could write your own thing to compile BPF filters. In general, the whole Linux userspace interface (with few exceptions) is considered stable; if you go below any given userspace tool, you're likely to find a more stable, but less well documented, kernel interface. Since it's all OSS, you can even use iptables itself as a starting point to build your own thing.
Re:Reverse engineering GitHub Actions cache to make it fast by formerly_proven on Wednesday July 23, 2025 @11:14AM (#44660146)
Nowadays you would use nftables, which like most new-ish kernel infra uses netlink as an API, and supports at least atomic updates of multiple rules. That's not to say there's documentation for that; there isn't.
Re:Reverse engineering GitHub Actions cache to make it fast by cameronh90 on Wednesday July 23, 2025 @03:32PM (#44663057)
I spent a decade and a bit away from Linux programming and have recently come back to it, and I'm absolutely blown away at how poor the documentation has become.

Back in the day, one of the best things about Linux was actually how good the docs were. Comprehensive man pages, stable POSIX standards, projects and APIs that have been used since 1970 so every little quirk has been documented inside out.

Now it seems like the entire OS has been rewritten by freedesktop and if I'm lucky I might find some two year out of date information on the ArchLinux wiki. If I'm even luckier, that behaviour won't have been completely broken by a commit from @poettering in a minor point release.

I actually think a lot of the new stuff is really fantastic once I reverse engineer it enough to understand what it's doing. I will defend to the death that systemd is, in principle, a lot better than the adhoc mountain of distro-specific shell scripts it replaces. Pulseaudio does a lot of important things that weren't possible before, etc. But honestly it feels like nobody wants to write any docs because it's changing too frequently, but then everything just constantly breaks because it turns out changing complex systems rapidly without any documentation leads to weird bugs that nobody understands.