Yeah. I just don’t think “runc” has ever been a bottleneck in any systems I’ve worked with.

It’s been years since I’ve been able to get excited about rewriting systems from a memory safe language into a memory unsafe language for “performance reasons”. As an industry, we have too much evidence that even the best humans make mistakes. Quality C codebases still consistently run into CVEs that a memory safe language would have prevented.

Rust exists, if performance is so critical here and if Go were somehow at fault... but it sounds like most of the performance difference is due to a different architecture, so rewriting runc in Go probably could have had the same outcome.

I wish I could be excited about crun... I’m sure the author has put a lot of effort into it. Instead, I’m more excited about things like Firecracker or gVisor that enable people to further isolate containers, even though such a thing naturally comes with some performance impact.

“Any of the systems you’ve worked with”

I suspect it has been a bottleneck in some of the systems redhat has worked with, or they would have no benefit in writing it. As the authors of openshift I’m sure they’ve seen some pretty crazy use cases. Just because it isn’t your use case doesn’t make it invalid.

> I wish I could be excited about crun... I’m sure the author has put a lot of effort into it. Instead, I’m more excited about things like Firecracker that attempt to enable people to further isolate containers, even though such a thing naturally comes with some performance impact.

gVisor is also very exciting due to how it handles memory allocation and scheduling. Syscalls and I/O are more expensive, though.

Heh, I actually edited gVisor in right before your reply.

I agree completely!

I did some hacking on crun this week to fix a bug, so maybe I can offer a perspective about why it’s written in C.

A huge part of a container runtime is pretty much just issuing Linux syscalls - and often the new ones too, that don’t have any glibc bindings yet and need to be called by number. Plus, very careful control of threading is needed, and it’s not just a case of calling runtime.LockOSThread() in go - some of these namespace related syscalls can only be called if there is a single thread in the process.

It’s _possible_ to do this in go, of course; runc exists after all. But it’s certainly very convenient to be able to flip back and forth between kernel source and the crun code base, to use exactly the actual structs that are defined in the kernel headers, and to generally be using the lingua franca of interfacing with the Linux kernel. It lowers cognitive overhead (eg what is the runtime going to do here?) and lets you focus on making exactly the calls to the kernel that you want.

Go is actually a really poor choice for the container runtime because much of the container setup cannot be done from multithreaded code[0], so it has to be done in C before the go runtime initializes. I do think rust is a better choice for this layer than C because there are still security risks, but getting rid of Go for this layer is a win. I'm not sure why RH chose to rewrite it in C rather than using rust[1].

[0]: https://www.weave.works/blog/linux-namespaces-and-go-don-t-m... [1]: https://github.com/drahnr/railcar

> Go is actually a really poor choice for the container runtime because much of the container setup cannot be done from multithreaded code[0]

This was addressed in 2017/2018 [0], it's no longer a poor choice.

[0]: https://github.com/golang/go/commit/2595fe7fb6f272f9204ca3ef...

While the particular issue of network namespaces and locking the os thread was fixed, there is still c code that must run before the go runtime starts to work around the issue that you cannot do some of the necessary nsenter calls once you have started additional threads. The c code to make runc work is encapsulated in libcontainer/nsenter[0]

[0]: https://github.com/opencontainers/runc/tree/master/libcontai...

this seems like a pretty trivial amount of C, by comparison, and a pretty solved problem now.

This I entirely agree with you on but I’d expect it due to the resources available at the time were more fluent with C than rust. A shame really.

Or they want to be more platform-agnostic than rust and llvm.

Which platforms does Red Hat support Linux containers on that Rust doesn't support? Which platforms is anyone else frequently using Linux containers on that Rust doesn't support? I'm pretty sure the venn diagram here would be a single circle. Rust supports the major architectures.

Portability seems like an unlikely reason here, but I would pick memory safety over portability every single day of the week. Rust is far from the only language that offers memory safety, but it is certainly one of the most compelling ones for projects that would otherwise be written in C.

"runc" is battle tested, written almost entirely in Go, which offers memory safety, and the performance delta under discussion is only about 2x for a part of the system that seemingly only has a measurable performance impact if you're starting containers as quickly as possible.

This just isn't a very compelling use case for C.

Who knows? Who cares?

If I worked for RedHat I would still seek to write as portable as possible just on general principle or for the sake of my own future self.

But I agree it's merely a possible reason and may not be a major one.

It is almost 50% faster for launching containers. That’s considerable, and at scale could be an actual cost savings. If nothing else, efficiency is good.

Also, small c projects can be written properly with discipline. This is a very talented team on a very small and well scoped project. It can be written properly.

> This is a very talented team on a very small and well scoped project. It can be written properly.

Talent does not preclude you from making mistakes

Most projects start out small and well scoped. That doesn’t mean it’s going to last.

There's also the copious amount of static analysis and warnings built into gcc and clang these days, not to mention safer build options. Writing C today is quite a bit better than even 10 years ago.

Better, yes. Safe, no. Even with ASAN and modern C++, vulnerabilities happen all the time.

This is C, not C++. Modern C++ is kind of a mess with how much stuff they’ve added to the language. This is not relevant.

> Modern C++ is kind of a mess with how much stuff they’ve added to the language. This is not relevant.

Yeah and C++ would be a way better language to write critical system daemon in 2020 than C. Both safer and more productive while keeping the exact same portability and performance as C when necessary.

Most safety issues of C (buffer overflows, use-after-free, stack smash) are not a problem anymore in modern C++.

Yes, writing new userland software in C in 2020(1) is non-sense.

- Use at least C++ if you are conservative.

- Use Rust if you aim for absolute safety.

- Use Go if you accept the performance hit and do not need libraries.

There is zero excuses to C in userland in 2020.

The only excuse is some Red Hat folks seems to practice C++ hating as some kind of religion.

That exactly what give us "beautiful" monstrosities like systemd or pulseaudio with their associated shopping list of CVEs [^1]

Even GCC maintainers switched to C++, by the sake of god, do the same.

----

[^1]: https://www.cvedetails.com/product/38088/Freedesktop-Systemd...

> There is zero excuses to C in userland in 2020.

I wouldn't go quite that far. I'm generally in the C++ camp (rather than C, that is) but there are significant advantages to a compact and stable language, and real disadvantages to a sprawling disaster like C++ that grows still more monstrously complex every few years.

This topic turned up a year ago: https://news.ycombinator.com/item?id=21946060

Your point stands though: well written modern C++ should be much less prone to memory-safety issues than well written modern C.

The Linux kernel still has memory safety issues to this day.

The C++ approach is to offer a feature-rich language so the programmer doesn't have to reinvent common abstractions. The C approach is to offer a minimal and stable language and let the programmer take it from there. It's not obvious a priori which approach should result in fewer memory safety issues. If I had to guess my money would be on C++ being the better choice, as it has smart pointers.

Saying C is easier to secure than Modern C++ is just confusing. The point of a lot of recent C++ features is to make memory safety easier to achieve. I personally don't think it gets anywhere close to safe enough for me to recommend, when Rust is such a competent option that people can choose instead. I would still rather people have the tools of C++ at their disposal instead of having to do everything themselves in C.

Please find me a well-known C code base that doesn't suffer from memory safety CVEs. You're the one making the claim that they exist... I can't prove that such a thing doesn't exist, but I can point to how curl[0][1], sqlite[2][3], the linux kernel[4][5], and any other popular, respected C code base that I can think of suffers from numerous memory safety issues that memory safe languages are built to prevent.

Writing secure software is hard enough without picking a memory unsafe language as the foundation of that software.

"Just find/be a better programmer" isn't the solution. We've tried that for decades with little success. The continued prevalence of memory safety vulnerability in these C code bases shows that existing static analysis tools are insufficient for handling the many ways that things can go wrong in C.

I think you mentioned railcar elsewhere in this thread, and I would find it easier to be interested in a runc alternative that is written in a memory safe language... which railcar is.

[0]: https://curl.se/docs/CVE-2019-5482.html

[1]: https://curl.se/docs/CVE-2019-3823.html

(among numerous others)

[2]: https://www.darkreading.com/attacks-breaches/researchers-sho...

[3]: http://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=SQLite

(the second link has quite a few memory safety issues that are readily apparent, including several from SQLite that affected Chrome)

[4]: https://www.cvedetails.com/cve/CVE-2019-11599/

[5]: https://www.cvedetails.com/cve/CVE-2018-15471/

And yet NASA uses C, not Rust.

I don't think you want to develop your software project like the NASA develops its critical software projects.

Indeed. Pretending it isn’t is equally silly.

External static analyzers also improved a lot.

You can ensure memory-safety in C by enforcing strict rules since the first commit.

Can you share any good examples of C codebases that have enforced these rules and successfully avoided having memory safety problems? Can you share any details on what rules a team could use to reliably avoid having memory safety problems when writing C?

The most security-critical part of container tech lives in Linux kernel and has always been in plain C.

Remember that runc contains quite a few lines of C:

https://github.com/opencontainers/runc/blob/4d4d19ce528ac40c...

https://github.com/opencontainers/runc/blob/4d4d19ce528ac40c...

Is what appears to be less than 2,000 lines including detailed comments etc. "quite a lot"?

This seems like a manageable amount to very carefully maintain and not at all like writing something entirely in C.

To be fair, much of runc is written in C as well. And go is really terrible specifically for the case of runc.

Most security software is written in C. WireGuard, OpenSSH, OpenSSL, OpenVPN, all the Software from OpenBSD folks. Crun is being used without root privilege in most cases so it’s not a huge problem if the runtime is ever escaped.

Contrary to popular belief, memory safe C is possible to write, and with modern linters it’s even easier.

Yes it’s easier to shoot yourself in the foot, but with practice you can learn to miss every time.

It probably isn't worth trying.

djb wrote remotely exploitable C for qmail, applications written by the OpenBSD team have had exploitable memory issues, Microsoft still doesn't get it right, neither do Linux kernel devs.

You say that as if the likes of Microsoft or Linux kernel devs consist exclusively of elite C devs.

I'm very glad languages like Go and Rust exist, but saying you shouldn't write C because you might create memory leaks is kind of like saying you shouldn't write multi-threaded code because you might create race conditions. Yeah, it adds complexity to your code, but it's sometimes worth the overhead it saves. Whether that trade-off is worth it is always up for debate.

Oh, I agree it is a tradeoff. But the parent said "you can learn to miss every time". Can anyone point me to a C language project where the developers consistently miss every time?

It’s possible with rules, experience and guidance.

Granted, I learned to write C 25+ years ago, and have worked as a C programmer for 15 of them, writing mostly embedded software for mobile phones (pre smartphone), airport sorters, but have also written financial software (mostly for parsing NASDAQ feeds), but the point is that most of the software I’ve written has had close to a decade of “runtime”, and while I started out making the same mistakes as everybody else, you learn to put ranges on your heap pointers like strncpy instead of just blindly doing strcpy.

Checking the size of your input vs your allocated memory takes care of a lot of it. As for memory leaks, it’s not exactly hard to free a pointer for every time you malloc one.

People are terrified of C, and yes, Go, Rust, Java, C# makes it harder to make _those_ mistakes, but that doesn’t mean it’s impossible to write good C code.

And it’s not like projects written in Go or Rust are error free. They just struggle with different errors.

As for good C projects, check stuff like postfix, nginx, and yes Linux or FreeBSD.

Are you saying that every project written in C has suffered from a memory leak issue at some point? Most C/C++ code I've personally written doesn't even use the heap.

I work on avionics and we use C/C++ now and then. We have a ton of rules regarding memory management (pretty much everything stays on the stack) and I can't recall anything I've ever been involved with suffering from a memory leak.

If basically everything stays on the stack, you’ll have a much lower chance of seeing a memory leak, by definition.

Exactly. Using C/C++ doesn't have to mean using the heap.

Using the heap has nothing to do with the code being safe or unsafe.

All I'm saying is that it's a trivial oversimplification to say you can learn to miss every time.

Don't need a leak for C code to be vulnerable, in fact not using the heap helps - just one missing bounds check on user input and you can write to the stack. OTOH, W^X should catch those.

It’s not just the developers. The code gets reviewed and merged as well. I don’t know which projects but those are the highest quality C code based.

Writing in C doesn’t create complexity but lots of traps to fall in. Writing in C doesn’t save overhead over Rust. Your rust code can be very low level and safe and C isn’t that close to hardware as it used to be anyway.

It’s not for performance. It’s to settle an old grudge with Docker. All code coming from Docker must be erased.

See also: podman; cri-o; buildah.

Grudge? You mean when Jessie Frazelle walked around dockercon with a badge saying, “I will not merge patches for systemd” just out of principal? In hindsight it is fairly obvious picking a fight at redhat summit was a very poor long term strategy from docker. It forced them to ultimately sell the orchestration parts of their business and massively limit the free tier of docker hub.

Thank you for illustrating my point perfectly.

“Grudge?” (proceeds with details of grudge).

What was described could be the basis of a grudge, or it could be that Red Hat were convinced that the current project would not accept changes Red Hat would benefit by even if it helped many other people. At that point, and at the size of Red Hat and how important this technology is becoming for them, it only makes sense for them to secure some of the technological foundations they rely on. It costs them very little in resources in relation to the business they are basing on that technology.

I don't think it's a stretch to look at the events as described and see them as Red Hat doing what they need to work around someone else's grudge. It could be any number of things, but I don't think the information presented so far lends itself towards Red Hat having a grudge.

runc, containerd and the docker client are critical infrastructure to a whole lot of people, and they seem to be doing just fine running it and, when needed, contributing back. Only Red Hat seems to have a problem with using those projects without forking or re-writing them. Could it be because Red Hat considers it of strategic importance for their business, not just for those projects to be reliable (they are), but to be controlled by them to a sufficient degree that they can act as gatekeepers to upstream, thus preserving their business model?

That could be, but if Frazelle did actually walk around with a button saying “I will not merge patches for systemd” I think it's simpler and safer to assume Red Hat decided that relying on Docker was risky given how important it was to their future products and the apparent animus on display from the Docker maintainer.

If I carpool with someone and they like to go around and loudly proclaim "I refuse to listen to anything kbenson has to say", maybe I find someone else to carpool with or drive myself? That only seems prudent. That doesn't mean I have a grudge against that person, it just means I don't want to trust them with getting me to and from work anymore.

There is a picture of her badge in this article. It is not something I was alleging. It is fact:

https://lwn.net/Articles/676831/

Sorry, it was poorly worded on my part. I didn't actually doubt that, but prefer not to state other people's assertions as fact unless I know it is so for myself. I prefer to have my assertions grounded by what I'm basing them on, which is less about doubt of the source and more about keeping myself honest when arguing and not taking too hard a stance on stuff I haven't verified myself, since it's easy to come across as authoritative when I don't intend to otherwise.

Thanks for the evidence though, it's always best to have, even if it can be a pain to drum up when it might not really be called into question anyways. :)

Sorry the way I replied came off as defensive when it wasn’t meant to be. Your approach is to assume the best and that is always a good one.

It’s no secret that there were tensions betwen Docker and Red Hat that culminated around 2016, when Red Hat’s desire to capitalize on the success of containers ran into Docker’s own ambitions to compete with Red Hat. Those tensions were eventually resolved (or at least diminished) by Red Hat shifting gears to Kubernetes as their “next Linux”.

The drama you’re talking about is from that period - 2015-16. But “crun” was launched in 2020, a full four years later. Frazelle left Docker years ago, as well as most of the people involved in those feuds. Runc, containerd and docker are chugging along drama-free, and are maintained by multi-vendor teams. There is no interest from anyone outside of Red Hat in forking or re-writing those tools. It’s just a tremendous waste of energy and in the end will be problematic for Red Hat, because they will be building their platform on less used and therefore less reliable code.

The world’s container infrastructure runs on containerd, runc and docker. Not crio, crun and podman. Someone at Red Hat clearly is having trouble accepting that reality, in my opinion because they’re having trouble moving on from old grudges. I really wish they did, because all of that wasted engineering effort could be deployed to solve more pressing problems.

> It’s just a tremendous waste of energy

> The world’s container infrastructure runs on containerd, runc and docker

I don't consider alternate implementations of widely used software to be wasted evergy almost ever. Was Clang a waste of energy because everyone was using GCC? There are many reasons why multiple implementations may be useful, competition and being able to cater to slightly different common use cases obvious ones.

I'm not sure why any end user would wish for there not to be multiple open source projects looking to satisfy a similar technical need.

> in the end will be problematic for Red Hat

That may be, but I don't really care about whether it's good for Red Hat. I care that it increased user choice and maybe at some point features and capabilities offered to users (whether through podman or pressure on docker).

That’s fair, there’s always a benefit to alternate implementations. I think those were started for the wrong reasons (interpersonal conflict rather than technical requirements) but perhaps in the end it doesn’t matter.

I'll gladly use crio, crun and podman instead of the docker alternatives.

runc and containerd wouldn't exist if Redhat hadn't created the OCI specification and work around docker to make things interoperable. Docker, Inc was not playing ball and then redhat had to work around them. It wasn't a grudge it was just them keeping their business safe. The Open Containers Initiative was essentially designed to force Docker Inc to try to play better with others. They donated containerd to the OCI due to the reality that docker was about to be left in the dust.

For reference, this was the shirt that Docker Inc was giving away at Redhat summit years ago. I was there. I saw the shirt, and thought it was in very poor form. https://twitter.com/SEJeff/status/1125871424126767104

As Rambo said, "But Colonel, they drew first blood."

Your history is wrong. Docker created the OCI, not Red Hat. Docker donated both the spec and the runc implementation.

Containerd was donated (also by Docker) to CNCF, a different organization, and I believe a few years later.

You are correct that Docker did those things because of pressure to be more interoperable.

Reading your linked tweet (“don’t mess with an engineering behemoth”) you seem to agree that Red Hat is rewriting away Docker code because of a past grudge?

Docker and Redhat were both founding members of OCI. CoreOS probably was responsible for a lot of the initial conversations. They wanted a standardized container runtime after finding some limitations in docker which caused them to create rkt. Docker inc wasn’t super into changing anything and as history as shown, wasn’t terribly into working well with others. The industry was going to move forward without them but they were offered a spot on the OCI if they played ball.

OCI would have made docker inc entirely irrelevant if they didn’t join it. They donated runc (I was wrong originally, dyslexia sucks sometimes) as the reference implementation so docker continued to stay at the forefront.

They donated a very minimal implementation of containerd to the CNCF when kubernetes wanted to do things with the container runtime interface (CRI) to make it more pluggable. As a more purpose built CRI, crio is better suited for k8s.

Docker did not create OCI any more than Redhat did. Redhat and a collection of other orgs did. Docker and Redhat were besties after Alexander Larson added device mapper support to docker. This is what allowed docker to run on any Linux distro that didn’t include the out of tree aufs. Look it up. Redhat then made a lot of money doing container multihost orchestration with openshift v3, which deprecated their own tool, geard. When Docker inc accepted a ton of VC money, they realized they actually had to monetize things. This is the part where the relationship started to seriously sour. The company that helped make them so popular was now suddenly a huge competitor. This is what lead to the problem.

It wasn’t a grudge from Redhat I mentioned in the linked tweet. It was docker inc thinking docker was anything other than commodity. It just is some very nice UX and cli tooling with a container registry around Linux namespace and Linux control groups. They just put it all together very nicely for developers to ship code faster. Podman is also a commodity. At its core, it just is some json parsing and pretty wrapping around Linux namespace and cgroups. Redhat knows this, and they don’t hide it. They monetize openshift and quay, not podman.

Sorry if this is a bit scrambled. Long posts on mobile are difficult.

I understand what you’re saying but it is incorrect. Docker was in fact the originator of OCI in every way that matters. They made the decision to create it; drafted the founding documents and went back and forth with Linux Foundation on the content; chose the names (initially OCF, then later OCI); negotiated with LF the governance structure and initial board composition; negotiated the list of external maintainers with commit access, including a Red Hat and CoreOS employee; chose the announcement date and venue; drafted the announcement content along with LF; coordinated with PR teams of other founding members. All this in addition to being the original authors of both the spec and implementation.

Neither Red Hat nor CoreOS were involved in any of those steps, nor were they even aware of them until the last minute. When they were invited it was on a “take it or leave it” basis. They took it, then tried to pre-empt the Docker/LF announcement by a few hours to make it look like they were launching it. Those were tense days and there was very little trust between those companies.

Source: I was involved in the process of creating OCI.

Well thanks for your input, and I stand corrected. I liked Solomon and he always seemed like a super nice guy when I met him, but it did seem that there was a bit of a crisis when Docker Inc realized, "Oh shit, we have to make money for these VCs!" and then it got really awkward. I was sad to see it happen as Dan Walsh and Solomon are both really good peeps to me.

Yeah the tensions were really unfortunate. In the long run it resulted in more competition which benefits all of us; but in the short term it made people’s lives more stressful than they needed to be.

I actually have a slightly different explanation of what set it off. I don’t buy the “docker is suddenly under pressure to make money” explanation. Docker was a VC funded startup since 2010. They already had a business when they pivoted to Docker. As far as I know they kept the same investors and board. So it seems unlikely that they suddenly remembered that they needed to make money. More likely they planned to make Docker as ubiquitous a platform as they could, at which point there are many known avenues to monetize. Also, the CEO they brought on board, Ben Golub, had just spent a year at Red Hat after selling his previous startup to them. So he was certainly familiar with Red Hat’s business and strategy, and probably on friendly terms with their leadership (this is speculation on my part).

And in the early days Red Hat and Docker were in fact partners. Docker engineers even implemented features specifically to please Red Hat; for example devicemapper support and storage drivers, which were developed by Docker for Red Hat.

The problem is that the partnership was built on a misunderstanding. Docker was hoping for a distribution deal between the two businesses, with revenue share etc. and intended to keep control of their open-source project. Whereas Red Hat was hoping to make Docker their “new Linux” which involved no partnership with Docker and instead taking or at least sharing control of the project as they did with Linux. Each side was slow to realize the true intentions of the other, whether by naivete or deception I don’t know. This still surprises me because both side’s intentions were rational and entirely predictable. I think a lot of wishful thinking, and possibly some individual incompetence in leadership were involved. For example Red Hat has never actually partnered with a smaller startup and shared revenue with them in the way Docker was hoping, so I’m not sure what made them so confident it would happen. In any case, when the fog lifted, bitterness and conflict quickly followed.

In the end Red Hat shifted gears to Kubernetes as their “new Linux” which was a much better match for them. Then proceeded to rewrite history to minimize Docker’s role; which is unfortunate but understandable, I guess. I know a lot of Docker employees are personally hurt by it, to this day. It doesn’t feel good to see your work swept under the rug for reasons of competition and ego.

Alexander Larsson did the lions share of the work on device mapper, and here are 29 PRs to show it (including the very first with the devicemapper label):

https://github.com/moby/moby/pulls?page=2&q=is%3Apr+is%3Aclo...

In fact, Solomon's own words from https://github.com/moby/moby/pull/2609 "Integrate lvm/devicemapper implementation by @alexlarsson into a driver", so unless you're more knowledgeable about the history than Solomon, I'm going to have to discount your take on this entire thing.

Regardless, I think things are at a much better place now. It is unfortunate that there were tensions and more so that Solomon had such a tough time through it all. He's a super nice guy. But this is tech, and there will always be lots of personality conflicts. It is one of the precious few constants.

It is absolutely not correct that Larsson did the lion’s share. What he did was implement a Go wrapper for libdevmapper, which exposes a very low-level API. It is the Docker team that implemented devmapper-based container storage, as well as the whole storage plugin system which was now required to support more than one storage method. The original devmapper lib is utterly undocumented and Larsson’s wrapper did not fix that. So getting that feature to work was an all-consuming task and it is the Docker team that did the bulk of it.

You can see all this from the early history of the devmapper directory: https://github.com/alexlarsson/docker/commits/a14496ce891f1f...

So why does podman include code from docker?

Excellent question, especially since the code is copy-pasted without crediting the original authors. Could it be that they’re trying to minimize the role of Docker as much as possible even when re-using their code?

> Excellent question, especially since the code is copy-pasted without crediting the original authors.

That seems like a serious allegation. Can you back that up?

It’s right there in the repo history. Start from the first commits. I don’t know how serious of an allegation it is: copy-pasting open-source code without crediting it properly is rude, but not illegal.

No, that’s totally illegal. The Docker Engine is licensed under the Apache-2.0 terms, which require attribution.

It is actually against copyright law, and redhat legal would take any allegations very seriously. What in specific do you think is copy and pasted code?

Docker is properly attributed to, see https://github.com/containers/storage/blob/a4cc7aa79e050c976...

I think OP wanted to say that Podman hates Docker what is not I feel when I'm interacting with the community there. People who use Podman do it because of it's additional features that Docker does not have, like starting an Container from a rootfs or mounting the currect directory in a container using "." as path. Also building containers using a shellscript is supported. No need to learn the Dockerfile syntax. There are lot of small things that make Podman better integrated into Linux systems: running the container via systemd is a lot more intuitive.