> but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front.

Yup, that's always been the big reason to use containers for me. Startup time and runtime performance are nice benefits, but the memory usage is the giant win. Freeing memory in response to the apps need and also not needing extra memory for running the various OS parts and pieces.

The down side is, of course, security. But that was always the case with containers.

I think in the longer run, WASM might displace a lot of both in practical terms.

Why would wasm replace containers? If you're going to run a binary why not just compile it for the local system?

We've always had 'compile once, run anywhere' but there's always been caveats and gotchas.

wasm targets the wasm runtime Virtual Machine (ie: a JavaScript VM). Offering fine grained isolation compared to virtualizing the whole operating system.

You could say the same thing for anything that targets JVM or CLR, and they're far more mature than any JS runtime.

Or even Lua, which is trivial to sandbox.

Something still compiles a WASM binary for the local system. Possibly, being able to optimize the WASM without recompiling it from source might be a win? Not needing separate binaries for ARM and x86 is nice, so it should run on a Mac more easily. Also, it runs on an edge server or in a browser, even on a phone, if you care about that.

I don’t think it will replace Docker files since they let you package up such a wide variety of existing server software and WASM is more limited. But if your software does compile to WASM then maybe you don’t care about that.

I think of WASM more like a plugin format, but I expect there will be a lot of engineering effort put into optimizing it, like happened with V8 for JavaScript. Not all web standards win, but betting against one that’s well-established and has a lot of support seems like a mistake.

For serverless (as in AWS-lambda-like) I agree. In that usecase WASM provides a better security barrier than containers, with faster cold-start time (which is really important for the scaling promise of these services).

For the stuff people run on their Kubernetes clusters I have more mixed expectations. Containers are more universal, but I can totally see a microservice architecture running as a lot of WASM runtimes with a handful of containers.

Particularly in the case of Amazon lambdas, those are running in VMS already (firecracker). Why wouldn't you skip the VM and use a static binary from rust instead?

> The big problem with Katacontainers is not whether or not they are slightly faster or slower than containers, but the fixed memory allocation which means you must first know and then allocate the maximum amount of memory they might ever need up front.

Conversely the problem with containers is that memory allocation including the OS page cache is not guaranteed. That's bad for a lot of applications, especially databases. It seems Docker has some support for shared page cache but it's not in the Kubernetes pod spec as far as I can see. [0] You would probably need some kind of annotations and a specialized controller to make this work.

[0] https://github.com/kubernetes/kubernetes/issues/43916

In Kubernetes 1.25+ page cache usage accounting is improved thanks to use of cgroupsv2. https://kubernetes.io/docs/concepts/architecture/cgroups/

Slightly off topic, but regarding the larger memory footprint of Kata containers, what is your opinion on KSM effectiveness in general for VMs?

We had a lot of reports of ksm/ksmtuned consuming a lot of CPU and not making a lot of difference. I think it works well for certain workloads, and can be quite pessimal for others. There are also security concerns because you can leak information about (eg) what glibc is being used by another tenant using timing attacks. So you'd probably want to turn it off if multiple tenants can be using a single node.

Kata containers support memory ballooning like most modern VMs: https://en.wikipedia.org/wiki/Memory_ballooning so a fixed allocation isn't needed, reducing over provisioning

https://github.com/kata-containers/kata-containers/blob/d50f... uses virtio-mem

This isn't a substitute (nor is virtio-mem, the modern equivalent). The problem is the application running in userspace inside the guest cannot request more memory when, for example, it does a mmap or sbrk.

* * *

i couldn't understand the comment that "the application running in user space cannot request more memory" - can someone explain whats the point of memorybalooning anywhere if an application cannot signal when the system should actually provision physical memory from the 'baloon'

There isn't a point, that's the problem.

The systems administrator can use ballooning to give more memory to a VM before launching a new application. This avoids the need to shut down the VM to give it a new role.

There is still a benefit to ballooning support even if it's not exposed to userspace within the VM, because VMs aren't always used purely to host a single infinitely-long-lived application without outside intervention.

Which application languages and frameworks support this kind of dynamic memory allocation? For predictability in performance and throughput reasons we benchmark our Java applications on specific cpu and memory constraints and specific heap and memory settings. How would an app in a container suddenly give back ram? A garbage collected application may be able to do that by collecting garbage. Possibly. But others?

No idea about Java, but any C program will request memory using mmap, and may give it back using munmap. This doesn't work when the program is running inside a VM, but does work for containers (which are basically just regular processes).

With VM balloning, VMs are also able to claim and release memory to the hypervisor/host OS.

Not driven by the guest application they don't. It's frustrating that I actually work in this area and know what I'm talking about. I worked on Katacontainers back when it was Intel's Clear Containers in the late 2010s. So many people in this thread do not have a clue.

How does this work in practice? If the application didn’t need this memory anymore because it is done with the work/data, shouldn’t it already have freed or munmapped it? Is there a signal that can be send to a process to free up and return memory?

Linux supports hot attach ram or the VMM could support memory ballooning. VMs don’t necessarily need to all be backed by physical RAM.