BuildKit in depth: Docker's build engine explained

Re:BuildKit in depth: Docker's build engine explained by pimterry on Wednesday February 07, 2024 @03:35AM (#39286016)

I've spent a bunch of time playing around with Docker's API directly. BuildKit is notable because its API is completely undocumented AFAICT - certainly the API listed under https://docs.docker.com/engine/api/v1.44/#tag/Image/operatio... is just the API for a normal non-BuildKit build. If you want a BuildKit build, you have to use the Docker CLI.

I did some digging and wiresharking, and I'm pretty sure it's undocumented because the API is _insane_. It starts as HTTP/1 from the Docker client to the Docker engine, but a key BuildKit feature is that the engine pulls files & data from the client on-demand, which is hard in a normal REST API, so how does it do that? By renegotiating the connection to flip the direction after the client connects.

That means: the client sends an HTTP/1 request, the server offers to upgrade to HTTP/2 by in reverse, and then the server becomes an HTTP client and the client becomes an HTTP server, still on the same existing connection. All actual communication then happens as gRPC, but backwards.

Absolute madness, and very difficult to document or support in 3rd party SDKs (and so they haven't) but it's very clever. Some more context here: https://twitter.com/i/web/status/1423353288129396740

[ Parent | on HN ]

Re:BuildKit in depth: Docker's build engine explained by kylegalbraith on Wednesday February 07, 2024 @03:44AM (#39286072)

This is one of the things that led us to write this post! It's almost all undocumented, as you're mentioning, and there are quite a lot of complexities underneath the surface that often feel very inefficient as well.

We're working on another project in this realm that might interest you. Happy to send over more details via email if your interested in better build APIs. My contact info is in my profile.

[ Parent | on HN ]