This is the same trick used by the old 'scratchbox' development environment that Nokia used for the maemo SDK. If the stuff you compile in it includes classic "configure and make" C/C++ projects then you'll probably also find that replacing the /bin/sh binary with an x86 one improves speeds -- configure and make spend quite a lot of time spawning shells to run bits of shell script.

Hey, thanks for the insight - I didn’t realize this was a thing for some Nokia envs already - pretty cool!

re: x86 make and shell, it probably won’t work well here because once a process tree goes to x86 land, qemu cannot see it and it can’t “natively” run an arm binary.

Essentially we can only hook execve when emulating - when we run a native binary we no longer have any insight into any execve call it may make. (Though with some additional work I suppose one could also use LD_PRELOAD to hook execve)

This is why binfmt_misc and registering the interpreter with the kernel would be superior, if not for kernel level access requirement

QEMU doesn't hook execve anyway, at least not upstream -- when a QEMU emulated process invokes another we rely on the binfmt-misc magic to cause the kernel to run QEMU on the child. We don't special case "the binary being exec'd seems to also be for the guest".

It looks like maybe you've made some local patches to QEMU that relate to execve?

I know the qemu-bsd-user fork has some hacks for the bsd-user stuff that looks at the binary to be exec'd and does different things based on the target since there's some optimizations you can get out of this if you don't rely on the kernel 100%... Bits of them may even be in my latest patch-set :)

Cool stuff! Do you have a public branch you're working off of? I'd love to follow the progress as I wasn't aware that qemu user emulation for BSD was even a thing.

I didn't get a chance to peruse beyond briefly skimming the source on GH, but I couldn't find any special handling around execve - could you point me in the right direction?

I was looking at bsd-user/syscall.c, whose equivalent in linux-user is where my repo's execve hook lives, if that makes any difference.

Yep! I have a patch to qemu based on one that never made it to master upstream. It’s a sub module in the repo

I like how cross-compiling works on Debian, you install the native cross-compiler on your amd64 system, auto-install the cross-dependencies built natively on arm64 systems onto your amd64 system and then build. Of course, not all dependencies are cross-installable and not all packages are cross-buildable, which is why there is a QA effort for automatically doing test builds.

https://wiki.debian.org/CrossCompiling http://crossqa.debian.net/

This should be at the top: https://github.com/valkmit/aws-graviton2-on-intel#justificat...

Thanks, I’ll add it now

I think they are suggesting to move the justification section of the github page to the top of the github readme. (Instead of posting another comment on HN)

Justification:

https://github.com/valkmit/aws-graviton2-on-intel#justificat...

FreeBSD has used this same trick for half a dozen years or so to build arm, mips and riscv binaries using fast Intel servers. The FreeBSD build system can build itself to produce binaries for the target (so the compilers run natively) and qemu handles the rest with user-mode emulation via bsd-user (which I happen to be in the process of upstreaming).

Any chance you can add a high level explanation on how this works?

Prepare a rootfs for ARM64 system. chroot into it and emulate every binary with qemu-user. Now replace gcc etc with one configured as cross-compiler, but still point its sysroot to the usual path (/usr/...).

This is equivalent to building a x86-to-aarch64 cross-compiler and point the sysroot to a ARM64 rootfs, but more intuitive to users unfamiliar with cross toolchains, and more friendly to those mad configure scripts which insist on assuming host = target and detect everything automatically.

You forgot the following steps: Profile to see what's the slowest thing Make a native version of that. In FreeBSD package builder setup, we also have /bin/sh and a few other heavily used binaries that are native because it shaves hours or days off a package run (itself takes 10 days, even optimized like this).