Sure, it's possible. But I don't think it's easier. You get into all kinds of path/rpath fun and the worst thing is that things may work but you end up using the wrong environment by accident. Chroot could help, but docker has it all nicely wrapped up with the tooling.
I’ve done what you mention with Docker and alpine builds (for alpine Python wheels, for example) but honestly it’s a lot more painful to ssh into running containers when figuring out the build script than it is to just throw up a clean VM.