If you bind crux-kill-other-buffers[1] to a key, you might end up with a workflow you're happy with.

[1] https://github.com/bbatsov/crux/blob/master/crux.el#L496