An in-between option might be to use https://github.com/drtyhlpr/rpi23-gen-image -- it generates a Debian distribution for you with (more or less) only the packages you specify.

The most recent image I built ended up being around 700MB, but I included some stuff you may not need, so you might be able to get that smaller.