The embeddable package from python.org [1] is 20 MB unpacked. To install different sets of packages per workbook, just (edit: have Excel automatically) modify the module path as appropriate.

[1] https://www.python.org/downloads/release/python-3114/