Theoretically (I have no idea about the Pi specifically) it's entirely possible for the bootloader to speak USB, load the kernel + initramfs from the device into memory and then then boot a kernel which knows nothing about USB.

This can be done with (for instance) u-boot.