> I was aware that they are a separate asset and don't belong to the assets of the broker. Little did I know that brokers could still just lend them out

AFAIK they could lend them only if that is a part of the contract between you and your broker. My broker has a separate optional amendment to the brokerage contract for that, which is required only if i wanted margin account instead of regular brokerage account.