They're not legally required to be instant, but they usually are. The "ambition" set out legally was half a day, if your transaction is initiated in the morning it should clear that afternoon. It's just that in practice instantly is the realisation of this ambition when things work. And I don't think there's any API requirement, the regulation just explains the consumer experience, that you can transfer relatively small amounts of money (enough for a nice car, not enough for most houses) fast ("Faster Payments" is the name given to this feature), and doesn't dictate how.

For large transactions transfer for a fee already existed. CHAPS will move much larger sums of money (it's typically used to buy property so certainly millions but perhaps more) for a modest fee. You wouldn't want that fee on your weekly groceries, but when you just bought a house who cares?

Last I looked the backend for Faster Payments wasn't actually built. The big banks decided instead "temporarily" to just trust each other. If Bank A says Cathy sent Mike £5000 then Bank B where Mike's account is will credit Mike £5000 (probably instantly), presumably Bank A will reduce Cathy's account by £5000 and the two banks agree they'll settle things at the end of the day. This is only scary if Bank A might not actually have that £5000 to give Bank B at the end of the day when it's settled, which in principle should never happen under current financial regulations.