I am not a banking programmer, so take with grain of salt. Perhaps there's some evil super-complex-and-unavoidable issue at play, but I doubt it.
Banks do this millions of times each day on the global ATM network. I can swipe my bank card at most any ATM in the world and within seconds the funds are removed from my account and dispensed into my hand.
In that case, the card is indirectly linked to my bank account. It's only when you know the exact bank account and routing number from which you want to remove the funds that you're forced to use the archaic ACH system.
Clearly the technology for instantaneous confirmation of funds transfers already exists, but it has yet to be expanded beyond its original purpose.
Making a withdrawal from an ATM/ePOS does not debit your bank account immediately, but it does reduce the limit (i.e. available funds) on your card immediately. Checking your "balance" at an ATM shows the card's total available funds, not your bank account balance.
ATM operators are often not the same as your bank, even though they may show your bank's logo at the top. There are fees at every stage of the transaction so ATM operators and ePOS transaction acquirers can decide whether to ask your bank/VISA etc whether you have the funds. For small, low risk transactions they will often wait, preferring to batch a bunch of debits to a single institution at a quiet (cheaper) time of day.
Your card is reconciled with your bank account daily (usually daily). A payment to a third party takes longer - the payee's bank needs to complete reconciliation too and, as other commenters have pointed out, the network breaks sometimes.
Banks can improve the time it takes to transfer funds to third parties and have done so in many countries but it has taken government intervention. The float does play a part, but getting competing banks to agree to a standard is almost impossible.
Side note: Your card has many, many limits most of which you will never encounter. Examples: total cash per ATM per day; number of cardholder-not-present transactions per day; total spent in gambling establishments per month.
Banks (and credit card companies) make a lot of money on the float, so it's against their natural incentives unless they can win more business as a result. I think they probably could, but it would take a coordinated roll-out of new products or acquisition/integration with an external company... which is difficult to do because of regulation. So you're in a position where you'll either need more or less regulation to do it - either the government mandates it, or the government deregulates and a competitor starts up who currently doesn't have market share, so has nothing to lose by introducing the new features.
It's not about the float. It's about the up-front investment in the standard, which millions of bank customers have already implemented. It's also about the risk of allowing instantaneous funds transfers; the current delay allows for at least some mitigation. It's also about the fact that there is no new standard yet. (Based on what I know about NACHA, there won't be for a while.)
Money already seems to be moved somewhat instantly, but only ever out of my accounts.
I find it surprising that it's been 15 years since the commercialization of the Internet and (near) instant global data communication and it still takes banks 3-5 days to confirm payment (e.g., the time it takes to send a physical letter, coast-to-coast, via the USPS). This is the timeframe of a standard Electronics Fund Transfer (ETF) for bill pay, bank-to-bank transfers, etc. The only alternative, as far as I know, is a wire transfer, which is a tedious and expensive option.
Does anyone know what's preventing the U.S. banking system from improving its electronic transfer capabilities? Given the importance of the free flow of money and goods (and liquidity) to our economy, it seems like the quick movement of funds would be a significant priority for our overall banking system.
(Note that if banks could confirm electronic transfers within a few seconds (and if we added an extra layer of security beyond simple passwords, such as a challenge/response system), we could significantly reduce the occurrence of fraud. Near instant confirmation would also alleviate, or eliminate, the issue of short-term credit mentioned in this article.)