By "inventory nightmare" you mean "theft", right? Because that's the real problem. There are already self-serve lines in many stores, but they still have to watch people to try to prevent theft. And it turns out people are really, really slow if they aren't trained as a cashier.

Sure, the cellphone eliminates the slowdown, but it also eliminates the primary way they currently prevent theft.

Amazon is working on a store that works a lot like you said, with your cellphone, but doesn't rely on the customer doing anything other than logging in. After that, tons of cameras take care of the rest.

Imagine that each piece of inventory had a per-item unique RFID permanently attached to it. Not just a model unique RFID, unique to that particular instance. Imagine that your phone can read that RFID. Your phone also has an app that can connect to a database, identify the particular product that the RFID represents, and complete a transaction online. Once the transaction is complete, the store's security systems are updated so that that RFID can pass through the entrance scanners without the alarm going off.

Potential pitfalls include the cost of the RFIDs, the ability to detach the RFIDs at any point, the ability to read the RFID on your phone, and alternative entrance / exits of merchandise (e.g. toss through a window).

Are there other reasons that this could not work?

Or just set up self-serve stations. Literally every item has a barcode.

more self-service kiosks would be an easier solution rather than building an entire new system.

If you used your phone, then it would easy to shoplift. Buy 7 items, but "forget" to scan the last two. Self-service kiosks at least prevents this with weight sensors, cameras, and a human kiosk manager.

Some Coop supermarkets in Switzerland allow this.