The 'staging' area is implemented through the index. And the index is used for more things then just deciding what gets in the next commit. A lot of gits speed comes from caching Stat data of files so that it does not have to hash the complete working tree for each operation. That's not something you can just ignore.

Someone proposed splitting this up the other day, but even that would come at the cost of performance and an increase of complexity.