Maybe the current version of your extension doesn't use that permission to track history and browsing data. But what about after an update or two or after it's been sold to the highest bidder, unbeknownst to the user? Worst case scenario is totally what I'd want to see as a power user.