The common way transaction fees are handled in US equity markets is called the "maker/taker" model. When a participant adds liquidity to the exchange order book, AND that order is executed, they are given a very small rebate per share (they are the "maker"). The other side of the order, the firm who "removed" liquidity from the order book, is charged a very small fee. The difference between the rebate and the fee is the spread the exchange earns for executing the trade.

Note that not all orders that add liquidity qualify for rebates, the rebate may depend on particular order properties as well as the way in which the trade executes.

(not a trader but I've researched it a bit) Yes, often per share traded. A lot of traders use Interactive Brokers which has a few different pricing structures[1].

[1]: http://www.interactivebrokers.com/en/p.php?f=commission

Sub-pennies, negotiated directly with exchanges. They do the same negotiation that Interactive brokers does, except unlike interactive brokers, HFT players can trade 10% of market volume which give them significant leverage.