Indeed, even QoS optimisations need not violate Network Neutrality if done in a destination neutral way, e.g. using the IP QoS flags. You could offer someone 1Mbit of dedicated low latency bandwidth out of their normal higher bandwidth connection, and let them use it however they want.