And since it's non-refundable unless you can prove there is a cheaper option, it's sunk money if there's nothing significantly better. I'm honestly not sure how much stronger this is than just using the multi-flight tool on ITA Matrix in a lot of cases, especially since the secret is basically they'll hunt for the hidden city flights, which with a small bit of research you can do yourself.
It's also one reason why low-cost carriers like Easyjet and Ryanair have thos low operating costs, they only offer point to point flights so there is no hidden-city option and no need to pay for a department to scan for it...
From the airlines' perspective those fares actually make sense. A direct flight should be more expensive than an indirect one, even if we are talking about the very same flight.
It's not a bug, it's a feature and here to stay...
http://www.nytimes.com/2011/05/08/magazine/mag-08subversion-...
and more recently...
http://www.aol.com/article/2014/12/31/united-airlines-suing-...