A UI for creating a shopping cart. We can rank two different variables here:
1) Number of users making a purchase. 2) Number of users returning goods because they didn't actually want it.
If we focus only on the first option in our user testing (as any shopping cart user testing will do), we will generally make the 'fine print' descriptions of items as small as possible. We will make it very easy to buy an item with a single click. This is going to give us the best 'purchase rate', guaranteed.
However, we are going to get increased returns as users will miss important product details. We will get lower 'user happiness' with our store as users will feel a bit cheated and have to turn to external sites for more information. In the long run, users may stop using our store entirely and our purchase rate will fall.
This is a classic example of 'short term' or 'greed' based optimization. I feel nearly all UI testing I've seen does not cater to variable B - only to variable A. Catering to variable B isn't even really possible with a UI or A/B test?
I understand it (someone please correct me if I'm wrong) as being the same as usability testing. In other words, putting users in front of a particular design with a task you want them to complete. You then observe anything they struggle with. You can also measure how long it takes them to complete the task. These things inform you about how good your design is at the same time as making some design flaws very evident. One thing that is sometimes included (but I personally disagree with) is asking the user for an opinion.