I agree that interesting stories don't have to adhere to this structure (and by all means, stories should break the rules!). But, the western market generally demands it for salable material, especially novels.

Even in this conflict structure, the best antagonists are the ones that believe they are doing good. It's not a trite good vs. evil construct. They have their own desires and a philosophy that can be a logical way to solve a problem (but in conflict with how the hero wants to solve it).

Ghibli films do this best. For example, in Princess Mononoke, the antagonist (Lady Eboshi) is initially presented as the evil forest destroyer, chopping down trees to make villages and industry. But then, you find out she does this to provide shelter and work for social outcasts. It's a moral dilemma with no clear right answer.