I have only programmed very little AI for games (standard Alpha Beta Pruning), but I think I can already understand why AI in games is not better. It simply takes a lot of time to create a very good AI. It is not done with having a good baisc idea (like Alpha-Beta Pruning for chess), you then also have to tweak it endlessly.

I think if you can come up with an effective way to create good AI, game developers would be delighted to use it. It is only time constraints that makes most of them stick to "good enough".

Also, the game has to remain fun for the humans.