Although, since an ASIC is application specific, couldn't you pack more cores (since you can ignore the circuitry that you don't need) or otherwise optimise for the application? If you can do more work, or do the same amount of work with less energy, then it could still be worth it.

I am obviously ignoring a rather important factor: its unlikely a "small" player could make a more performant ASIC than a "big" player like the GPU vendors, who have invested billions into building high performance hardware in a cost effective way. I'm not really asking "why don't people do this", but rather asking if its possible at all given a big enough budget.