Leaving asbestos where it is can be better, but it'd depend on how well contained it is. If it's in a building that's showing its age, removal might be the only option.
I'm willing to give him benefit of the doubt on "nuance" given the number of filters we're seeing things through here. (It's bad enough relying on wikipedia for climate-related info at all, but when it's on a relatively obscure topic based on source material in another language, that's a whole new level of indirection.)
I doubt it's ok to paint him with a wide brush. He seems like a very intelligent person, extremely respected in his field and who's not afraid of a good controversy. Exactly what was the context of the falling objects dispute I don't know, but it's a fair chance there's some context to it. From a cursory look it seems to be more about how to teach the phenomenon then about the physics involved.
Losing 22 workers in France in the 1990s/2000s on a single asbestos removal project is extremely unlikely.
http://en.wikipedia.org/wiki/Claude_All%C3%A8gre
> In 1996, Allègre opposed the removal of carcinogenic asbestos from the Jussieu university campus in Paris, describing it as harmless and dismissing concerns about it as a form of "psychosis created by leftists". The campus' asbestos is deemed to have killed 22 people and caused serious health problems in 130 others.
I think I'd pass on his endorsement, personally...