Actively campaigning for a future where humans lose their humanity without the least thought as to the potential dangers? Imprudent at best, crazy at worst.
And I certainly wouldn't characterize singularitarians as advocating a human-less future without considering potential dangers. Anyone associated with the Singularity Institute for one will at least claim to consider potential dangers.
I'd place John von Neumann as having been extraordinarily sane, and one of the most productive minds in history. I'd put Eliezer as far crazier and more potentially destructive than Kurzweil — there's not much distance between "creating survivable Singularities" and Butlerian Jihad. At least Kurzweil did productive things before he decided to live forever and tell us all about his magnificent future. Yudkowsky's brother died, and then he decided that nobody should have to die again.
"Primarily concerned with the Singularity as a potential human-extinction event, Yudkowsky has dedicated his work to advocacy and developing strategies towards creating survivable Singularities."
My pure speculation is that many people feel a very understandable sense of unease, but it gets expressed as doubting the possibility of the singularity future. As opposed to accepting it as possible but advocating that we try to avoid it, or at the very least approach it with extreme caution.
Kurzweil, on the other hand, comes across to me as very Pollyana-ish. He seems to spend very little time considering what could go wrong with a Singularity, and much more time eagerly anticipating our impending god-like future.