The difference between bugs in non-crypto software and bugs in crypto software is that bugs in crypto software can have much more severe and far-reaching consequences. So, while I might trust a programmer to write decent non-crypto software, I would prefer not to trust them with writing crypto software.
edit: actually, there's more to it than that, on second thought. Crypto also requires a greater depth and breadth of expertise. The math knowledge required for general programming is trivial by comparison; about the worst it usually gets is vector-based math, or simple calculus, or big-O notation. But to understand crypto well enough to implement it correctly requires a much greater knowledge of mathematics -- something which most programmers don't have.
> You know what the weaknesses of Javascript-based crypto are, and you know how to implement crypto. In that case, why not do it?
Because I (the rhetorical "I" in this case) know what the weaknesses of JavaScript-based crypto are. :-)
The same goes for the non-security things I've listed. The problem is much deeper than getting crypto wrong -- most programmers today get everything wrong. That's why I think if you're the type of person that can think about programming, it's not too unsafe to implement AES in Javascript and use it. You know what the weaknesses of Javascript-based crypto are, and you know how to implement crypto. In that case, why not do it?
Remember: most non-crypto software is massively incorrect. If we can trust people to implement crypto, why can we trust them to be programmers at all?