It's not just about performance, or really not so much about performance - e.g. when Java added generics, they didn't make it any faster than it used to be when you had to use Object everywhere.

It's about type safety - i.e. catching bugs earlier and more reliably. Every time you have to write an explicit downcast, you are throwing away the benefits of using a statically typed language in the first place. Which is fine, but then why stop halfway, and not just go full dynamic? It doesn't really make sense to have strong typing for scalars but not for collections.