To be fair, generics are only needed for as long as you insist on static typing. Languages with dynamic typing where types are checked at runtime do not need generics.
And generics is complicated topic. I am all for static typing, especially for large code bases. But sometimes when I venture too far out in generic land I find myself wondering if it is worth it? A well-designed library that uses generics prudently can be a bliss to use. But the implementation code can get near-unreadable.
You can always omit them, if you're prepared to deal with code duplication, but I think you're downplaying its significance: it is not just "maybe some" code duplication, nor is it a "a few casts". It's sometimes a deep, murky swamp you're forced to trudge in, and it's not pleasant.