Beyond that it's not necessary because most people follow the law. The same irresponsible people who speed like crazy would also remove the speed limiter. Also there are general driving conditions, emergencies, private roads, and other situations where people use the full capability of their vehicles. And police are still need to patrol the roads for more than just speeding.
Overall a built-in speed limiter just limits freedom, costs more resources and enforcement, punishes responsible owners, and prevents them from enjoying their vehicles because of a tiny percentage of bad drivers. In fact Germany's autobahn with completely unrestricted speeds has a better safety record than other freeways which shows that fast cars at high speed are not the issue at all for safety.
Why is it such a bad idea to just fit in speed limiters and be done with it?
And that last part? It's pretty much an empty argument, as if we don't already regulate/restrict/ban many things with the potential to cause harm (cars included).