https://www.ams.org/notices/201410/rnoti-p1249.pdf
Back when I was using Mathematica a lot (2005-2010ish) the Mathematica usenet group kept an archive of all errors in Mathematica people had found, but it seems to have disappeared off the Internet.
If you browse the [bugs] tag in mathematica.stackexchange.com you'll find a few as well (although admittedly many of the issues there are user errors rather than Mathematica getting it wrong)
Here are two that I found:
the character table of some finite groups weren't square https://mathematica.stackexchange.com/q/252775/7936 this one makes me really nervous as this implies there's not unit testing for the most obvious checks (it's a theorem that the character table must be square and the check is trivial to implement)
conjugate transpose didn't do anything to vectors https://mathematica.stackexchange.com/q/258165/7936
There are plenty more where Mathematica produces the wrong antiderivative or limit.