(Many computers THINK they have IPv6 connectivity when they do NOT and so if they see an AAAA record and try to connect to it, it can cause a delay while that fails. Most browsers are better at this now (they will try to connect to both and show whichever answers faster), but not all network things are browsers.)
I have personally checked the result from my then-home connection (unfortunately, through he.net tunnel) and it worked, and it was properly logged by Apache. The CEO had native IPv6 on the phone, and it also worked for him, and, according to Apache logs, IPv6 was also used. I have also checked that the site was still accessible over IPv4. So I indeed don't know what went wrong.
On the other hand, I did not have access to Google webmaster console and had to trust the CEO that the SEO problem exists. It might have been a coincidence. And again, the SEO problem was in 2018, and is likely irrelevant today even if it really existed back then, that's why I said "in the past".