I see the opposite right now, with transfer rates being about the same but latency over IPv4 being quite bad compared to IPv6:

Pinging www.google.com with IPv6:

    $ ping www.google.com
    PING www.google.com(sc-in-x67.1e100.net (2404:6800:4003:c02::67)) 56 data bytes
    64 bytes from sc-in-f103.1e100.net (2404:6800:4003:c02::67): icmp_seq=1 ttl=108 time=4.46 ms
    64 bytes from sc-in-x67.1e100.net (2404:6800:4003:c02::67): icmp_seq=2 ttl=108 time=3.73 ms
    64 bytes from sc-in-f103.1e100.net (2404:6800:4003:c02::67): icmp_seq=3 ttl=108 time=3.73 ms

And with IPv4:

    $ ping -4 www.google.com
    PING  (142.250.186.100) 56(84) bytes of data.
    64 bytes from fra24s06-in-f4.1e100.net (142.250.186.100): icmp_seq=1 ttl=107 time=317 ms
    64 bytes from fra24s06-in-f4.1e100.net (142.250.186.100): icmp_seq=2 ttl=107 time=317 ms
    64 bytes from fra24s06-in-f4.1e100.net (142.250.186.100): icmp_seq=3 ttl=107 time=317 ms

I have no idea why that happens. It's not on all sites though, so it's not like my ISP adds latecy to all IPv4 sites, but rather something to do with routing. The IPv4 traffic might be routed to a google datacentre further away.