Out-of-bounds read and write in the glibc's qsort()

Re:Out-of-bounds read and write in the glibc's qsort() by scythe on Monday February 05, 2024 @02:06PM (#39265490)

Trivially, NaN != NaN, so this function is not even properly reflexive (i.e. for a = b = NaN we find cmp(&a,&b) = cmp(&b,&a) = 1).

[ Parent | on HN ]

Re:Out-of-bounds read and write in the glibc's qsort() by jstimpfle on Monday February 05, 2024 @02:44PM (#39266054)

I mean, yeah. It's not like there is a lot of meaning to NaN. For most applications, trying to sort NaNs is probably a bug in itself. So I think this is likely a perfectly valid comparison function.

To make a point, here is a sorting function I'm using.

    static int cmp_seqnos(const void *a, const void *b)
    {
        uint64_t x = (const uint64_t *) a;
        uint64_t y = (const uint64_t *) b;
        if (y - x < ((uint64_t) 1 << 63))
            return -1;
        return x != y;
    }

I think this should work if there are uint64_t a, b: b - a < ((uint64_t) 1 << 63) and all values x from the set to be sorted are in the range a..b. Note that a < b isn't a requirement here.

Sorting such numbers can be useful when dealing with sliding windows.

[ Parent | on HN ]