For grayscale data there's very little ambiguity, since chroma and colorspace conversions aren't involved. Basically just rounding in DCT, for which you can make reasonable assumptions.

Moreover the JPEG XT spec (not to be confused with JPEG XL or the ton of other X-somethings JPEG made) has specified precisely how to decode the classic JPEG format, and has blessed libjpeg-turbo as the reference implementation.