The (decimal) 40 characters were A-Z, 0-9, and four punctuation characters -- I don't remember which. Three characters could be stored in a 16 bit word, so a 6.3 filename required 3 words or 6 bytes. The '.' separator was not stored, of course.
The implementers of CP/M or one of those early PC OSes copied the idea but not the encoding and picked 8.3 as a slightly more usable size. Floppy drives were relatively capacious (!) so bigger filenames were reasonable.
The only reason I know this PDP-11 trivia is that end users could recompile the RT-11 kernel to save space, and DEC distributed the OS sources to enable that. The distributed version was written in assembler and had had comments stripped, so it wasn't quite like reading the Lions book, but you could read it if you really wanted to.
Why is that, exactly? I grew up in that era, I remember when floppies were actually floppy, and I've been well into computers ever since, but I don't believe I could tell you why. If I had to guess, I'd guess that FAT used a fixed number of bytes for each filename.