But if there were "no extension", that would be a 0-length extension, rather than a 3 character extension with 3 spaces.

Even if it has to be 3 characters, shouldn't they at least be \0?

Since spaces are in fact not allowed, choosing between 0 and 0x20 is pretty arbitrary here. You’d probably go for 0 today because C became popular and 0-padding has the useful property of also 0-terminating the string, but at the time that was less of a consideration.

I assume any extension that consists solely of any combination of \x00 and \x20 is to be understood as "no extension"

In what world are files required to have an extension?

They weren't required to, but the convention for the shell was, unlike unix with its X bit:

extension COM: just load the file into memory and start executing from the very beginning (I think the max size of these might have been 64KB or something).

extension EXE: this is a "full" executable with different segments - load it properly

extension BAT: this is a "shell" script, run it with the interpreter

The advantages of COM over EXE were slightly smaller file size on disk and slightly faster startup as you didn't have to set up segments and stuff first. Some versions of DOS even had a program called EXE2BIN that converted an EXE file into a COM file, if this was possible. The joys of the 16-bit "tiny" memory model ...

The space for the name is pre-booked, so in a sense all files always have a full 8-char name and 3-char extension, whether they like it or not.

If you want to support <8-char names, <3-char extensions, or no extension at all, you need to decide on values that indicate this, because those bytes will still need filling with something.

(On 6502-based systems, 0 would be a reasonable choice for a padding value, because you get a free zero check every time you read a byte. The 8086 doesn't work like this... I don't think any particular value suggests itself. Perhaps they thought picking a printable character would be useful.)

Picking the space character, apart from being I think an established convention already (the CHAR(n) type in databases), definitely has the advantage you can just "blit" the filename onto the video buffer in text mode when doing a DIR command. You don't have to bother with checking its length or anything, just copy 11 bytes and insert a space (not dot in the original DIR listing format) after the 8th one. If the file doesn't actually have an extension, this just prints the right number of spaces so the other columnms are aligned.

EDIT: see https://en.wikipedia.org/wiki/Dir_(command)#/media/File:Spar... for an example.