* LLVM defines an abstract container format, called the bitstream. The bitstream has "bit" in it because it's bit-aligned instead of byte-aligned.
* LLVM IR has two on-disk representations: a textual representation and a binary one. The binary one is stored within the bitstream container and it's code, hence bitcode. The bitcode representation is much more space efficient than the text one, so it's the one that's (almost) always passed around. As a result, people tend to refer to specific bits of LLVM IR that have been serialized to disk as "bitcode."
* LLVM hasn't stood for "Low Level Virtual Machine" for a very long time; probably over a decade. They (presumably) changed the name for the same reason you've pointed out: it's a little bit misleading. That being said, you can interpret LLVM IR as if it were just a compacted interpreted program: `lli`[1] is an official tool that does exactly that.
TL;DR: bitcode is a representation of LLVM's IR. The "bit" in it is a reference to the underlying structure, which is relatively unusual among program container formats (almost all are byte-oriented, as you've observed).