That's the OP's point. At the time, the community was split between word-level, which has the shortcomings you're describing, and byte-level which is uselessly compute intensive. BPE was the first reasonable in-between. BLT improves on BPE by having the the compression learnable rather than precomputed