Yeah, trading off embedding dim vs hidden layer dim is a common trick that effectively means trading size on disk (and in memory) for inference speed. It's obvious the model will need to be more powerful if the embedding carry less information. Still cool they got the size down by 90%.