Agree, and the article sets up a weird dichotomy between empiricism and innovation. The empirical work of adapting a Transformer architecture to be smaller or faster or work better on a particular task is scientific work. Like a lot of Comp Sci, a lot of it is engineering, but there is a clear scientific component in that you come up with a model first, then implement and test it (using a lot of pre-built components). It's certainly as scientific as e.g. designing a medicine.