The way it was explained to me, astronomers considered them metals because they’re much more rare than H and He. They’re not made in the core of fusing stars. That might seem trivial, but a huge amount of the universe is H/He compared to the rest of the periodic table.