It is, in fact. (17 USC § 506) As with many things that are both crimes and torts, the crime is defined differently and somewhat more narrowly than the tort.
When I say "downloading a Youtube video isn't a crime" I'm indicating a category difference that means more than "specifically the government can't convict you over it." I mean that Youtube doesn't have a legal leg to stand on to make you stop; it's not a questionably legal activity that we ignore, it's not an illegal activity that's too hard to prosecute, it's not something you should feel bad about. It's not a (generic sense) "crime".
It's not just "you won't face a specific kind of legal trouble" -- you're allowed to do it. Scraping websites is allowed, and if someone sues you over it and you fight the lawsuit with decent lawyers you will win. Copyright violation doesn't fall into that category -- you can end up facing legal consequences for violating copyright. But downloading videos from Youtube is not inherently copyright violation.
It's possible I should have been used a more generic word than "crime" to avoid that confusion.
----
[0]: To be fair, I've also made the same distinction to people about lawsuits/crimes before, so I'm not sure I really have any grounds to call you out here :)