Just because a CYP is involved, doesn't mean the target substance is hard on your liver in any way. Liver toxicity is highly non-linear with dosage, and damage typically starts abruptly when substrates are depleted. The classic example is paracetamol, which is quite safe at reasonable dosages (and without alcohol) where the liver is well within its limits, but causes damage rapidly at higher dosages when you run out of glutathione.

This nuanced explanation is sadly lacking in many places I encounter the phrase "liver damage". I have frequently read articles which would lead one to believe that things are bad for your liver linearly and are "unsafe at any speed". It's irritating.

Moderation is the important thing here. It's easier to avoid consuming drugs every day compared to eating a high fructose diet.