Do you really need photons and quantum stuff though? "Just" learn about how dielectric materials work + how EM waves work from Griffiths and then everything will make sense.

Depends on what you want. A useful approximation, or the latest approximation that is closer to how it really is.

What insight is being gained by going with the more complex model? You can work out a lot using classical EM and predict a bunch of cool optical/diffraction phenomena without a lot of math, and the math you do is crystal clear. QFT or even just QM+time-dependent perturbations are a pain in the ass in comparison.

If you have engineering mindset (how to do something), then, sure, you are better off with the most useful approximation. If you have scientific mindset (how things are), then practical usefulness of the model of reality is secondary.

I'm looking at Griffiths right now, and 9.3 Electromagnetic Waves in Matter states (emphasis in the original): "Since [the dielectric constant] is almost always greater than 1, light travels more slowly through matter - a fact that is well known from optics."

I'm still confused. How is it an inaccurate simplification to state that the speed of light in glass is lower than in a vacuum?

I wouldn't call it inaccurate at all. I think it's a perfectly good model. You can explain the phenomena otherwise using a more detailed model, but you don't need to.