There's nothing wrong with using SDL for making a game (well, maybe a couple of things, but that's not relevant here), but it simply isn't the smallest GL scaffolding library out there.
But I got bit by half-assed "GL" ways of doing things before. SDL has everything, in layers, so you can grow into it. Unlike "portable GL" libraries which are discrete, isolate dead-ends.
GLFW does not attempt to handle OpenGL for you, it handles windowing and input for you. I personally really don't want SDL anywhere near my projects because it imposes its own structure upon me. (I don't use Allegro, even for 2D code, for similar reasons.) I am simply not all that fussed by writing up a VBO handler and some pretty simple OpenAL code myself--it isn't difficult enough for me to warrant the huge blob of extraneous SDL stuff.