Personally I like SFML because you can just slap it into a quick project and get things rolling quickly but I agree with it trying to handle too much. I suppose the main reasoning was that it was intended for creating video games. At least you can pick and choose different headers ¯\_(ツ)_/¯