I'd recommend starting off with easier primary sources. Per that lesswrong post about coming up with solutions too early and how that constrains thinking, the same critique applies here.
But, cultural background is useful, I would place that in a different category than the secondary summaries of philosophers' work.
The problem is: you have to start somewhere. And with most philosophers, the primary texts aren't going to make any sense at all if you don't have the context build up already. Secondary sources are a necessary evil at that beginning stage.