But you can't really pick up most primary sources and actually get anything out of them without understanding the historical/philosophical context they were written in. If you can find someone with little to no philosophical background who can pick up A Critique of Pure Reason or the Tractatus and actually get anything out of it then you've probably found Kant or Wittgenstein reborn.

That's why the parent commenter said you need a guide/instructor to properly understand what is going on in these books, and I agree with that. But for people without such a luxury, secondary sources and articles are a good method for gaining some rudimentary understanding of the philosophical topics.

I don't disagree with some of your larger point. Most people will get the fullest benefit out of a classroom setting, with an engaging and well-read class leader and an engaged and thoughtful group of students to argue with.

However, I still believe that a determined outsider can get real benefit out of philosophy on his or her own. (Not the fullest benefit, but significant benefit.) I completely agree with you about cultural/historical/philosophical background, which is why in my answer I made a point of recommending not just titles but specific editions for classical works. Maybe we could call this the best of both worlds: good primary sources, filled out with enriching secondary material, all in one edition? (It does tend to make those editions more costly than some, and also heavier.)