I'm not sure that it's harder. A good textbook or two combined with diligently reading, rereading and doing the problem sets is probably about as successful as class attendance in the average case, I would say. I think the problem is finding a good textbook and forcing yourself to do the work.