Technically, all you have to do is follow the written instructions. But there are a surprising number of maybes in those instructions. You hit a checkbox that asks whether you qualify for such-and-such deduction, and find yourself downloading yet another document full of conditions for qualification, which aren't always as clear-cut as you'd like. You can end up reading page after page to figure out whether you should check a single box, and that single box may require another series of forms.
My small side income takes me from a one-page return to several pages, and next year I'm probably going to have to pay estimated taxes in advance because that non-taxed income leaves me owing at the end of the year more than some acceptable threshold that could result in fines. All because I make an extra 10% doing some evening freelancing.
Most people's taxes shouldn't be complex, but in practice they're more complex than they should be.
If I can do this, most people can do a simple 2-page 1040EZ.