I made the mistake of joining startups early in my career. I still suffer because of those decisions. It is much better to work for a big brand when you are young. Yes, in startups you get to wear a lot of hats but unless you are really lucky, you never learn anything in depth. Big companies have enough good people to learn from if you are motivated enough (as I was). And worst of all, unless you are one of the founders, and the startup is reasonably successful - then they will bring a "VP of Engineering" from a big company after two years irrespective of how much you have helped them build their systems and probably deserve that role.
In another, it was lots of JVM knowledge (both threading and memory management/garbage collection), caching, distributing work which needed to be calculated, etc. We found several JVM thread locking bugs and worked with Sun to fix them. I worked with some really great people in their 40's at this one and learned a lot from them.
I suppose the majority of startups are different these days though? Mostly RoR and Node based social things. The biggest worry is if the Facebook Like and Share buttons work.
If you are working somewhere that is only a CRUD website at it's core and doesn't have anything else going on, I suppose there isn't much to learn on the backend side of things.
Some of the people from my graduating class in college started at really large companies and learned nothing because they were such a small cog in a really large machine. I went to a company which varied between 40 and 50 people and had the opportunity to learn as much as I could soak in.
Anyway going to a startup is a lot like gambling. You'll most likely be looking for another job soon, but if it pays off it may pay off really big. Who cares if they bring in a "VP of Engineering" if I can cash out my options for six figures?
If it's been two years, the options are worthless, and it looks like you're going to end up a low level employee at a larger company, then it's probably time to move on.
It is hard to speak for all startups since they're a dime in a dozen.
The sweet spot is to find a company developing a new product or something of the sort.
Where I work now I started on a new product that was young, and since moving on from that project, we have replaced many legacy applications/products. I've done my longest employment stint (4 years) of my career because it's been my best job at this point.
I haven't yet been a founder, but I've been at 3 startups which have been acquired. At this stage in my career, if compensation is your primary concern, my opinion and experience is that your best bet is a bigger company who offers stock as part of your compensation somehow. I've definitely made more at a bigger companies than I have as an employee of successful startups with liquidation events.
However, when I was younger, startups were a great place to get interesting experience and network with some great people. I'd recommend working at them if you are in your twenties, or founding one if you have the right personality and backup plans for that. I'm in my mid 30's and would be relatively hesitant to be an employee of startup in its early days now.
You also need to consider if it's a true startup, or just a smaller mom and pop lifestyle type company which is not growing like crazy. The latter is probably better for someone like me, but a quality startup in your 20's is something you'll look back on fondly no matter how much it may have sucked at the time. It's a roller coaster ride but you'll get great experience and have a great network of people to know when you get older.
Keep in mind I'm speaking as an employee-- a founder of a startup is a totally different thing.