Apart from the price, I personally can't really complain about the modern Lego sets, either. They are nice puzzles, and my kids like to play with them. After a while they fall apart, and new things can be built.
I don't think Lego wants to pay for expensive Hollywood brands. They always try to establish their own brands, too, like Ninjago, Lego Movie, Nexo Knights. Often it seems the Hollywood stuff simply sells better.
Sales slump in 1993 - I don't think that agrees with your thesis. Maybe they put out so much stuff out of desperation, to find something that sticks.
Some parents prefer not to have brands or advertising within their children's toys, and Lego used to be free from this influence.
Of course they were commercial. They did a space theme because they thought it would sell, just like they are now doing Star Wars because they sell.
In the 1990s, almost no sets were "random boxes of colored bricks" and they were all themed, but the themes weren't the commercialized junk they are today. You had Castle, Pirates and Space without any references to movie characters and the structures were rather harmonic without too many special or overly-sized parts. The minifigs had serious faces and were not cartoonish. A child would be able to invent its own story and get creative with those themes, rather than using the toy to play when it's not watching the movie.