Multi-story buildings take only a few days of brick-laying by 4-5 people. Before you can lay the bricks though, you need a firm foundation to carry those bricks, including those concrete floors which take a while to dry up.

There is still a bunch of work in construction that can happen while brick laying happens (plumbing and electrical installations in the floor, pillars to carry roof/next floor, ...) and there is usually a mandated licensed professional to oversee the construction (or few, one for safety, one for quality and ensuring the project is actually followed).

And you can also start doing things as parts of walls get completed (installing windows, insulation, more electrical installations). After the building is put under a roof, that's when the real work begins :), though that's even easier to parallelize.

Brick laying is literally one of the quickest parts and most effective parts of making a building look like a building, but it's nowhere near the most costly thing.