I'd say if a company takes venture capital it's a start up, whereas something based on bank loans and self capitalisation is just a business. But I don't have any justification for that - just my association of the word.
I thought a start-up needs to have a product/service that aims for ultra-high-growth.
That seems to be a relatively accepted definition around here, but not a dictionary definition.
Yeah, it became popular, so no definition is really true anymore. Everybody tries to claim it for themselves to get the coolest tech people and the most VC dollars.
It's often a bit confusing.
I get a job offer from people who say "We are a start-up" and then they are just another dumb agency. If I want agency work, I freelance.
I heard it only for the business you describe, but lately every newly created design/tech-consultancy here calls itself a start-up.