What made smartphones useful was people discovered new uses that you just couldn't use a computer for. You can't do turn-by-turn navigation with your laptop in the car. It's nearly useless if you're standing at a street corner in an unfamiliar city and want to know where the closest coffeeshop with wifi is. You can't use it to take a picture of the cute little ducks crossing the road and send it to your wife. It does you no good when you want to hail a car to take you home.
People haven't yet discovered similar uses for a smartwatch. It's highly likely that they exist, though. What the watch gives you is hands-free information: you can receive quick, 1-5 second notifications even if you're carrying something or immersed (sometimes literally) in physical activity. I've already found this useful for getting notifications that my founders' group has changed where we're meeting while I'm driving there, or that my wife has already gone to the grocery store, or for checking off items on my shopping list while carrying shopping bags. There are probably other bigger uses waiting to be discovered.
But your phone can also give you that, through earphones or headsets. Smartwatches must find new uses that a phone cannot also do as well, which is a much harder bar to clear.
For example:
> notifications [...] while I'm driving there
Phones can do that already, and will do more and more as CarPlay etc become mainstream.
> checking off items on my shopping list while carrying shopping bags
Siri can probably do that already (I use Trello for my shopping lists and it works fine as it is, tbh).
The one use where smartwatches will likely be unbeatable, forever and ever, is fitness/health: carrying around a phone in those circumstances is simply more annoying than carrying a watch. Once they get cheap enough, they might replace smartcards and badges for access control. If smartglasses become mainstream, they could augment each other (your glasses display and record, your watch stores and computes) and actually replace phones altogether; but alone, I don't see them going anywhere.
People might also not have 5-6" devices in their pockets, but special glasses with HUDs or smart contact lenses...
[1] Smart contact lenses are such a crock. Seriously, guys, how do you think you're going to power these things?
For the same reasons we use smartwatches with our phones today. It's more plausible they'd fit a great cpu, cellular phone, gps, etc on a watch size, than they'd fit it in a decent glass frame.
So the watch could handle most things, plus there are interactions that can't be done with glasses (like measuring one's heart rate, or similar).
It could basically split smartphone features into two gadgets that are easier to carry around. For that, though, we still need quite a few leaps in miniaturization.
I.e. never? I mean, I can picture making phone calls or dictating SMS by voice with a smart watch, but people are not going to be browsing the web or taking pictures with a tiny wrist device when they already have a 5-6" inches display device on their pockets, and almost no one would ditch the latter for the former if they had to choose.
I think it's likely that smart watches will continue to be niche devices for the foreseeable future, at very least until someone figures out a killer feature that sets them apart from smartphones.