Depends on the distance. As soon as you are out in a small boat there may be too far to the shore, especially if the water is really cold.

But at least keep the life jacket near you (ie on the boat), that way you can put it on if necessary - and never ever go out unless there is a suitable life jacket for every person on the boat.

Also never wear an inflated life jacket inside a ship (if it sinks and you become trapped you can't escape).