I can't remember if we ever did this, but the solution seems straightforward, just start an upgrade container that has both versions.
We definitely did not use the container that's in the locker library though. Ours did some custom checks on startup.
There is an issue about this (from 2014), but it has yet to be resolved: https://github.com/docker-library/postgres/issues/37