That's the usual arrangement, yes: phone - WiFi AP - ethernet switch - desktop; no internet needed. SSH is not some only-works-in-the-cloud magic, just a TCP/IP service; as long as you can access the server, it is cloud-agnostic.

I do use this arrangement with split DNS: inside the local LAN, the desktop's DNS resolves to the desktop directly; outside, it resolves to the gateway's external port, whence it's NATed to the desktop. SSH from anywhere :)