Limit access in the firewall/load balancer/webserver to whitelisted ip addresses. If you don't have static ips at your office or a vpn, include the ip of a VPS or dev server, and use ssh forwarding (-D for a socks proxy) to access the admin site. And if you're running everything from one server, you can even limit connections to localhost and tunnel connections through there.