This got me reading up on the topic, and apparently you can download a free book from the FAA that covers how to do celestial navigation: Flight Navigator Handbook, https://www.faa.gov/regulations_policies/handbooks_manuals/a...

(Apparently early models of the Boeing 747 even had a sextant port built into the roof of the cockpit.)