The companies that make keycaps have molds for the profiles they make. GMK has the original Cherry molds (medium height caps with cylindrical indentions, Signature Plastics has a few profiles including Retro looking SA (Tall caps with spherical indentations, each row a different sculpt that gives it a gentle dish when looked at from the side), DSA (flat profile, each row the same with a spherical indentation), DSS (A medium profile generally dished with spherical indentation, DCS (a medium profile with cylindrical indentation), and G20 (a flat profile with no indentation).
So generally when you put together a custom keycap set, you pick a profile, if you want double shot legends there might be some extra molds to make, but those are pretty pricy (I think I heard about 6k a pop). If you go with dyesub you have a lot of legend freedom, but you are limited to dark legends on lighter caps.
Yes. HP calculators had that.
There's another way for small volume. Laser engrave the legend with a CNC laser cutter. Then brush on laser marking powder, which is very similar to printer toner. Rerun the engraving pass at very low power, hitting the same spots just engraved. Power has to be low, just enough to melt and seal the toner, not vaporize it. Then brush off any remaining powder. This is good for cover plates with legends, signs, and such. The letters are slightly below surface level, so they tend not to wear off too fast.
In contrast, the group buys in the enthusiast community are works of love, with multiple rounds of checks and adjustments to get it right. Sometimes facilitators like Massdrop force projects out the door because of cost even though the creator of the design wants to do more rounds of tweaking are necessary, and it shows. (Example: the recent /dev/tty keycap set.)
Seems to be quality issues with every XDA run so far though. Keyboard fanatics drop a good amount of money on these sets so quality is extremely important in such a small and passionate market.
https://pimpmykeyboard.com/