My curriculum in school involved creating our own integrated circuits on wafers. There are lots of nasty chemicals involved, but it's not out of the reach of a dedicated hobbyist.

The biggest challenge is ion implant, which you can probably outsource to a lab.