It could conceivably be done even below the schematic level, though I'm not sure how much room modern processes have for this sort of thing now that we're talking about how many atoms wide a transistor is. I've been told that there was a period of a few years in which a kind of "copy protection" proliferated in IC layouts. The layouts would be tweaked to exploit quirks of the originating company's fabrication process (e.g. a pattern might look like a diode but actually function as a resistor when fabricated), and this would sabotage attempts by other companies to clone the chip (at the time, Japanese and Soviet clones were major concerns).

The lower a level, the easier to add a backdoor, the harder to audit. To add the backdoor to the handwritten assembly code is much easier than to do this with (reasonably clean style) C, and both are easier than add it to Standard ML or Haskell code. The same is true with verified formal hardware description specifications, Verilog and lithographic mask.

So the path toward trustworthy computing, besides cheap fabs, is higher level tools, projects like Kami and CakeML, proof checking, automatic verification and synthesis.