The problem is really that multiple parties may be able to claim the rights to all of your output while you were on their payroll, which is obviously messy from a legal standpoint.
Of course you can try to get that clause removed during contract negotiation, but I doubt many people would be successful on that negotiation (you might have better luck to exclude some of your open source contributions).
"A typical silicone valley company's contract usually have some clause saying that all your creative work while contracted with the company, even on off hours, belongs to the company."
My experience, working for a number of silicon valley companies, has been the exact opposite. There is a clause that says (in legalese) "we don't own stuff you do on your own time with your own equipment if it isn't related to what you do at work"
It is my understanding that California public policy actually requires companies to put this disclaimer in work contracts.
In Germany it definitely would not, at least not without it being specifically allowed by your contract.
But maybe in the US, as long as you didn’t have a conflict of interest?
Would the amount of total time worked matter? Would doing it in multiple states simultaneously matter?