I would even argue that any time your dream job makes an offer, you should feel free to leave your current job regardless of status, loyalty, and so on.
This is true, but not without context.
In my scenario above, the circumstances were similar. 2-3 weeks into a new job.
Excluding someone who has been unemployed for an extended period of time, it seems odd that in 2-3 weeks, losing your job would shatter their life.
---
And I wouldn't exclude companies for being "shattered" either. It's very possible for companies to be formed around the reputation of a single individual, and if that individual left, would be shattering. Especially in early stages of a financing round.
Your current best offer will be willing to stretch out their offer for a week. And the other company will be willing to accelerate their decision process, so as to bridge this two weeks gap.
But once you made your decision, you should stick with your decision... at least for a year, all things other than this 'dream job' being equal. The dream company and the dream job-- if you got in today, you are likely going to get that again in a year.
If you got accepted a job offer, and within 2 weeks got a job offer for more money, and at your dream company, would you feel obligated to stay with your initial decision?
If you made the rational (but arguably unethical choice), do you think your reputation should be tarnished, and future employers think hard before extending an offer?
I know it's not the exact situation as the OP, but situations are rarely so straight forward, and often have nuances that we aren't aware of.
To the OP, at the very least, you probably dodged a bullet. So blessing in disguise.