You're describing the sort of operation a nation-state's intelligence services would plan, and I'm describing the sort of thing anyone from a crazy ex-lover to an unethical competitor might do.
Your scenario is obviously more likely to succeed and less likely to be detected, but mine works more often than not. Being sure I have not done anything to attract the attention of an intelligence agency is not sufficient to be sure nobody has gained physical access to my laptop and installed malware.
Lots of things can go wrong with this sort of plan. But it's not a one-person job.
You might have your rootkit totally prepared and ready to go, and find that the person has a sub-1-kilogram sized netbook/small laptop that they take with them literally everywhere they go, in a backpack, small bag, or purse.
You might have an attack prepared and get there to discover that the laptop has had its USB ports disabled in the BIOS, with a BIOS password set, and insufficient time to work around that.