The way I've done it in the past is that a feature will have an "anchor" or someone who owns it. They'll work on most of it and be in charge of interfacing with product on how it should work.
That person would switch pairs every day, working with a few people who then pick up some of that context. Being hit by a bus is the classic example, but a more realistic one is that the person goes on vacation. Now, instead of just being 1 person with deep context on that feature, there's potentially a few others that will know it well enough to solve the problem.
The people that are constantly switching will eventually own a feature and won't have to multitask all the time. Also, when you're pairing with someone, you get to share some of the focus with that other person, which I've found to make switching contexts a lot easier.
I think it's admirable to want to spread knowledge around as much as possible, I think having individual developers "own" specific applications or features is a net benefit. Sure if a developer leaves, they take their knowledge with him and someone else has to get up to speed, but that will necessarily happen anyway.
There is only so much room in your brain; the more features or projects you work on, the less you remember about any of them. Being able to go to the feature X "guy" most likely means the problem will be solved significantly faster. That person's caches are already primed, the pages are loaded from disk, and they can hit ground running. That is a much better case to optimize for than the off chance somebody gets hit by a bus. We also know that multitasking is bad for productivity so the less people have to focus on the better.