Not really. That kind of code complexity is simply not worth the tradeoff. You'll waste a bunch of time fixing bugs in your dual representation codebase. It's always better to do the simplest thing first, and then optimize if and only if it turns out to be necessary. In this case, the simplest and most effective way is to use strictly vector representation, because it will result in the lowest complexity codebase.

Modern computers are monstrously fast. So fast that it's hard to imagine just how fast they are. The more time you spend thinking about optimizations before they're necessary, the more time you're not spending on implementing features.

If it's just for fun, though, go for it. A learning exercise is always nice.