If you already knew that Nabokov's interviews consisted entirely of remarks prepared in advance... why would you post that as a counterexample to "he wasn't able to speak perfect English"? That's the whole reason he prepared his responses.
Choirs frequently perform songs in foreign languages; nobody takes that as evidence that the members are able to speak those languages.
The reason Nabokov wouldn't speak in public without index cards was not ignorance but perfectionism. Index cards were his instrument—don't forget that he wrote his novels on those cards as well. For Nabokov, speaking in public without his index cards would be like not being properly dressed, or maybe dressed at all. Plenty of writers have that anxiety, including ones who never leave their native language.
You're still right, though. Nabokov's spoken English was nothing compared to his writing, and his genius would have made him painfully aware of that. The OP contains a partial version of this famous quote: "My private tragedy, which cannot, and indeed should not, be anybody's concern, is that I had to abandon my natural idiom, my untrammeled, rich, and infinitely docile Russian tongue for a second-rate brand of English..." The key word there is tragedy. To him his English was second-rate, and he would know.