More to the point: Bananas are radioactive precisely because of the large amount of potassium they contain -- naturally occurring potassium contains a radioactive potassium isotope. You must consume some amount of radioactive material as part of your daily potassium intake regardless of whether it comes from bananas or other food sources.
Brazil nuts are another matter, as they are radioactive because they contain radium, which your body doesn't need.
Don't try so hard to avoid radiation sickness that you wind up malnourished.