Statistical significance doesn't mean an effect is "true", or "real". So getting rid if it shouldn't cause you any issues.