I'd also be worried about the model being biased. For example, perhaps the model is "better" because it uses correlated features it shouldnt be using (due to fairness laws, etc.)