Worst thing that can happen is that you get hired by a terrible company with a toxic work environment that doesn't respect its employees. That's wasting way more time and energy than any wasted interview.

So if you see red flags and have other options that look more promising, walk away.