It's also worth noting that if you are born in a specific caste, and convert from your religion, you will still be subjected to caste discrimination from the folk in the original religion.
Obviously under the rules of the current Supreme Court anyone can claim their bigotry is because "religion" and carry on doing it. Don't want to serve someone? claim doing so violates your religious beliefs. Renting a property? ditto. Employment? again.
I'd be interested to know whether any other major religion has anything quite like the caste system.
There may be a degree of mobility between ranks, but mormonism at least makes it explicit that a black person for instance starts at a subservient rank, and can never reach the top rank, as that is reserved for white people.
Your position in a job, or the availability of that job, or housing, or in fact anything, should never be restricted simply due to some religious doctrine.
Whether or not that's particularly convincing or a reason not to pass it is debatable, but I think it's a valid reason that merits consideration.
Any reasoning beyond this ends up being gobbledygook.