I think you're tilting at windmills here. The type of people who think gambling should be legal and is entirely a personal decision are the type that would agree with you.
Only take it a step further. The state shouldn't move heaven and earth to get anything privately owned built. That it's a casino only makes something inherently bad even worse.
If you want to gamble your house away in a private bet, go for it, but the local government shouldn't move heaven and earth to get a casino built to enable thousands and thousands of people to make those decisions in the name of tax revenue.
Similarly, I don't think you would be taking away people's freedom in any meaningful way if, say, you limited lottery transactions to say, $20.