You may be mistaking American secularism with laicism.
> It is not the freedom to impose your religious mores on other people who do not share your religion, or even those who do.
I agree. This is why the Christian concept of marriage should not be imposed on others, like Muslims and Mormons who are perfectly happy with polygamy.
That's a common viewpoint today, but there's nothing in the 1A about _private_ exercise of religion. The public exercise of religion is very much what is protected though there are limits when it affects others or infringes on other laws (e.g., bigamy, etc.)
Tax breaks and other marriage incentives have a legitimate function in domestic policy (e.g., it's very much in the state's interest to encourage two-parent families for raising the next generation of productive citizens). While those reasons are not always entirely uncontroversial, it's unfortunate that they are sometimes swept up into the debate around "getting government out of the marriage business."
Civil (ie. government sanctioned) marriage is a social construct that most people want to, and eventually will, enter into. People enter into marriages because they want the legal framework around it. If there were no legal framework around marriage people would have to run around filling out paperwork everywhere setting up their partner as their next-of-kin.
There's also currently no non-marriage way to create the same sort of civil partnership that marriage brings, stuff like spousal privilege and unlimited inheritance just to name two. And these are things people want. If all people were interested in was religious marriage or partnership/love/companionship they either wouldn't bother getting married or wouldn't bother registering their religious marriage with the state.
Its currently impossible to get government out of marriage without also fundamentally changing our concept of marriage.
There's arguments that the "family/marriage" concept is benefited too much in society, and I'd probably agree with that.
"Religious liberty," especially in the context of the American founders, is the freedom to practice your faith privately in whatever manner you wish, and to speak about that faith publicly without fear or persecution. It is not the freedom to impose your religious mores on other people who do not share your religion, or even those who do. To imply otherwise is to bastardize what this country was founded on and does a disservice to the founders.
If you want government to stop meddling in marriage, I hope you're fighting against tax breaks and other marriage incentives just as hard as you are against allowing two people you'll never meet to enjoy the same recognition straight people do.