There are also religions that require you to rest every 7th day. Having those two leap days would be anathema to them.
You would end up with the "religious week" and the "standard week" moving relative to one another over the years, in a classic case of trying to fix messy details and thereby adding more details (as in https://xkcd.com/927/).
Even without that issue, many Orthodox Christians still use the Julian rather than the modern Gregorian calendar for religious purposes (while still using the Gregorian for daily life). That's why Orthodox Christmas and Easter are on different days than the Catholic/Protestant ones.
Even the Roman Catholic Church that uses the Gregorian Calendar has moveable dates: Easter Sunday always falls on a Sunday and not on fixed day of a fixed month. On the other hand, Christmas day is always on December 25. Go figure.
The point of weeks is to have consistent cycle that can organize life. Also, why is it advantage to have the same day of week for same day of year? I suspect most people would prefer that their birthday wander around instead of always on Thursday. Calendars solved the problem of figuring out the day of week, and computers make it even easier.