I keep seeing this stated as fact without any citation whatsoever.

Here's an argument with citations that the term "master" in the git context does indeed come from the master/slave terminology:

https://mail.gnome.org/archives/desktop-devel-list/2019-May/...