I would be interested in evidence that "mundane" activities were directly connected with the decline of the program, but I think you're inserting modern biases of race into the public sentiment of the 70s. I don't think people looked at golf explicitly as a "rich white man's game" in a pejorative sense, nor do I think the people that would, were pulling the strings in Congress or NASA.

My understanding was one of "Okay, we beat the Soviets to the Moon, now what?"