I actually have an older phone (a first-gen Galaxy S) on which I installed Cyanogenmod without Google's services, which is basically stock Android. In spite of opinions to the contrary, the phone is quite usable without any of Google's services on it.
I've got Firefox for web browsing, OsmAnd for maps, Apollo for playing music. All 3 are open-source apps. At some point I even installed Amazon's App Store on it, to be able to get some proprietary apps, like Facebook, installed. But then I realized I don't need all that bloatware, as the web versions work fine.
You should try out this experiment if you've got an extra Android lying around. After you find apps suitable for whatever it is you want to do, it really is OK. The level of polish is not the same as with the Google-enabled Android, but it's still much, much better than the experience you get on low-budget non-Android smartphones.
People forget that Amazon already forked Android and that without Android they wouldn't have been able to deliver Kindle Fire. That's the power of open-source, even if you choose to deny it.
"For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software."
This might be a fun experiment for a developer. But for the broad user perspective even replacing launcher is a big deal.
The point is, Android is currently the go-to mobile OS for anyone who wants to scratch their itch. like Linux has been on the desktop. And there have been plenty who have taken up this offer (e.g Amazon, Chinese & Indian manufacturers, Android console makers), except they aren't end-users.
I'm actually curious as to why F/OSS applications that aren't extremely high profile or commercially backed (Firefox) seemingly tend to have UI/X as what is probably the last possible priority ever.
Yes, the renderer needs some work both in terms of prettiness and speed. It's not as polished as Google Maps, but it's definitely usable. That the map data is from OSM is a feature by itself, OSM maps are often more detailed than what you get from Google, and you can download all data, which is great for traveling. It even supports navigation.
Your argument above is in the same line as people claiming that end-users do not benefit from open-source. In some sense it is correct, as end-users do not have the technical know-how to assemble their own operating system, to make changes to source-code, etc... But practically speaking, end-users don't have to do it by themselves just like they don't have to fix their cars by themselves (e.g. I helped friends install Cyanogenmod on their phones, which is a PITA to deal with, but once done, it's done).
The power of open-source is in the ability to fork it. Not all people or companies can do it, but if there's demand for it (e.g. Google's stewardship goes awry), then a successful fork will happen. Building an operating system from scratch is hard. Having stock Android available as a base for new operating systems is an incredible gift.
In the meantime, yes, forks of Android are only for hackers wanting to play around. But if this leads to the rise of alternative marketplaces, like F-droid, then this will benefit everybody. Not sure if you notice it, but the biggest problem that Android has in terms of openness and lock-in is Google Play. Fortunately Android allows installing software from third-party sources, but unfortunately there aren't many alternative marketplaces out there.
I think it would make a lot of sense if the base Android OS (AOSP, as I understand it?) were only as functional as it needed to be for useful apps to run, and then the "Google app stack" or "Cyanogen app stack" or whoever else's, open source or closed source or anything in between, could be dropped in to provide the user-facing functionality.
The key seems to be getting to a point where important pieces of that underlying functionality aren't wrapped up in the "Google app stack".
I think Google is doing what's best for Android, but Android is not open source anymore.
Firefox OS might be a better choice for open source comminity to spend time on. I haven't been following latest development, how's the hardware situation on FF OS?