It seems to me that the problem with Bitcoin is that its "consensus rules" (the flowchart to follow to decide a block of transactions as either being valid or invalid) aren't instantiated anywhere, but rather are just buried within the particular implementation details of BitcoinCore.

In an ideal world, you'd split these up:

1. there'd be a currency steering committee (with some structure whose ideal form is above my pay-grade) that produces a machine-readable validity ruleset (sort of like a DTD or XML Schema);

2. and then there'd be a proliferation of competing clients that all know how to parse those rulesets, and use them to validate blocks; but which otherwise go about their business of being Bitcoin clients in entirely separate ways.

With such a split, issues with the management of "BitcoinCore the FOSS reference client implementation" wouldn't overlap with issues with "Bitcoin, the ruleset that, in being instantiated in a majority of nodes, gives them a consensus chain." Instead, the first would just be a regular code project, no longer at-all critical; and the second would be something more like the project of defining a new version of e.g. the ePub standard, or of drafting an RFC.

There are several problems with this. Firstly, a machine-readable validity ruleset would basically just have to be a library implementing the rules, simply because of their complexity - for example, one of the previous validity changes requires scripts of a particular format to be treated as hashes of the actual script, which is provided by the transaction redeeming it. Secondly, bugs in implementations can effectively cause the to enforce unexpected additional rules, such as the BDB lock limit which caused an unplanned hard fork a few years back. Thirdly, everyone involved still needs to be a skilled developer with knowledge of distributed systems - especially the currency steering committee, since changes to the validity ruleset can break the whole thing in subtle and unexpected ways - and the committee still needs in-depth knowledge of the actual implementations to know if they can cope with their planned rule changes.

The challenge is that the rules are only legitimized by the clients choosing to follow them. You can't launch new rules until at least 50% of clients by weight are following those rules. There is no way for the ruleset to somehow disable or ignore a misbehaving client, you need consensus.

Right, this would just make the difference explicit: given a good spec for defining the rulesets, a client would be able to just give a drop-down for which ruleset to follow (or allow you to point the client at your own by URL, etc.)

They have been split up. Bitcoin Core now builds a separate library, libbitcoinconsensus.