You split your project into two components:
1. a new, clean, minimal core with a nice, shiny, new API; and
2. an "OpenSSL emulation layer" that loads your core library, and wraps it in OpenSSL-compatible cruft.
It'd be very similar to, say, replacing Direct3D with OpenGL, and then writing a Direct3D emulation layer that calls OpenGL.
Unless there's an actual federation behind the project that can fund it and keep it stocked with talent and oversight, something Mozilla or Apache could do, it will continue to be an unmitigated disaster.
The only reason it wouldn't have been caught in unit testing is because Apple isn't unit testing their own SSL code. If Apple doesn't commit to doing security well why do people expect OSS to?
People did rage over Apple's SSL issue, but Apple handled it swiftly, pushing out updates to all affected devices as quickly as practical. It's also worth noting it was a validation bug, not something that could leech arbitrary bits of data from a device.
I have no idea how some open-source projects get anything done, their development process is insane, but they manage to hobble along. Something as significant as OpenSSL needs to be held to a higher standard.
If the OpenSSL team can't get their act together, it would be worth it if a larger non-profit organization took the initiative and made a tested, rock-solid implementation of same with an optional compatibility layer.
I've had to integrate with OpenSSL before and the code in there is truly mad. Well written code should be boring and obvious, but OpenSSL is the polar opposite.
But how did it ever get so widely adopted, if it's so hard to use?
And even if it is widely adopted, that still shouldn't stop anyone from forking it and fixing all the problems.
Usefulness and lack of alternatives?
Look, if this was about something fairly inconsequential, like a library for a game or a blog or something, sure, use some badly maintained, badly documented, inconsistent library. But when you care enough about security to use SSL, surely you care enough to use something that's put together in such a way that you can be sure it does what you think it does? Bad documentation and inconsistent error handling in a security package sound like an unbelievably dangerous idea. And if it's really an unmaintainable mess, like everybody says, then how can bugs possibly be fixed? How easy would it be for the NSA to insert some crucial vulnerability into the code base?
Security, moreso than anything else in computer science, needs to be clear, transparent, understandable, well documented and well maintained. If it's not, mistakes will be made and security will be compromised.
Take the aforementioned error return inconsistencies -- how do you possibly deal with refactoring that for consistency when so many other projects are consuming OpenSSL as a library? You're trading one big problem (new clients are likely to get error return handling wrong) with another (if you modify all the returns for consistency, existing clients are now expecting the wrong return codes or, if you're smart and rename the functions to avoid that, at the very least the 3rd party consumers now need to do a lot of rewriting and deal with a painful crossover phase supporting both the old and new APIs).