In my view, there are two good reasons.
First, the legal job positions that only require a mechanistic understanding of the law have been automated. The days of needing a "country lawyer" whose practice consisted of basic family law like wills are gone, thanks to LegalZoom and similar services. The jobs that were best filled by these people have been automated away. In the jobs that remain, it's not enough just to know the law, but you must learn to think in a legal/analytical way. This is not a skill that comes easily just from reading statutes and some case law under supervision.
Second, the size and complexity of the American legal regime, including the administrative state, has made it necessary for lawyers to (a) specialize; and (b) be able to issue-spot in areas well outside their expertise, to know when to look for help. Although a mergers and acquisitions attorney might be able to apprentice a young aspiring lawyer in that area, they're unlikely to be able to transfer all the latent knowledge they have of other areas of law. As an example, I have about a decade of experience in my practice areas, and would be wholly incapable of apprenticing an aspiring lawyer in the area of, say, commercial paper to the point of being able to pass the bar exam. The first requirement (depth in a specialty area) makes most practicing attorneys ill suited to train someone to standard in the second requirement (latent breadth).
Given the high cost of lawyers, and the general understanding that you avoid going to court as much as possible, as it can easily bankrupt you before achieving any good, its hard to imagine that's actually true. The current options are to pay a huge amount, or give up entirely; I'm sure there's a market for the middle ground (inexperienced but cheap lawyers), and a reason no one can access (eg regulation)
There is no such regulation. A fresh graduate could theoretically argue a case in the Supreme Court--it just requires a pro forma application and payment of a $200 fee to be admitted to the Supreme Court bar. A JD you hire for $20/hour on Craigslist can, legally, represent you in almost any proceeding, as long as she is upfront with you about the fact of her inexperience (and is willing to shoulder the malpractice risk).
There is no market for inexperienced but cheap lawyers, at least for business litigation. If you're a plaintiff, you can get experienced lawyers to work on contingency. If you're a defendant, you have huge potential exposure if you lose the case. If you really have no money, the plaintiff will realize it can't get blood from a stone and will settle cheaply. It would be incredibly risky to roll the dice and try to fight with a lawyer who lacks experience.
Caveat there - there's room in the boiler room, at least on the bigger cases.
Not exactly the kind of work kids dream of when they apply to law school, though.
There isn't. Law is the same profession it was 150 years ago. (There are new tools, of course--electronic research, e-discovery, etc.--but you don't really learn to use them in law school anyway.) The JD requirement is just one instance of the larger capture of American society by academia.
Your point about supply and demand is astute. The JD requirement didn't replace the apprenticeship requirement, it supplemented it. Today, lawyer supply is still limited by the opportunities for new lawyers to go to trial, handle deals, etc.--nobody would hire a lawyer who has only academic or theoretical experience in those areas. But the winnowing process happens after a whole bunch of people have taken six figures in non-dischargeable debt.