The issue with rallying is that it used to take place in remote locations with little or no crowds. There are no safety barriers to protect the audience. Additionally, the rally audiences got crazed by deadly dangerous activities like standing in front of the cars until the last second or trying to touch the cars when they zoomed past (there's a story that a mechanic once found a severed finger in the side mirrors of a car).
It became impossible for the race organizers to control the crowds and make the race events somewhat safe. That's why "exploding popularity" was an issue.
Group B was terminated after an accident in 1986 Rally Portugal (?) when a Ford RS-20 car flew off the track and into the audience with dozens of fatalities and serious injuries. This was only a few weeks after the fatalities of driver Henri Toivonen and co-driver Sergio Cresto in the Rally of Corsica. Toivonen's last interview concluded with: "these cars are too fast for the human mind, if you lose concentration, you're dead" [paraphrased from Finnish], then he closed the car door, drove off and was never seen again.
This is how crazy it was: https://www.youtube.com/watch?v=eYIH7ApCxoM
I spent a couple minutes on the Wiki page and this little picture sort of sums up rally racing - they soberly resisted writing "flying" in the label that says "Toivonen driving the Manta 400".
http://en.wikipedia.org/wiki/Henri_Toivonen , http://upload.wikimedia.org/wikipedia/en/6/61/Henri_Toivonen...
By the way, a colleague of mine owned an Audi Quattro Sport back in the late 90's. It was a very complex car, many components were unique to this model and hand-built. Maintenance was difficult, very expensive and very frequent. He bought a Subaru Impreza WRX "for those 8 months of the year that the Audi spends in the workshop".