With the more recent European emission rules, and the massive "dieselgate" scandal of course, there is simply far less appetite for further diesel innovation nowadays and in the future. Pump availability at the gas station of Diesel itself is also now pretty poor in North America vs Europe too. VW/Audi also had a great reputation for refined Common Rail diesel engines in this period as well, ironic given dieselgate.
The common rail stuff got so refined Audi even offered convertible A4s with diesel for the first time, diesel cabrios were largely unthinkable before due to lack of engine refinement. Generally these engines were still noticeably diesel sounding at cold idle, but once up to temp after 5 minutes it got a lot harder to tell unless in the cabin looking at the low rev limit on the dials.
Pretty good discussion here as well (cool didn't even know there _was_ mechanics stackexchange): https://mechanics.stackexchange.com/questions/26513/whats-th...
Edit: would assume 4-cyl pistons need to actuate/fire ~twice as much as an 8-cyl engine at equivalent rpms? maybe that would explain smaller engines being more clicky idk
While there is some noise from the injectors (in some gas-direct injection engines you can hear them as well, if you listen carefully) the sound you are hearing in an old diesel, that handful-of-marbles-in-blender-clatter, is due to the more abrupt combustion in a diesel engine in general (only air is compressed, and the fuel burns almost instantly on injection) and imprecise/poor timing of the old mechanical injection systems. Modern common-rail electronic-injected diesels are almost as quiet as gas engines.