Many of the diesel vehicles are public buses, and these can be replaced pretty quickly with electric or natural gas (or hybrid) buses. Others are large delivery vehicles, and with some "encouragement" (restrictions) the owners either replace them, or rearrange their fleet to use their newest vehicles in cities.
For example, London has done both, and is already seeing improvements [1]. I noticed when I visited over New Year that the air seemed fresher, so I'm glad I've found some statistics to show that -- it wasn't just a one-off.
[1] https://www.london.gov.uk/sites/default/files/ulez_six_month...
It has improved slightly in the last 2 years but it will take a full adoption of EVs (and increased public transit) to resolve. It's not like France is lacking for fuel taxes or road taxes and insurance costs. Demand is still too high.
I haven't checked to see if this has changed since then.