I believe objects with destructors can keep an object alive for an extra collection phase, but I believe if that’s the case it can easily be solved with a using block before allocating the next object.
A System.Drawing.Image holds operating system resources (GDI+) and these resources are released when the Image instance is disposed. If the instance isn't explicitly disposed then the finalizer will do it but this only happens when the garbage collector collects the Image instance.
Allocating many Image instances without disposing them might exhaust the available resources (Windows bitmap handles or whatever) but the garbage collector doesn't see any memory pressure and does not perform any collection so the finalizer does not dispose Image instances that are no longer used.
The garbage collector has an API (GC.AddMemoryPressure) where an object that has unmanaged resources can signal that it's consuming additional memory to inform the garbage collector's decision of when to perform a collection.
It’s true that in that case it makes sense to «delete» an object. But in that case you can either use a using block or manually call Dispose, right?
For unmanaged object I am surprised it would keep it in memory for a bit since it is also the mean by which you release any lock on a file or a connection. If you don't execute it straight away, you potentially create bugs.
I think the reason why Microsoft was telling people not to call gc.collect is that it interferes with the optimisations and heuristics that the garbage collector maintains to optimise when to do a GC. But I must say I didn't notice any abnormal behaviour when I did. But I would only do if I absolutely have to.