Pins on the ephemeral segment are generally bad in that the quantum allocator has to be aware of them and squeeze objects between them.
The GC is not permitted to eagerly move pinned objects out of the heap. This is because there are two ways an object can be pinned: a pinning GC handle or a stack scan reports a local as pinned (e.g. the "fixed" keyword in C#). The GC does not know until a GC is already in progress that an object has been pinned and, at that point, it's not legal to move the object so it must stay where it is at the current point in time.
Typically this isn't a big problem - pinned data structures in .NET code are either pinned for short periods of time (to pass to native code), or are reusable large big buffers that stay pinned forever. Large buffers are always allocated in the large object heap right away. You can always allocate native memory directly in which case the GC doesn't care about it.
This may be changing since recent updates to C# and the runtime have introduced the concept of interior pointers to objects, where you can have a raw pointer to a field within a GCable object. Right now those are constrained to living on the stack only, so the period of time in which the object can't be moved/compacted as a result is relatively short.
GCHandle is a struct so you have to explicitly call GCHandle.Free().