Besides, I seriously doubt that there's much gold in unambiguously recognizing stack pointers. I try to put as much as possible on the stack, but it just doesn't seem like much. The more stack allocation you manage to do, the more code bloat gets generated. Any malloc'd pointer will look just like a GC pointer, so you'll be paying the penalty for all of those.
How these tradeoffs play out will be difficult to determine in advance. If you've the confidence that it will work out favorably, by all means, give it a try.
Of course, one could add user annotations, but annotations are generally disliked.
I don't see why. A small amount of manual tagging for extern(C) things—done in this case as part of the standard library, so opaque to user code—will go a long way in that regard. And in any custom allocator, automatic inference should work just fine, since the data returned will be clearly derived from the already-tagged malloc/mmap/VirtuallAlloc/whatever (type info does still have to be associated manually, of course, as part of the GC.addRange dance, to avoid spurious pinning).
You'd also have to keep around a massive amount of data at compile time. A variable that's a pointer to a struct, can't just be represented as a pointer to a struct. It'd have to have a map of which pointers in the struct are managed for every pointer to that struct.
The way the D compiler is implemented now, a struct S has only one instance, as every pointer to a struct's type points to that same instance. And still sometimes people compile programs so large the compiler runs out of memory.
You should revisit that corner of the thread to revive your memory. The discussion was about adding an extra 1% performance penalty for reasons, not the importance of a 1% delta either way.