Most GC implementations that I can think of know which bits in memory are pointers and which aren't, and only scan the pointers.