Is your hard drive almost full? I've found that when drive space is running very low, OSX deletes some of Spotlight's cache, probably to make sure there's space for hibernation and swap. When it gets back enough space, it rebuilds the cache again.

The problem is that if you are constantly orbiting a certain point of fullness, it goes crazy and keeps deleting and rebuilding all the time, causing slowdowns.