The 'referers' are in Consider Phlebas, a small group of humans among trillions who are able to reliably predict the future better than Minds. This would be one argument in Gurgeh's favour, but Banks later admitted how flimsy the idea was.