Buddhists are not "people of the book" because the Buddha was not a God, and didn't have prophetic access to the teachings of a God. His views on karma and rebirth, for example, were those of the society he sprang from; they were not the result of transcendent insight. He was not some kind of perfect being.

Buddha became more God-like as the centuries passed; some Prajnaparamita and later texts describe him as being the height of seven palm trees, for example. But he's never been considered infallible, like a prophet.

> they were not the result of transcendent insight

That's exactly his selling point, that through deep meditation he had profound insights, regarding impermanance and no-self. But yeah, that was his own realisation, not just some words some god said to him that are supposed to be infallible.