Buddhists are not "people of the book" because the Buddha was not a God, and didn't have prophetic access to the teachings of a God. His views on karma and rebirth, for example, were those of the society he sprang from; they were not the result of transcendent insight. He was not some kind of perfect being.

Buddha became more God-like as the centuries passed; some Prajnaparamita and later texts describe him as being the height of seven palm trees, for example. But he's never been considered infallible, like a prophet.