The victim code doesn't need to have some explicit pointer arithmetic, it just has to have some sequence of bytes, somewhere in its address space (the "gadget"), that can be used to read a memory address based on a value stored in a register that can be affected by input supplied from the hostile process. The branch prediction is used to speculatively execute that code. The "Example Implementation on Windows" section in the Spectre paper goes into more detail about this.