I still don't see what the problem is - you know the type of memory array you're referencing into (int, byte, whatever), you know the type of pointer you're dereferencing (byte). So is the problem going to be that it will be slow? In portable CL, yes, but there's always ways to get at the implementation's array-accessing internals and fake type-displaced arrays (esp. if you have control over the internals, as this hypothetical project would).

Well, that's a good point. In these days of high-quality open-source implementations, games can be played under the covers that would be impossible in portable CL.