A related problem is word alignment; getting this right is important for passing data from Lisp to vector processors (and to a GPU I presume, but I haven't done that).
The other issue is GC: When the C function is running, it's important that Lisp's garbage collector not move the memory C is using. Some Lisp implementations do the laziest thing and just stop Lisp until C returns. Others are more sophisticated and keep C's heap separate -- but that requires more copying. Still others mark the memory C uses in a way that tells the GC "don't touch" but that can lead to pathological fragmentation in extreme cases.
The above makes it sound worse than it is. 99% of my use of a Lisp FFI has resulted in a performance increase because C is in general about twice as fast as CL for low-level stuff--which is one of the reasons to use an FFI in the first place.