It's 5120 bytes for the both of them, the access is not at all random (due to color popularity), and the real cache killer is the image itself. You're dealing with a few megabytes for the image.

Oh, the lookup tables are cached just fine. It's just that they crowd out something else.

That "something else" is already crowded out by running through a multi-megabyte image.

At least the instruction cache is barely affected, which would not be the case for a math library function.

You're not running a multi MB image but a small block thereof. Not that 5-6KiB LUT matters, but the extra conditional multiply ops or power or logarithm function evaluations do. Latter are need a few MACs for a good approximation (at least 12 bit precise, preferably 16 bit - like 5th order polynomial), former have masks, worse for SIMD.