Well, the problem is when you have a []interface{} that contains a lot of float64 mixed with other things. If the size of the slice is large, I don't think you can assume it's all cached.
If the slice is large, it's less likely to be cached either way.
True, but there is also locality of reference (having everything in a contiguous block uses less cache than pointer-chasing) and garbage collection pressure.
In principle, using NaN boxing or a similar technique to store non-floats, you should be able to store floats in an []interface{} as efficiently as in a []float64. Scripting languages do this, but not Go.
If it may not be a pointer you have to branch, which can be add some overhead to the lucky path (every pointer deref will have to wait for type comparison in machine code)
If the float has just been written, the data should be in cache so the performance penalty is probably not too bad in most cases.