The page tables change in a kernel context switch, whereas they don't in a user context switch. Depending on your CPU and OS, this may or may not matter; e.g., modern CPUs typically have caches that are tagged with the context (e.g. PCID on Intel), but the Spectre flavor of the day may require that you simply flush the entire L1 cache on context switch anyway if you're unlucky.

The biggest cost of a context switch, though, is typically that your CPU goes to do something else, which reduces the efficiency of any caches (including branch prediction etc.). This is the same whether you have kernel or userspace context switches, assuming the threads you're switching between touch different code and/or data.