What do you mean by usually? What do the "bad" cases look like and how often do they happen? How are you measuring that key access and is it happening across cores? Are you setting any OS level settings around isolation?