Maybe the added difficulty comes from needing to understand the grammar (particles) and the different forms of speech for different levels of politeness (after a few years, it now weirds me out when a white dude calls himself 'watashi').

Edit: Actually, if the methodology is "average time spent to learn" another possibility is that the average is slower because the barrier to learning the language is just low enough to attract a bunch of slow learners that wouldn't be interested in Chinese or Korean. (Reminds me of a study where decreasing page weight in KBs increased average page load time because users on the slowest connections were now able to load the page and skewed the average upward.)