That might've been your personal experience. But the Defense Language Institute, which has trained thousands of people over the decades, gives both Chinese and Japanese (along with Arabic and Korean) a level "4" rating - the highest one, requiring (IIRC) 2200 hours of practice. A large part of it depends on the individual - if you find grammar easy to pick up, then Japanese will be easier. The opposite will be true if you find pronunciation easier.