It takes longer to access elements that aren't in cache, but that's still a constant factor. It may be 100x or 1000x but once you're going all the way to main memory it's not going to get any worse as your array gets bigger.