You should see it from the other side: a cache miss takes longer the bigger the data you have to fit in your cache (as you go through bigger and slower cache levels).

So, as your N (length of data) changes, access time changes too.