You say "no, dates are harder than that", but it sounds like you agree with me exactly based on the rest of your post. Time is hard because human representations of it are varied, complicated, and often arbitrary--not because there's anything fundamentally hard about the math of time (again, relativity notwithstanding). Contrast that with numbers which are inherently difficult to intuit about apart from issues of representation.