That's how I figure it works in my head.
My earliest memory is a vague memory when I was in a crib. Based on certain specific details of that memory my parents place it to when I was around 2.
I remember vague details of the homes we lived in after I was 3 or 4, but not the first home we lived in at all.
Apparently I had a very bad fall at some point and put one of my new teeth through my lip and made a bloody mess everywhere. I have a scar there to this day, but don't remember the event in the least.
I remember my mother teaching me the alphabet with a foam alphabet mat [1] and how to tie my shoes when I was probably 3 or 4. I also remember the moment my mother taught me the difference between how to say "three" and "tree" and permanently corrected what would have been a difficult speech impediment.
I remember my Kindergarten teacher's name and my favorite toys from when I was 4 and 5.
But the number and quantity of my childhood memories explodes after I entered 1st grade at 6. I feel like I remember more about 1st and 2nd grade then I do about most of the rest of my elementary and middle school education combined. Some really detailed events even.
When I was 3 or 4 I broke my nose playing on a playground. I remember that well. And I remember things from kindergarten and a lot of things from grade one.
A large majority (including me) are completely blank about anything before age 6 or so.
I know tough that before age 25 all memories where real, but now getting older the memories are becoming memories of memories, and it saddens me a lot.
I was a pretty introspective kid. I thought about my past and tried to recall my childhood even at the young age of 5. So perhaps my earliest memories are really just memories of being 6 or 7 and thinking back to when I was 2. Second hand memories, so to speak.