You can hide a lot behind "presumably mostly"!

My memories aren't entirely internally consistent. Memories fade over time, too, so even if they formed perfectly, I certainly misremember many events. The earliest ones are from when I was a kid, when I didn't understand the world, so I naturally place the least faith in their accuracy.

When I do have conflicting memories, I use logic and common sense to try to figure out which is correct, and hedge my bets against memories which might be false.

So if I were an intelligent brain formed with random memories, over time my natural tendency, in order to remain sane, would be to correct my memories to be internally consistent, as much as needed, and not examine the rest too closely.

That conclusion isn’t quite logically rigorous. You need to have a prior on the odds of being a Boltzmann brain in the first place. If that prior is extremely high, your conclusion won’t hold.