It sounds like you are assuming the joint distribution of returns in the future is equal to that of the past, and assuming away potential time dependence.

These may be valid assumptions, but even if they are, "sample size" is always relative to between-sample unit variance, and that variance can be quite large for financial data. In some cases even infinite!

Regarding relativity of sample size, see e.g. this upcoming article: https://two-wrongs.com/sample-unit-engineering