You can't say that without saying how large the sample is. Certainly, if the random sample size is "100% of all ballots" it would be enough.
Risk Limiting Audits take that into account. There's a rigorously backed formula for determining how big the sample size needs to be based on the margin of victory. The closer the election, the more ballots are included in the sample.
In fact, you're right that a _very_ narrow margin requires a random sample that ends up being very nearly or actually all votes.
Georgia is a perfect example of this. Georgia applied a Risk Limiting Audit to the Presidential Election. Because the margin was so close, the RLA required them to sample every single ballot cast. The first "complete hand recount" of Georgia was actually a Risk Limiting Audit.
Here is the GA SOS reporting out the results of that first statewide hand recount: https://sos.ga.gov/index.php/elections/historic_first_statew...
As a made up example: maybe some precincts had scanners that worked 100% perfectly, and some had units that occasionally mis-scanned with a rhythm.
Georgia was the closest percentagewise in 2020. 4935487 votes were cast between Trump and Biden, and the difference was 11779 votes. However, even if Georgia went to Trump, Biden still wins, so you're forced to check states with even larger gaps.
However, even in Georgia, to claim the election should have gone the other way with 99% certainty, it is not hard to check wherever you want. This was done by many groups, for example here [1]. Of course, given the sheer idiocy of Trump followers and news support, they don't educate their supporters on any of this, because the facts destroy their claims.
Since voter records are public in pretty much every state (every one I've looked at over the past few elections let you download them), it's trivial for researchers to do this sampling. They do this every election, and that is how they make the statements they do about the extremely low rates of fraud - they did do the testing.
Trump and co did it too, and found nothing. They had the lists immediately, because they started publishing names they claimed were fraudulent, and in each case the media showed them wrong. So they were looking at the names.
And even with those names, and the ability to sample, they still didn't put forth any solid evidence. Sampling irregularities would have been enough to get more eyes looking. But they knew (just like Trump knew in 2016 with his Clinton 3M illegal votes charade that he eventually dropped) that this is a lie.
[1] https://sos.ga.gov/index.php/elections/3rd_strike_against_vo...