The most sensitive tests have a limit of detection of a couple hundred copies/ml, and the collection kits usually use 1ml of VTM or equivalent. I find it hard to believe that a significant number of people will have enough viral RNA in their nose to put hundreds of copies onto a swab unless it's their own body that's producing those copies.
This has been confirmed in studies. Researchers have taken samples that are PCR positive at various cycle counts, and tried to culture viruses from them. Here's one:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7427302/
"RT-PCR cycle threshold (Ct) values correlate strongly with cultivable virus. Probability of culturing virus declines to 8% in samples with Ct > 35 and to 6% 10 days after onset; it is similar in asymptomatic and symptomatic persons."
(Note: this is not the only one of these papers that has been published. This is just the first one I found with a quick search.)
PCR determines how many exponential amplifications are necessary to get a detectable amount of the target RNA sequences.
The amount of that target RNA floating around in the environment does slightly impact the base amount present in even covid-negative patients.