Sure, it means you assume that during the signal detection everything was working exactly as it was during background data collection. There are going to be other implementation specific assumptions like what thresholds to use regarding environment noise triggers, accounting for sensor drift, etc. These data-driven models are approximations, I doubt the people actually coming up with them really believe they are 100% true. If you give a real life example I will point out exactly where there are issues.
> "If you are using the SM as your background, then yes you do believe it to be 100% true"
I don't know what this is supposed to mean. Of course you are assuming your background is 100% true. Do particle physicists actually believe this though? Not from my reading:
"So although the Standard Model accurately describes the phenomena within its domain, it is still incomplete." https://home.cern/about/physics/standard-model
As for the SM, we are looking for those places where it is incomplete, and to do this we must assume it is complete and look for strong evidence that we are wrong. To date, it doesn't seem like we see though, as far as collider physics is concerned.
1. If you're using a data driven background, you're background model isn't the Standard Model but that there isn't something special about your signal region.
2. If you are using the SM as your background, then yes you do believe it to be 100% true, since what you are looking for is evidence that there exists new physics not described by the SM.
In other words, the theory under test is the SM, and it can pass very easily. If it fails despite this, that's evidence that it's incomplete. If it passes, the limit setting stage tests whatever new physics we were looking for, and gives an upper bound on how sure we are that it does not exist.