Prior to that change, lying in the clearance process was the one thing that was absolutely disqualifying. As you noted candidates are incentivized to lie, and so "did you lie here, where it is likely to benefit you?" becomes an effective screening mechanism for people who are willing to compromise their ethics for personal gain.
They don't care if you're gay, but they care if you're closeted. They don't care if you do drugs, but they do care if it's such a problem you could become financially beholden to someone over it.
More accurately, they don’t necessarily care if you did drugs in the past. Current drug use, or very recent drug use is a risk, as addiction is a huge risk for both judgement and susceptibility to blackmail, and even controlled drug use is illegal. Also, being a previous drug addict extremely risky drug use can speak to your current judgement.
Heres the DCSA clearance appeals page showing people getting or keeping their clearance from drugs and other issues: https://doha.ogc.osd.mil/Industrial-Security-Program/Industr...
If you have connections to a hostile country - family, friends, business, spouse, etc. they can (will) become targets of intel ops.
So your wife is from Russia? GRU or FSB will start to pressure family and friends of your spouse.
Anything that is worth anything to the enemies of your country, they will find a way. Which is why agencies that require top secret security clearance will rather just set the threshold extra high, and lose out on potentially good talent. Better to be safe than sorry.
- Drug use
- Financial crimes
- Close ties to hostile countries (China, Russia, Pakistan, Iran, North-Korea, etc.)
And at least in my country, it's not the intelligence agencies themselves that handle the security clearance, but rather a dedicated agency/authority that processes all the security clearances in the country.
Now, if you've never been arrested / charged / convicted on the two first points - who would know? I'm 100% some candidates would simply lie.