I don't see any evidence either that most people believe that or that it's true.
I personally find it more likely that high school grades and extracurriculars are probably the most biased portions (grades because a large portion of them usually depends on assignments with loose grading rubrics, which are empirically linked to greater racial and other bias for otherwise similar responses, and extracurriculars because both access and evaluation of them is impacted by cultural factors of both the student, the evaluators, and other members of society.
Determined by who?
UC Berkeley Chancellor Carol T. Christ, along with the UC system’s chief academic officer, Provost Michael Brown, said Friday that research has convinced them that performance on the tests is so strongly influenced by family income, parents’ education and race...
> UC should keep SAT and ACT as admission requirements, faculty report says
> University of California faculty leaders are recommending the continued use of the controversial SAT and ACT as an admission requirement for now, citing UC data showing the standardized tests may actually help boost enrollment of disadvantaged students, according to a highly anticipated report released Monday
https://www.latimes.com/california/story/2020-02-03/uc-shoul...
"The new yearlong faculty review found evidence that most UC admissions officers offset much of the bias against disadvantaged students by evaluating standardized test scores in the context of their high schools and neighborhoods."
"Among students with SAT scores of 1000 — the 40th percentile — half of Latinos were admitted compared to less than one-third of whites."
I've heard the claim that the SAT is biased against lower SES for a while now, but more on the left than the right, and I've never heard "most" biased.