Not only are you not alone, that's both the default view of many (most?) programmers and the stated claim of many closeted racists who want to detract from incremental steps by referencing orthogonal issues and derailing conversations.