You'd guess wrong. We have record keeping around how most people use words, and according to those records "white collar" is used in this fashion of: "relating to the work done or those who work in an office or other professional environment."

And you confirm that they work in an office environment, so that bit isn't in question. Perhaps you are confusing most with all? It is certain that not everyone sees "white collar" that way, but we're not talking about everyone.