It's mainly the media. While there was probably some far right groups/people who decided to go along first, the "association" of the OK sign with white supremacists was popularized by the medias