I don't think anyone is trying to make the case that a single stylistic preference impacts readability, but in aggregate many diverging preferences definitely can.
Add in programming styles and a flexible language like Ruby can look like entirely different languages from one dev to the next (Ruby is a very traditional OOP capable language, or a weird but capable lisp, mix the two and readability suffers in new and exciting ways).
I don't understand this argument. Are people really having trouble reading code because it has two different quote characters? This goes against my understanding of "readability"? What is it that is really meant by this?