I think the primary factor that causes this perception is the lack of any proper dialect continuum between English and another language. There are of course a few minor ones like English<->Scots and English<->Jamaican Patois, but nothing significant that any sizeable number of speakers would encounter.

Dialect continuums between languages are basically the norm around the world. You can even connect Sicilian to Portuguese for example.

I would assume the primary reason English isn't part of any major one is because Britain is an island, otherwise English would probably connect to Dutch and join the West Germanic continuum. It would then appear just as fractured as any other language.