It's more the other way around: having non-objects allows things to change.
Being able to transparently bridge between Objective-C and Swift is very valuable, particularly because the vast majority of library/framework code is written in Objective-C.
So Swift Arrays are designed to be bridgeable to NSArray whenever possible, for example by simply pointing to an NSArray as their backing store. NSArrays can be non-contiguous, largely due to them not really being arrays in the strict sense, but more like Smalltalk OrderedCollections, meaning you can, for example, add and remove elements from the front.
However, non-object Swift types cannot be stored in an NSArray and would thus never be expected in Objective-C code. Therefore, there is no need to bridge those, meaning that you can use different kinds of backing store.
Why would elements being classes change this?
Wouldn't an array of class elements only hold pointers to those instances, so it wouldn't be different than "an array of Swift value types" at that regard (the extra dereference would only be needed after getting the element pointer out of the array).