(cf. https://en.wikipedia.org/wiki/Shell_theorem for some proofs)
In the case of a charged conductive sphere, the charges are free to move. With no charge present other than that on the sphere they will arrange themselves spherically symmetrically.
But if there is other charge near the sphere (outside or in), shouldn't that cause some rearrangement of the charges on the sphere, breaking in most cases the spherical symmetry?
Also, one of Maxwell's equations (in integral form) says that the total electric field passing through a closed surface is proportional to the amount of charge within the surface. If you consider a spherical surface just inside a uniformly charged sphere, with no charge inside or outside the sphere, there can be no flux passing through this surface (since the total flux must be zero and the field is obviously symmetrical). In fact this is true for any closed surface inside such a sphere.
Here is an example using gravity but electrostatic forces should operate the same way. https://www.grc.nasa.gov/www/k-12/Numbers/Math/Mathematical_...
You can only feel forces if it's some other shape.