I think your gut feeling needs some serious recalibration. Please consider "the Hilbert matrix" (http://blogs.mathworks.com/cleve/2013/02/02/hilbert-matrices...) and more generally condition numbers (https://en.wikipedia.org/wiki/Condition_number), and in regards to "computationally efficient differential inversions based on small changes to the matrix", consider what happens to a polynomial's roots when you make small changes in the coefficients.