It has been a long time since I studied functional analysis, and I was never a physicist of any sort, but my limited understanding is that there are two common generalizations for the infinite case, both of which handle the cases of most interest to physics but are not available for general operators, and which mostly agree up to a constant factor, or are at least capable of producing compatible results in some sense. I'm not sure I'd say they "nicely generalize", though I'd admit to being hasty in saying that they don't generalize.
And yes, I agree that they're widely used in physics (which is why I think they should be taught to math/physics majors).
In which bit of math do people actually instantiate an inverse?
I am assuming you are being rhetorical here. I also think your working hypothesis is that the only use of a matrix inverse is to solve a linear system of equations. Under this hypothesis it does make sense that there is no utility to computing inverses. However, this hypothesis is incorrect. There are tons of cases where you would want to get at the entries of an inverse, but has nothing to do with solving equations.
Here area few examples: consider data generated from a multivariate Gaussians. Now you want to know which components are independent of each other. This is given by whether the inverse of the correlation matrix is zero or not. Another example, say you want to minimize a function defined over matrices, but want to minimize it only over the set of positive semi-definite matrices. One way of doing these is to use a barrier function: a function that blows up astronomically when you approach the boundary of your constraint set. For positive definite matrices such a barrier function is the logarithm of the determinant. I could go on...
In symbolic computations with very structured matrices, one sometimes constructs explicit inverses. This is an extreme corner case, and never really comes up except in some very simple physical models (in my experience).
I'm a big fan of Axler's(1) approach of simply avoiding determinants entirely as much as possible, though his text isn't really suitable for a general course. An approach that combines elements of Axler's and factorizations that are actually used in computation (schur form, the SVD, ...) can be nice, and is more useful to non-math/physics majors than the standard curriculum.
(1) "linear algebra done right"