The correct setting is a rigged Hilbert space: given an algebra of operators A on a Hilbert space H, let S be the maximal subspace of H such that |sa| is finite for any s in S, a in A. These are your states. Operators in A don't necessarily have eigenvectors in H, but they do have eigenvectors in the space S* of all continuous linear functionals on S. So <x|, for instance, is just the map `psi -> delta_x(psi)`.
This has the advantage of not having any funky "rigged" states suddenly appearing in your calculations and is also exactly how we deal with non-projective measurements in finite dimensional quantum mechanics.
See here, for example
The spectral theorem, rather than decomposing X in terms of a sum of eigenvectors & eigenvalues instead decomposes it as an integral over the spectrum with respect to the (spectral) projection-valued measure.
Now it is fair to question whether this "observable" is really observable, but it certainly works out mathematically consistently in the normal way we do things in quantum mechanics.