I'm in California now (and have been for over half my life, at this point), but I suspect I'd say "back east" when talking about my past, at the very least. For me, there's a sense of time involved, as well as direction.