It's true that you need to be an expert in the field to tell the crackpots from the experts, but if you don't understand enough of the paper to tell whether it is realistic or not then you probably will not gain very much from reading it.
I discovered ArXiv over a decade ago, but from early on, it was filled with people of questionable insight, if not sanity, and I began to avoid it. There are some notorious USENET trolls who used it to build their credibility, and it kept me away since.
But it's a much better academic tool than a commercial one, and I would not be surprised that a lot of cranks put stuff up there and claim respectability by association, especially in theory or mathematics.
The entire string theory community publishes all their papers on hep-th before sending sending them to journals (which nobody reads as everything is on ArXiv). Of course, maybe you will argue that string theory is crackpot physics...
I spend most of my non-work hours there and it's horribly broken. Case in point, I just went there now to tally the most visible aspects of its brokenness, turns out it's down, again.
Yeah, CiteSeer is horribly broken. ArXiv is mostly for crackpot physics and has a tiny CS/EE index. What else? I refuse to pay ACM, Springer, or IEEE.
Google Scholar just .. looks "search enginey" and doesn't have the summary bar with citations, years, etc. Also, it heavily links to the above "resources" that I refuse to pay for.
CiteULike is just as lame as CiteSeer in terms of UI, but lacks the info-bar on the side, and adds crappy social elements.
CiteSeer gets the caching right though, which is why I keep going back to it.