But those Arxiv papers which were not published elsewhere but which got many citations, those were read by others, i.e. reviewed by peers, i.e. peer-reviewed.

Arxiv only lacks the initial quality filter by peer review.

I'm also working in the field of machine learning. In those niche fields I work more specifically (speech recognition), I can usually still get a lot out of Arxiv-only papers. I can pretty easily see the main idea and see if there is some usefulness in the paper or not w.r.t. my own research e.g. by good experimental analysis. In don't really feel overwhelmed in the amount of papers. I don't really see the problem.