I've often heard that hysteria, but it seems overblown in my experience in the Washington, DC area. Yes, you will likely see a homeless person or two when you go to the library, especially when the weather is bad, but they certainly haven't "taken over" the library or generally bother anyone using the library for its intended purpose.