That said, just asking people with apparently inappropriate dogs what task the animal has been trained to perform would be a first step towards removing fake service animals. If they can't answer, I assume there is something you can do, but I don't know what that is exactly. It may depend on your state.
In the case of a specifically badly behaving fake (or even real) service animals, section b of the same document says: "A public entity may ask an individual with a disability to remove a service animal from the premises if - (1) The animal is out of control and the animal's handler does not take effective action to control it; or (2) The animal is not housebroken.", so a dog barking or pooping in a business can definitely be ordered removed.
"Public entity means -
(1) Any State or local government;
(2) Any department, agency, special purpose district, or other instrumentality of a State or States or local government; and
(3) The National Railroad Passenger Corporation, and any commuter authority (as defined in section 103(8) of the Rail Passenger Service Act)."
I believe the ADA is https://www.law.cornell.edu/uscode/text/42/chapter-126
[1]Businesses may ask if an animal is a service animal or ask what tasks the animal has been trained to perform, but cannot require special ID cards for the animal or ask about the person's disability.
I find it mildly annoying to see dogs in restaurants and grocery stores, even more so in the work place, but I've heard of a lot of people abusing these laws here in LA to be able to get into apartment buildings that don't allow pets.
As a neighbor, if you're animal isn't well-trained, I don't want to live near it in an apartment building. Were I someone allergic to pet dander, I wouldn't want to unknowingly live somewhere that a dog/cat had lived for years no matter how well you cleaned it. Especially at the expense we have to pay for rent in this city.
I think taejo has it right in her/his comment about how US owners act about this stuff:
"I don't think this would fly in the US, where cultural attitudes tend to be less "we've all got to do our bit to make society pleasant for everyone" and more "it's a free country and this is my dog, I'll train it if I want to"."
This seems to be true. About a week ago I witnessed my boss question someone on if their dog was a service animal. He was adamant that it was but his demeanor betrayed himself.
I never saw someone get so maliciously defensive before, the person had to be told to leave because of his attitude from that point on. He actually called back and threatened my boss (on voicemail!) some time after leaving.
That one simple question blew him out of the water. Dog seemed nice despite his owner, but it didn't seem to be doing anything but dogging about.