No, LTE devices will happily connect to any random DIY base station, without any visual indication that this is taking place...

Maybe with GSM, not with LTE.

LTE handsets connect to any base station, at least on physical layer and management protocols. This is required for mutual authentication.

Some attacks abuse error indication prior authentication for location leaks and denial of service.

Once the connection is established, you're good.

How? Which keys do they trust?

LTE uses symmetric crypto with a pre-shared key stored in your SIM-card and in a database of your provider.

Base stations receive a derivated key. If the base station is not connected to your provider's network, you cannot establish a connection due to the lack of a common key.