There's been quite a few different iterations of ChatGPT and bing with different behaviours in this regard: it depends somewhat on the base GPT version, the fine-tuning, and the prompt. Bing very famously at one point was extremely passive aggressive when challenged on basically anything.

And while there's nothing intrinsic to the structure and training goals of LLMs which directs them towards more structured reasoning, it does seem that in the process of learning to predict language they pick up some degree of it, however flawed it may be.