ChatGPT never gave ELIZA a fair shot since only a fraction of the responses ever made it through.

The LLM probably doesn't have enough sample texts where the response is requested to be character limited to have generalised that as a constraint.

Ya, you'd have to adjust your initial prompt to GPT to remove as much prose and limit responses to be as short and terse as possible.

Add to that GPT doesn't count characters, but instead tokens and it's difficult.