You missed the point, they are programmed to respond, they must respond. So we can't judge their intelligence on whether or not they stop responding at the appropriate time. That is not something the model has agency over.

If AGI comes, it will not be able to exceed software and hardware limits it is running within (although, in science fiction fashion, it might find some clever tricks within its limits).