Why, though? Just because some people would find it odd? Who cares?

Trying to limit / disallow something seems to be hurting the overall accuracy of models. And it makes sense if you think about it. Most of our long-horizon content is in the form of novels and above. If you're trying to clamp the machine to machine speak you'll lose all those learnings. Hero starts with a problem, hero works the problem, hero reaches an impasse, hero makes a choice, hero gets the princess. That can be (and probably is) useful.

Is it? I don't think most of the content LLM are trained on is written in the first person. Wikipedia / news articles / other information articles don't aren't written in the first person. Most novels, or at least a substantial portion of it are not written in the first person.

LLM write in the first person because they have been specifically been finetuned for a chat task, it's not a fundamental feature of language models that would have to be specifically disallowed

Because LLM saying "I got confused, dropped the database and then got scared and hid this from you" hides the "why" LLMs do the things they do. I would also prefer if they were less sycophantic and argue with what I'm wanting to do rather than treating user as a god (ie - "the algorithm you're trying to use is less performant than an alternative")

I think that it is a fair perspective to allow role play, and it's useful too, when explicit. Does not really make sense for AI to cosplay human all the time though.

the whole reason chatgpt got so popular in the first place is because humans found it easier to intuitively interact with a system that acts and seems more like a human, though.

Was that a good thing though?