By the way the last official answer I got from OpenAI regarding the markdown issue was that they thought it wouldn't be a problem because there will be mitigations. My assumption now is that they thought they could maybe fix indirect prompt injection, but it doesn't look like there is a solution.
My current takeaway is that with LLMs, the assumption always needs to be that the LLM is capable of outputting any content, and the client needs to safely (and securely) handle whatever is returned.
Btw. I show cased these kind of image based prompt injections in July (with Bard and Bing Chat, which presumably already used OpenAI's tech), and the example I created (the robot slipping on a banana) was shown during Blackhat in Las Vegas this year. Tweets: https://twitter.com/wunderwuzzi23/status/1681520761146834946
Even if development were to freeze, they didn't improve at all from this day onwards, and their many monumental flaws (prompt injection, hallucination, inability to reason etc) were never solved, I still think they'd be worth studying and using.
I expect we could still spend years figuring out new capabilities in the models we already have access to today.