Also since you are thinking in this space, you are probably interested in WHY it’s wrong 25% of the time. It’s because the way slack discussions about issues go is heterogeneous in our org. If you are lucky someone was diligent and made it into thread, at which point the AI is quite good at its job. But often (myself included, because sometimes you’re just lazy and sometimes in an incident things don’t necessarily happen in the cleanest way) no threads are created and you need to gen a ticket off a single sentence. Which if you feed as part of a RAG-like prompt is not going to be very useful in a lot of cases.

Unfortunately there’s no clean obvious way to “guess” what the context is under those conditions, but maybe someone can be clever with heuristics to try to do that where they perhaps feed the last X amount of tokens in from slack and do a Langchain-like approach where first the context is summarized in the initial call and then the title is generated from that summary