I think you could implement the entire thing in CUE, minus moving from/to python. We're already going what the last 3 points instruct the user to do for integration. We are using Go rather than Python, so the CUE to language types has better integrations. Of course, there is the prompt engineering side, but that is just text with instructions for the LLM that need to be well crafted.

One thing we have seen is that you need to adjust your prompts when OpenAI updates their model. Given they only support their dated models for so long, it seems increasingly difficult to make the case to build on top of LLMs you cannot control the life cycle for.