Also thinking about ways to ensure the bot answers common questions correctly while still being able to personalize responses. Working on something called "answer shaping" where an admin can write out a response and tag with the question it responds to. Then the bot would first check to see if the human question matches a cached question, and if so would prioritize using info in the cached answer in its response. Seems like this can give the bot freedom to personalize the answer but make sure it includes the right stuff.
I understand why you want the flow to work this way (single source of truth, fix things at the root, simplifies everything), but, respectfully, it is really bad from a UX perspective. Here are the main reasons:
1. Not all admins will have the ability to edit all source pages. Both from a permissions perspective (eg, a zendesk ticket or slack message created by someone else) or a technical ability perspective (eg, you need to edit html and create a PR).
2. People are busy and lazy. If I can see the problem in the answer, notice it's wrong, and correct it right now on the page where I see it, I will. Otherwise, I often won't. Think busy CS agents, developers in the midst of problem solving, etc.
Yes, supporting this workflow makes life harder on you, because it's technically more complex, but it's the way people will want to use this product.
That said maybe there's room to store bot-specific stuff in CB. For example, tagging passages with "exclude this from training data" if they're causing bad answers for some reason.
From a pure engineering perspective, it is obviously the wrong solution, and your initial suggestion is the right one.
Yet....
I strongly predict that the forces of market demand and human behavior will push your product inexorably in this direction.
How do you handle curation? Meaning... if the model picks up some out of date info or misinterprets it, and a human admin notices that and wants to mark something as out of date or wrong, can they? I see this as similar to the way I can correct ChatGPT over the course of a chat session, and it will remember the corrections.