I think one thing you can try is to figure out what lies where, so chunking arbitrarily will not work as well as chunking with headings for e.g

For e.g, if the question is: "What should the opposition leader have mentioned in their response that the minister would have found difficult to answer."

An embedding based search will find it fairly difficult to match this against a text. Based on my experience, you have to figure out what is a nonanswer first (i don't think that's easy but gpt4 is very good at a lot of language based stuff.)

you can try Q1: question A1: Answer then prompt GPT, do you think A1 answers the question and then save it. And then, Q1: question, A1: Answer, Q2: Follow up based on the following questions, do you think A1 answers Q1 and then save it to a db.

You can then augment it in the code with our own knowledge of how politicians lie, using certain words etc :) to improve what gpt4 might miss...

> "We're definitely moving forward with strong measures to improve the situation, and while progress might not be immediately visible, we're fully engaged in this essential journey."

This is how GPT4 answers the question... I assumed the question, but presumably will be along those lines :)

https://chat.openai.com/share/c736abf4-ae3c-4fbd-9427-b7d2f9...

Man, it seems the people writing politicians, or managers, statements seem at serious risk being replaced with a LLM...

Could you ask an LLM if they were persuaded by a speaker of one side of a debate as a method of evaluation? Ie the bot's before and after opinion based on fine-tuning with the pro and con arguments?

I was also thinking about a society of bots type application where you could have autonomous bot researchers, debaters, judges and audience. Would be interesting to feed in the topics and grab some popcorn

I think you could ask an LLM if they were persuaded, but I don't think you'd get meaningful data from it

Leaving aside the bots that are trained to answer "as an LLM I do not have opinions..." it's going to be a very basic probabilistic yes or no based on tone and numbers of pros and cons rather than knowledge of the surrounding political context and higher order reasoning about the accuracy of the claimed pros and cons.

Yeah the bot has both no initial position and no internal opinion on fact that an argument could operate on. It might write you a cogent response but it's fundamentally not a question an LLM can meaningfully answer.

Relevant useful dataset: https://paperswithcode.com/dataset/debatesum

I like this question. I've always wanted to be able to compute, somehow, the "partial derivative of x with respect to y," where x is a proposition and y is an argument or a piece of evidence. It seems to me as though we're closer than ever before. The models are still fairly opaque, but I'm hoping the interpretability research will succeed and allow such introspection!

If you tell the LLM it is conservative it will report it was persuaded by the conservative speaker, if you tell it it is a liberal it will say it's persuaded by the liberal speaker. It will be happy to roleplay at being persuaded but I don't know why that would be useful?