News for hackers. Stuff that matters.
Re:AI Prompt Engineering Is Dead by msp26 on Wednesday March 06, 2024 @11:13AM (#39617431)
Tokenization errors are also really, really annoying to deal with.

e.g. why does the JSON output have silly whitespace/quotation sometimes? Obviously it's because the first token the model output was `{` and not `{"` like it should be. Obviously.

Re:AI Prompt Engineering Is Dead by binarymax on Wednesday March 06, 2024 @11:27AM (#39617585)
Totally. When using GPT I exclusively use function calling, which is far more reliable for JSON output. When using other models I don't even bother with strict JSON because of the error rate - and instead opt for HTML which is more forgiving when parsing.
Re:AI Prompt Engineering Is Dead by msp26 on Wednesday March 06, 2024 @11:49AM (#39617850)
nono, the same thing happens with OpenAI's JSON mode and function calling. It does output parsable JSON very reliably but it comes out mangled with a bunch of whitespace sometimes.

GPT-4-turbo's output context window is limited to 4096 so fixing this is relevant. You can use logit_bias for it.