Hallucinations you can't fix. Gemini is a bit worse there than DeepSeek, but there's not much research on how to fix that. The only one is the CaMeL paper by Google, where you tag every prompt and result and then for every assistant response or tool call you first check where it got that data and error if you notice fabrication. This one is really annoying to implement.
With larger models the fabrication starts when the context grows or if you have too many tools, for flash models it's much earlier. We use the flash models for repetitive agentic tasks, where the prompt defines clearly what to do and how. The whole run is about 4-5 steps typically, and context size stays in the comfort zone.
I've been using Pi to build custom extensions and wrapping workflows in shell processes to make it more deterministic and enforce certain validations, all guided by Fable. This isn't production work, though, just playing llm factorio at home.
You replay all your sessions against your harness, and then store all logs all output, everything to a safe place.
Finally use a blind judge to check everything, and score the output.
Then fix your harness, iterate again until better until you are in a point where it's just the model's weakness. If you get to that, use a bigger model.
I was mostly using DeepSeek on Pi, connecting to their API directly (not some third party provider).
I honestly have more issues steering Sonnet properly.
I'm using headless Pi with my own UI and sandbox client, https://github.com/gterzian/uni03C0, as well as a bunch of Pi extensions for things like accessing Web standards and browser use via CDP for testing.
Switching to 4.1 today...
Edit: it seems they pushed the date at which they route the Pro calls to new Flash, so today I ended up paying regular Pro rates thinking I was using the new Flash; an example of how their offering is not quite as predictable as I would like it to be (the other is cache performance being unpredictable).
I take the free chat gpt one writ with it in polish suddenly english.
I initially thought it was a trick, that using Chinese chars is somehow more info dense and it saves tokens to 'think' in Chinese.
But later on it became more erratic. I still wonder if token reduction would work that way.
Interesting though, when I ask questions in German or my native language, I rarely get Chinese answers. Looks like English is most affected.
API never answers in Chinese.
Hasn't happened in a while, last time was when I was testing fable 5 in june.
But, the web ui chat version of flash has very poor language following abilities in my experience:
You may ask it something in English, and get a thinking chain in Chinese with an answer in Chinese, or an English thinking chain and an English answer. Using the retry button on the same question has a 50/50 chance of any of those results.
Sometimes, asking something in English, but where information are mostly in another language may make the answer in the language where data has been found. The other day, I asked something about a local German thing, in English, and I got an answer in German instead. It’s as if all the language data stirred it away from the language of the user’s question.