I assume the corpus of text they were trained on matches that 30/70 split?
I assume there's some selection or fine-tuning that happens to improve models fed similarly unreliable training data for other domains... curious how this works.
For whatever it's worth, that a) Astra struggles less on this for some reason, and b) things tend to work better if you have a copy of the FSM and some other appropriate lit like Halderman and quality forum posts or alldata if it's appropriate and tell it to work from there. I've been working on a big search engine for this that feeds a known good corpus like an ass load of contemporary MotorAge mags, relevant technician texts, the FSM, other quality data sources like BITOG, known not-shit forums, along with OBD2 data via something like an OBDLink dongle or a J2534 setup. I still would not trust it, but it can be helpful to eliminate or brainstorm some possibilities.
Factory service manual, in case anyone else is wondering what the finite state machine in your car is for…
As a Codex user, what do you mean? Does Claude actually try to report issues back to Anthropic when something goes wrong?!
In general I've seen them split between 30% brilliance, and 70% pants of head outrageously stupid shit for electronics and automotive stuff (both diagnostics and repair/modification). I would not suggest even frontier models if someone didn't understand what they were doing.