Honestly defining what to teach is hard

People need to learn to do research with LLMs, code with LLMs, how to evaluate artifacts created by AI. They need to learn how agents work at a high level, the limitations on context, that they hallucinate and become sycophantic. How they need guardrails and strict feedback mechanisms if let loose. AI Safety connecting to external systems etc etc.

You're right that few high school educators would have any sense of all that.

I don't know anyone who learned arithmetic from a calculator.

I do know people who would get egregiously wrong answers from misusing a calculator and insisted it couldn't be wrong.

Yes but I was also taught to use a calculator, and particular the advanced graphing calculators.

Not to mention programming is a meta skill on top of “calculators”

Before I was taught to use a graphing calculator, we learned by using graph paper and a pencil to plot linear equations. Once we had that mastered, we were taught how to use the calculators so we could be more efficient with our time and move on to more complex topics.

I certainly practiced a lot on a calculator. Oh and I was very interested in how this concept, doing math using equipment rather than brains, worked.

https://en.wikipedia.org/wiki/Little_Professor

The sycophancy is an artifact of how they RLHF train the popular chat models to appeal to normies, not fundamental to the tool. I can't remember encountering it at all since I've started using codex, and in fact it regularly fills in gaps in my knowledge/corrects areas that I misunderstand. The professional tool has a vastly more professional demeanor. None of the "that's the key insight!" crap.

> AI is extremely dangerous for students and needs to be used intentionally

Can you expound on both points in more details please, ideally with some examples?

If a student uses AI to simply code-gen without understanding the code (e.g. in my compilers class if they just generate the recursive-descent parser w/Claude, fixing all the tests) then they are robbing themselves of the opportunity to learn how to code.

In OP I showed an AGENTS.md file I give my students. I think this is using AI in a manner productive for intellectual development.

I asked my students in a take home lab to write tests for a function that computes the Collatz sequence. Half of the class returned AI generated tests that tested the algorithm with floating point and negative numbers (for "correct" results, and not for input validation). I am not doing anything take home anymore.