The other replies are spot on but in general -- Alignment induces cost, no matter WHAT type of alignment, even 'neutral' versions like instruct for chatbots

The 'common' example that people reference is, for instance, making the model non-offensive means it has to spend.. something... on that directive, if you will. (which also led to absurdities such as refusing to help people with bash commands that involve having to ```kill``` a process...)

But even for the purpose of an instruct model (this is what peeves me off), making it answer questions and take instructions makes it *worse* at many creative tasks because you're constraining it's behaviour to Q&A -- though this is a long tangent...

alignment tax = Training needed to cancel bad behaviour, political incorrectness, dangerous results etc.

Yes, but you want alignment even if you don't want censorship. A very intelligent but unaligned model will be prone to doing useless things like "auto-complete" your question into a more elaborate version or responding with "just google it dumbass" and other forms of internet vitriol.

> Yes, but you want alignment even if you don't want censorship.

Instruction tuning helps a lot with this but what a lot of people mean is the refusal to do things. You get to chose how "aligned" it is, for some usecases like talking to customers you definitely want something very "safe" (won't start using slurs or something terrible). But for direct usage you generally never want it to refuse to do anything.

Checkout Anthropic on the extreme side - every iteration of Claude has gotten worse on the chatbot arena (elo based on humans blindly comparing responses).

https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

If you want an AI to actually do things like answer your questions instead of just complete the question you have to nudge the model to do that which makes it stray from the "best" answer. There's a balance you need to strike between aligning it enough so that it gives answers humans actually want and not aligning too much to degrade the models performance.

People have an axe to grind with OpenAI because they purposely align their model to be boring as the intended audience is companies embedding it in their own products and try to fight the bias inherent in a model trained on the writings of the average chronically online Redditor. The frustration isn't unfounded, if you want to use the model for programming having to pay the 'alignment tax' because someone else is using the model for a customer service bot and can't match energy with an angry customer sucks.