Do you run the model locally on your computer or how do you use it as an alternative to ChatGPT?

I have a linux box which i run it locally for batch processing, otherwise i just run it through poe (serves as a good comparative website endpoint for differently models i find.)

Can you elaborate what poe is?

I can run Mistral 7B, in 3-bit quantization, on my phone, and I get helpful answers for (just today) seeing environment variables in Python.

What do you use to run it on your phone?

The iOS app “MLCChat”

chat.lmsys.org has a zillion models you can use.

Check out LMStudio if you want to try. Open source and quite easy to use, doesn't need a beefy machine.

Thank you, it looks really good

Mistral fine-tunes are competitive with GPT 3.5 and 4 on many, many tasks, people are just enamored with the (admittedly impressive) ability of very large general models. In terms of business applications, the range of things that need GPT-4 to solve vs. Mistral 7B is diminishing small and infinitesimal when you factor in Mixtral. I’m all for doing advanced research to create the most interesting and powerful thing possible, but in more practical terms, OpenAI has zero moat. Once the infrastructure around open source stabilizes and we get a few more rounds of improvement to efficiency, I think this will be plainly obvious.

"milquetoastian nature"?

sorry, i apologize for using the term milquetoast without context ;)

it just kind of means "overly apologetic, meek, timid, unwilling to ever voice anything that may be even mildly controversial", etc. It apparently originates from H.T. Webster "The Timid Soul"

I had to lookup for that too: milque·toast: A timid or feeble person.

white bread

prudish

square

no fun

"Alignment tax?"

The other replies are spot on but in general -- Alignment induces cost, no matter WHAT type of alignment, even 'neutral' versions like instruct for chatbots

The 'common' example that people reference is, for instance, making the model non-offensive means it has to spend.. something... on that directive, if you will. (which also led to absurdities such as refusing to help people with bash commands that involve having to ```kill``` a process...)

But even for the purpose of an instruct model (this is what peeves me off), making it answer questions and take instructions makes it *worse* at many creative tasks because you're constraining it's behaviour to Q&A -- though this is a long tangent...

alignment tax = Training needed to cancel bad behaviour, political incorrectness, dangerous results etc.

Yes, but you want alignment even if you don't want censorship. A very intelligent but unaligned model will be prone to doing useless things like "auto-complete" your question into a more elaborate version or responding with "just google it dumbass" and other forms of internet vitriol.

> Yes, but you want alignment even if you don't want censorship.

Instruction tuning helps a lot with this but what a lot of people mean is the refusal to do things. You get to chose how "aligned" it is, for some usecases like talking to customers you definitely want something very "safe" (won't start using slurs or something terrible). But for direct usage you generally never want it to refuse to do anything.

Checkout Anthropic on the extreme side - every iteration of Claude has gotten worse on the chatbot arena (elo based on humans blindly comparing responses).

https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

If you want an AI to actually do things like answer your questions instead of just complete the question you have to nudge the model to do that which makes it stray from the "best" answer. There's a balance you need to strike between aligning it enough so that it gives answers humans actually want and not aligning too much to degrade the models performance.

People have an axe to grind with OpenAI because they purposely align their model to be boring as the intended audience is companies embedding it in their own products and try to fight the bias inherent in a model trained on the writings of the average chronically online Redditor. The frustration isn't unfounded, if you want to use the model for programming having to pay the 'alignment tax' because someone else is using the model for a customer service bot and can't match energy with an angry customer sucks.