Fun Facts

- Not all these models are instruct aligned

- Cohere's suite is free for personal use

- anthropic uses reinforcement learning from ai feedback for instruct alignment, removing humans from the process almost entirely(humans still make the rules the language model checks the responses against).

- open ai's models are pretty poorly calibrated compared to its peers. Calibration is how well the model's confidence solving a problem tracks with how it actually performs that problem. It seems to be a consequence of the way they are performing the instruct tuning (or maybe it's a consequence of instruct tuning in general). The gpt-4 report actually shows this. Base gpt-4 was excellently calibrated but rlhf knocked it right out. The excellent calibration was also independently verified (in the medical domain) from the paper about gpt-4's medical prowess. https://arxiv.org/abs/2303.13375

One of OpenAI's engineers / cofounders (Greg Brockman) specifically said it in an interview that GPT-4 itself is quite calibrated, it goes away with the fine tuning process (I don't remember exactly which part of the interview). Of course they are working on fixing it, I guess GPT-4.5 will contain the fix.

https://www.youtube.com/watch?v=YtJEfTTD_Y4

There should be a model trained only on what's publicly reachable through the great firewall.

Then that should square off against a model trained on everything..

I feel that the one with the great firewall filter might actually be better for some domains.

That's great, now I guess we need a page that ranks those benchmarks to tell us what we're looking at...

It seems to be missing Llama and other Llama derived models, and GPT4.

> last updated 2023-03-19

Yep, anything that old is ancient in the AI world

Singularity when? Plot twist:..

Singularity will happen when we have real quantum computing then we wont know if the Ai is alive or not or both.

I don't think that's how that works