News for hackers. Stuff that matters.

Jadiker

It looks like the hallucination score is somewhat related to perplexity in the sense that it relies on specific tokens. This could cause issues because rephrasing or using slightly different terms could lead to a higher hallucination score. E.g. if the correct answer is "John Smith is the world's best baker" then "Mary Kay is the world's best baker" would have a better score (lower hallucination) than "Leading maker of baked items across all the continents: John Smith" according to your metric.

Are there any plans to make updates to this score or add in different metrics for more accurately detecting hallucinations that don't penalize rephrasing?

Looks interesting! How does it compare to things like Optuna or RayTune or Weights and Biases?
I agree with many of the listed ones. Here are a couple I think I can add:

1. Knowing keyboard shortcuts. The most powerful one is to flip between screens on my computer (Alt+Tab for Windows, CMD+Tab for Mac).

2. Having a pipeline for AutoML such as Tpot in Python (http://epistasislab.github.io/tpot/) where it will automatically go through a bunch of ML algorithms and tell you which one did the best on your data.