News for hackers. Stuff that matters.

A comment on Anyone working on LLM tools for enhancing data quality?.

Re:Anyone working on LLM tools for enhancing data quality? by hxypqr on Saturday January 13, 2024 @09:33AM (#38980261)
Obviously, the cost of using LLM to solve this problem is relatively high. At most, it can only treat LLM as one of the signals instead of processing all the information. Combining LLM with tree models or graph models and adding NER technology may achieve an approximate solution to 2. Can you really trust the answers provided by LLM to be used in a production environment?
[ Parent | on HN ]
Re:Anyone working on LLM tools for enhancing data quality? by johannesboyne on Tuesday January 16, 2024 @03:49AM (#39010997)
The combination is truly key, how we do it:

- Combining LLMs, gradient boosting, and multiple statistical/heuristic based approaches - For LLMs: rely on (quite) large context length + fine-tuning - The (ML) models are used to extract mathematically provable quality checks

[ Parent | on HN ]