News for hackers. Stuff that matters.

scoresmoke

Before I buy, can you confirm the bridge is GDPR-compliant, AI-Act-ready, has a digital product passport, and passed its environmental impact assessment? Otherwise the local compliance officer will fine us before it even collapses.
Re:Show HN: Improving RAG with chess Elo scores on Thursday July 17, 2025 @03:57PM (#44597491)
I am working on post-training and evaluation tasks mostly, and I built Evalica as a convenient tool for my own use cases. The computation is fast enough to not bother the user, but the library does not stand in my way during the analysis.
Re:Show HN: Improving RAG with chess Elo scores on Thursday July 17, 2025 @01:30AM (#44589992)
I am thinking about this for a while and I think I’ll vibecode them. Not sure about WASM, though, as the underlying libraries should support it, too, and I am not sure about all of them at the same time.
Re:Show HN: Improving RAG with chess Elo scores on Wednesday July 16, 2025 @06:07PM (#44587331)
You might also consider a fast implementation of Elo and Bradley–Terry that I have been developing for some time: https://github.com/dustalov/evalica (Rust core, Python bindings, 100% test coverage, and nice API).
GPT-2 follows the very well-studied architecture of Transformer decoder, so the outcomes of this study might be applicable to the more complicated models.
Re:Rye: A Hassle-Free Python Experience on Tuesday July 09, 2024 @04:17AM (#40913677)
Ruff and uv are both excellent tools, which are developed by a VC-backed company, Astral: https://astral.sh/about. I wonder what their pitch was.
Re:NumPy 2.0.0 on Sunday June 16, 2024 @05:02PM (#40700079)
The most important changes are deprecations of certain public APIs: https://numpy.org/devdocs/release/2.0.0-notes.html#deprecati...

One new interesting feature, though, is the support for string routines: https://numpy.org/devdocs/reference/routines.strings.html#mo...

Thank you! I excluded the coding tasks as most annotators don't possess this expertise. I trust them in comparing pairs of dissimilar model outputs that don't require any specific skill but commonsense reasoning.

The only manual analysis was when I checked the passed/failed prompts of the top-performing model.

Show HN: Llmfao – Human-Ranked LLM Leaderboard with Sixty Models

https://dustalov.github.io/llmfao/

In September 2023, I noticed a tweet [1] on difficulties with LLM evaluation, which resonated with me a lot. A bit later, I spotted a nice LLMonitor Benchmarks dataset [2] with a small set of prompts and a large set of model completions. I decided to make my attempt without running a comprehensive suite of hundreds of benchmarks: https://dustalov.github.io/llmfao/I also wrote a detailed post...

( Read More... | 2 comments | on HN )
Re:GraalOS: Containerless instant-on cloud functions for Java on Friday September 22, 2023 @09:58AM (#37612144)
Yet they have a copyright notice in the footer: Copyright © 2023, Oracle and/or its affiliates.
Re:Ask HN: How did LLMs change your products? on Monday September 18, 2023 @07:25AM (#37554782)
It's all about timing and communities. In my echo chamber, everybody is currently making them a part of the product, but I'm interested in a more representative sample. Hiring usually precedes shipping.

Every success in your job search!

Re:Ask HN: How did LLMs change your products? on Monday September 18, 2023 @07:23AM (#37554765)
I found code LLMs to be very useful for rewriting nested for-loops in Python as nice vectorized operations in NumPy. One has to be careful about unexpected array materialization, but generally it works really well and saves a lot of time.
Re:Ask HN: How did LLMs change your products? on Monday September 18, 2023 @07:19AM (#37554747)
I have seen that many teams are trying to perform model routing by sending simpler queries to cheaper models and complex queries to more complex and expensive models, e.g., https://www.anyscale.com/blog/a-comprehensive-guide-for-buil...

Were these people laid off because of LLMs? It sounds pretty dystopic. Who were these people?

Ask HN: How did LLMs change your products?

Since the LLMs and other generative models had been around for a while, many companies claimed to build products on top of them or launch new AI features based on them.What happened at your company? How did these features impact your users, your product, and your management?

( Read More... | 8 comments | 3 points | on HN )
Re:Training and Aligning LLMs with RLHF and RLHF Alternatives on Sunday September 10, 2023 @05:55PM (#37460490)
Discussions about LLM alignment often miss topics of data quality and quantity. It turns out that current models like Llama 2 use 10K+ prompts and responses for supervised fine-tuning (SFT) and 100K+ human preference pairs. While the preferences are pretty easy to annotate, producing a good SFT dataset is uneasy.

https://evalovernite.substack.com/p/rlhf-math-aint-enough

https://doi.org/10.5281/zenodo.8186168

Re:Towards a New SymPy on Friday September 08, 2023 @12:25PM (#37435739)
Yes, this is a big disadvantage. But have you tried Mamba that aims at implementing Anaconda more efficiently? It works really well in most cases.

https://mamba.readthedocs.io/

Older »