abhgh

This is interesting. For a while I used to print out code and debug too - more due to resource constraints: didn't own a computer, and my high school's computers were available only for limited hours per day. I used to code my first iteration also on paper - so that in the limited computer use time, I could type in the code quickly and actually see it run. Wild times but I think it made me good at programming. My friend (who did have a computer at home, but lived far away) and I co-wrote a program called "graph generator" that could plot math functions, with me doing my share on paper and giving to him to type in :) This was in the Turbo C++ IDE and since GUI based programs were unusual (at least as a project in highschool - India, 1999) it temporarily made us popular. Well, wow, that was fun to reminisce!

I loved Windows Aero and the flip 3D view - the latter mostly because it looked so cool for that time!

Its funny - I was going to leave a similar comment - and I have, earlier, on a different thread. If people need fast classification, on a fairly scoped problem, it is very fruitful to start with an off-the-shelf embedding model like ModernBERT (which Laya uses) and stick a classifier in front - like a Support Vector Machine (SVM). For starters just tune the SVM, you don't even have to fine-tune the embedder - often it works very well, esp. given the compute needed. Plus you can get reliable confidence scores and generate explanations if you want them (using something like SHAP).

I have used Sol for manim work - that also is quite good.

This tells me the confidence of the LLM's belief about the response - which is different from the calibrated confidence score. The former also is useful (just not what I thought their advertisement sells - and from the article it seems like it tripped up others as well), and there are different techniques to extract such a value [1] [2], typically via "response sampling", i.e., interrogate the LLM slightly differently to see if it changes its answer.

[1] Semantic Entropy https://www.nature.com/articles/s41586-024-07421-0

[2] Kernel Language Entropy https://openreview.net/pdf?id=j2wCrWmgMX

Thank you!

I like this post. I haven't had time to dig into Jev (they aren't accepting new signups), but calibrated probabilities is one of their pitches that caught my attention. And I was wondering how does one offer them on user data. Standard calibration essentially ensures that if a score of 0.8 accompanies a positive prediction (assuming the simple case of binary classification), then if you gathered together all predictions with a score of 0.8, around 80% will be correct.

If you have just one example you're sending to a model, how would they guarantee 80% over your data?

FYI, for an overview, scikit's page on calibration is great [1], and my answer on Quora from a long time ago covers a specific type [2].

[1] https://scikit-learn.org/stable/modules/calibration.html

[2] https://www.quora.com/How-is-isotonic-regression-used-in-pra...

I do this too! Usually some idea in ML ...when I am driving I might suddenly remember what I was thinking about and then it is my personal podcast via Gemini-in-Maps. My only complaint is if you have follow up questions, you have to be quick, otherwise it cuts off the mic.

Was looking for a video on this specific work, but only found this [1] where the English translation seems AI-generated. It reminded me of thermoacoustic refrigeration which I found cool (heh) when I had encountered it many years ago - where the compression and rarefaction of sound waves is used to pull out heat. The fire extinguisher seems to operate on a different principle though - it needs to generate a specific kind of turbulence to push out oxygen from the fire site.

[1] https://www.youtube.com/watch?v=TGO7Jdfll7c

[2] https://en.wikipedia.org/wiki/Thermoacoustic_heat_engine

The searchable term is "Self-preference Bias" :)

My first encounter with any kind of study was the G-Eval paper [1]. They study whether their LLM judge prefers human or LLM-generated summaries (answer: it's the latter).

[1] Section 4 in https://aclanthology.org/2023.emnlp-main.153/

What you call "Weathering" has been a constant gripe of mine. We have LLM-driven softwares toward that almost seem to start from scratch every time a request comes in - there are mechanisms to learn or generalize, like writing out a memory, but they are not reliable or reliable in general. There is no convenient lever to be able to say "yes this is in the memory but the request seems like it needs a fresh scan of data, so ignore your memory", or the opposite "you can infer this from stuff in the memory - don't re-reason!". There is some work like Dynamic Cheatsheets [1] and Agentic Context Engineering [2] that have studied this aspect, but we are far from a generally reliable solution. And till we have that, I think the system variations for systems trying to solve this problem are going to be (a) LLM-leaning: create unstructured memory files, with human in the loop as a filter to reject inaccurate responses (b) LLM-as-a-layer: what you describe and the article kind of is doing.

[1] https://aclanthology.org/2026.eacl-long.333/ [2] https://openreview.net/pdf?id=eC4ygDs02R

Cool project! I tried something similar a while ago [1] - I wanted to load up an embedding model and semantically order texts, all in the browser.

So I pull ONNX weights from HuggingFace (MPNet, MiniLM), use Transformers.js to embed, and use a clusterer from scikit-learn (running on pyiodide - it was a surprise to me that this worked flawlessly) on the page - all client-side.

[1] http://sol.quipu-strands.com/

Oh wow, they are still around! I used to keep a Knoppix disk around for times when my desktop would run out of space and wouldn't boot (if I remember right). Boot it with Knoppix, delete some files, and then boot it normally!

You usually see this kind of analyses in conference papers, esp. if they have a datasets track. The NeurIPS Datasets & Benchmarks (D&B) track is a good example. But you will have to monitor the proceedings yourself closely - there is little chance of being accidentally exposed to them, because most blogs, announcements and popular media only mention a handful of the popular ones, e.g., Tau^2. For ex., across the years 2022, 2023 and 2024, 900+ papers were accepted in the D&B track [1] - of course, not all of them are LLM-related. I find them interesting because they often focus on specific system behaviors, and like you said, study them scientifically, so you can draw authoritative conclusions (or at least know specifically what part of a model's behavior you now know about, and what parts you don't).

[1] https://blog.neurips.cc/2025/09/30/reflecting-on-the-2025-re...

These "You're right to push back" scenarios are scary for me. I mostly code ML implementations, and some of the errors Claude Code (CC - have only used Opus 4.7) makes are very sneaky, and if you don't have sufficient experience in the area (I see this with people entering ML and writing their implementations with CC), you wouldn't know when to question CC and will let errors or future pitfalls silently slip into your code. A recent example was when there was data leakage in a model calibration step, which it refused to see as an error, till I wrote a detailed reason, and then it agreed that there was a "subtle leakage".

Reg PRs - for the ones with complex requirements what I am seeing is that time to initial PR is very short, and a ping-pong between the reviewer and developer begins, because in my cases (not all) the developer vibe-coded parts, and they didn't really understand the requirements deeply or their code, and it takes multiple iterations for them to fix it. You can argue this is a human problem but this is the net effect I'm seeing.

I am not sure but for complex cases it seems to me that the earlier sum of moderately long PR time + moderately long review time has been replaced by very short PR time + even longer review time. I am not sure if there's a net gain in these cases. Sometimes even if the code is functionally correct, it's verbose enough (e.g., too many intermediate functions) that I think they will impact future reviews.

I should go back to this (like some of the other comments suggest) - I think there is potential in suggesting multiple lines, and I would have really loved it to work, but it is clear that their ux testing was poor.

I am not particularly bothered by the speed but the AI suggestion clutter is quickly becoming an issue for me :( I type fragments of a line and it will suggest the next 5-6 lines. They may not be outright wrong, but they might not represent the way I like to do things. Pressing Esc. and refocusing on what I was going to type in anyway is a disruptive experience.

They are great for specialized use-cases: (a) where the problem is not hard enough (you don't need reasoning), or (b) diverse enough (you don't need a world model), (c) you want cheap inference (and you can make it happen hardware-wise) and (d) you either have enough data or a workflow that accumulates data (with fine tuning with enough data you can sometimes beat a premier model while ensuring low latency - ofc, assuming (a) and (b) apply).

I make it sound like a rare perfect storm needs to exist to justify fine tuning, but these circumstances are not uncommon - to an extent (a), (c) and (d) were already prerequisites for deploying traditional ML systems.

I notice you mentioned dspy - do you also support prompt optimization?

This is an amazing quote - thank you. This is also my argument for why I can't use LLMs for writing (proofreading is OK) - what I write is not produced as a side-effect of thinking through a problem, writing is how I think through a problem.

I once modeled user journeys on a website using fancy ML models that honored sequence information, i.e., order of page visits, only to be beaten by bag-of-words (i.e., page url becomes a vector dimension, but order is lost) decision tree model, which was supposed to be my baseline.

What I had overlooked was that journeys on that particular website were fairly constrained by design, i.e., if you landed on the home page, did a bunch of stuff, put product X in the cart - there was pretty much one sequence of pages (or in the worst case, a small handful) that you'd traverse for the journey. Which means the bag-of-words (BoW) representation was more or less as expressive as the sequence model; certain pages showing up in the BoW vector corresponded to a single sequence (mostly). But the DT could learn faster with less data.

I use Claude Code within Pycharm and I see the git diff format for changes there.

EDIT: It shows the side-by-side view by default, but it is easy to toggle to a unified view. There's probably a way to permanently set this somewhere.

Thank you. Your comment about LLMs to semantically parse diverse data, as a first step, makes sense. In fact come to think of it, in the area of prompt optimization too - such as MIPROv2 [1] - the LLM is used to create initial prompt guesses based on its understanding of data. And I agree that UMAP still works well out of the box and has been pretty much like this since its introduction.

[1] Section C.1 in the Appendix here https://arxiv.org/pdf/2406.11695

I was not aware this existed and it looks cool! I am definitely going to take out some time to explore it further.

I have a couple of questions for now: (1) I am confused by your last sentence. It seems you're saying embeddings are a substitute for clustering. My understanding is that you usually apply a clustering algorithm over embeddings - good embeddings just ensure that the grouping produced by the clustering algo "makes sense".

(2) Have you tried PaCMAP? I found it to produce high quality and quick results when I tried it. Haven't tried it in a while though - and I vaguely remember that it won't install properly on my machine (a Mac) the last time I had reached out for it. Their group has some new stuff coming out too (on the linked page).

[1] https://github.com/YingfanWang/PaCMAP

Thanks for the example. Yes, true, this is for expensive functions - to be precise functions that depend on data that is hard to gather, so you interleave the process of computing the value of the function with gathering strategically just as much data as is needed to compute the function value. The video on their page [1] is quite illustrative: calculate shortest path on a graph where the edge weights are expensive to obtain. Note how the edge weights they end up obtaining forms a narrow band around the shortest path they find.

[1] https://willieneis.github.io/bax-website/

Timefold looks very interesting. This might be irrelevant but have you looked at stuff like InfoBax [1]?

[1] https://willieneis.github.io/bax-website/

You don't - the way I use LLMs for explanations is that I keep going back and forth between the LLM explanation and Google search /Wikipedia. And of course asking the LLM to cite sources helps.

This might sound cumbersome but without the LLM I wouldn't have (1) known what to search for, in a way (2) that lets me incrementally build a mental model. So it's a net win for me. The only gap I see is coverage/recall: when asked for different techniques to accomplish something, the LLM might miss some techniques - and what is missed depends upon the specific LLM. My solution here is asking multiple LLMs and going back to Google search.

Love awk. In the early days of my career, I used to write ETL pipelines and awk helped me condense a lot of stuff into a small number of LOC. I particularly prided myself in writing terse one-liners (some probably undecipherable, ha!); but did occasionally write scripts. Now I mostly reach for Python.