JoshCole
One thing that becomes very clear from this sort of work is just how bad LLMs are. It can be invisible when you're working with them day to day, because you tend to steer them to where they are helpful. Part of game theory though is being robust. That means finding where things are bad, too, not just exploring happy paths.
To get across just how bad the failure cases of LLMs are relative to humans, I'll give the example of tic tac toe. Toddlers can play this game perfectly. LLMs though, don't merely do worse than toddlers. It is worse then that. They can lose to opponents that move randomly.
They can be just as bad as you move to more complex games. For example, they're horrible at poker. Much worse than human. Yet when you read their output, on the surface layer, it looks as if they are thinking about poker reasonably. So much so, in fact, that I've seen research efforts that were very misguided: people trying to use LLMs to understand things about bluffing and deception, despite the fact that the LLMs didn't have a good underlying model of these dynamics.
It is hard to talk about, because there are a lot of people who were stupid in the past. I remember people saying that LLMs wouldn't be able to be used for search use-cases years back and it was such a cringe take then and still is that I find myself hesitant to talk about the flaws. Yet they are there. The frontier is quite jagged. Especially if you are expecting it to be smooth, expecting something like anything close to actual competence, those jagged edges can be cutting and painful.
Its also only partially solvable through scale. Some domains have a property where, as you understand it better, the options are eliminated and constrained such that you can better think about it. Game theory, in order to reduce exploitability, explores the whole space. It defies minimization of scope. That is a problem, since we can prove that for many game theoretic contexts, the number of atoms is eclipsed by the number of unique decisions. Even if we made the model the size of our universe there would still be problems it could, in theory, be bad at.
In short, there is a practical difference between intelligence and decision management, in much the same way there is a practical difference between making purchases and accounting. And the world in which decisions are treated as seriously as they could be so much so exceeds our faculties that most people cannot even being to comprehend the complexity.
I use Python for a lot of machine learning. My vision transformers, for example, are in Python. There is a lot to like about the Python ecosystem. Throwing away libraries like ablumentations and pytorch because you move to a different ecosystem is a real loss. You probably ought to be using Python if you're doing machine learning of the sort that one immediately thinks of when they see ML.
That said, data science and machine learning are words that cover a lot of ground.
Python often works because it serves as glue code to more optimized libraries. Sometimes, it is annoying to use it as glue code. For example, when you're working on computational game theory problems, the underlying data model tends to be a tree structure and the exploration algorithm explores that tree structure. There is a lot of branching. Vanilla python in such a case is horrifically slow.
I was looking at progress bars in tqdm reporting 10,000 years until the computation was done. I had already reached for numba and done some optimizations. Computational game theory is quite brutal. You're very often reminded that there are less atoms in the universe than objects of interest to correctly calculating what you want to calculate.
Most people use C, C++, and CUDA kernels for the sort of program I was writing. Some people have tried to do things in Python.
> Are there any benefits of using it over Python?
There is an open source implementation of a thing I built. It solves the same problem I solved, but in Python and worse than I solved it and with a lot of missing features. It has a comment in it, discussing that the universe will end before the code would finish, were it to be used at the non-trivial size. The code I wrote worked at the non-trivial size. Clojure, for me, finished. The universe hasn't ended yet. So I can't yet tell you how much faster my code was than the Python code I'm talking about.
> And how is the interop with Python libs?
Worked for me without issue, but I eventually got annoyed that I had to wait for two rounds of dependency resolution in some builds. Conda builds can sometimes have issues with dependency resolution taking an unreasonable amount of time. I was hitting that despite using very few libraries.
In 2016, Clojure was not great for serious data science. That has changed substantially and not just via Java Interop.
- It now has cross ecosystem GPU support via blueberry libraries like neanderthal, which in benchmarking, outperform some serious Java libraries in this space.
- It has columnar indexed JIT optimized data science libraries via cnuernber and techascent part of the Clojure ecosystem. In benchmarking they've outperformed libraries like numpy.
- The ecosystem around data science is also better. The projects aren't siloed like they used to be. The ecosystem is making things interoperate.
- You can now use Python from Clojure via the lib-pythonclj bindings. In general, CFFI is a lot better, not just for Python.
- The linters are way better than they used to be. The REPL support too.
Clojure already had one of the best efficiency scores in terms of code written to what is accomplished, but now you also get REPL integration, and LLMs have been increasingly capable of leveraging that. There are things like yogthos mycelium experiments to take advantage of that with RLLM calls. So its innovating in interesting new ways too, like cutting bugs in LLM generated code.
It just doesn't feel true to me that innovation isn't occurring. Clojure really has this import antigravity feel to it; things other languages would have to do a new release for, are just libraries that you can grab and try out (or maybe that's the python)
Decided to found a company with some of those friends, bridging computational game theory and large language models :)
Have you tried experimenting with ham-fisted? I've found the libraries in the techascent part of the Clojure ecosystem to be very good performance wise. Ditto for neanderthal.
This gains a huge advantage, which allows even more concision: all code is data, so its easy to transform the code.
In Clojure if you want to add support for unless, a thing like if, but evaluating the opposite way you could do this: `(defmacro unless [p a b] `(if (not ~p) ~a ~b))`. Obviously if you wanted to do the same thing in Python you would in practice do `z() if x else y()`. However, you would do it that way because Python isn't as powerful a language. To actually do the same thing in Python you would need to...
1. Add __future__ support.
2. Update the Python language grammar.
3. Add a new AST type.
4. Add a new pass stage to the compiler.
5. Add a python library to integrate with this so you could use it.
Then you could do something like:
from __future__ import macros
defmacro unless(pred, then: block, else_: block = []):
return q[
if not u(pred):
u*(then)
else:
u*(else_)
]
This sort of, wow, it takes hundreds or thousands of lines more to accomplish the same thing outside of Lisp as it does to accomplish it within Lisp shows up quite often; consider something like chaining. People write entire libraries to handle function chaining nicely. `a.b().c().d().map(f).map(g)`. Very pretty. Hundreds of lines to enable it, maybe thousands, because it does not come by default in the language.
But in Lisp? In Clojure? Just change the languages usual rules, threading operator and now chaining is omnipresent: `(->> a b c d e (map f) (map g))`. Same code, no need to write wrapper libraries to enable it.
Nice payoff. Others have also called out the relationship to compression (https://www.youtube.com/watch?v=AKMuA_TVz3A).
I do understand the points you are attempting to make. The reason you're failing to prove your point is not because I am failing to understand the thrust of what you were trying to argue.
Imagine you were talking to someone who was a rocket scientist, and you were talking to them about engines and you had an understanding of engines that was predicated on your experience with cars. You start making claims about the nature of engines and they disagree with you they argue with you and they point out all these ways that you're wrong. Is this person going to be doing this because they're not able to understand your points? Or is it more likely that their experience with engines that are different than the engines that you're used to give them a different perspective that forced them to think of the world in a different way than you do?
> An LLM only learns through input text.
This is false. There already exist LLM which understand more than just text. Relevant search term: multi-modality.
> It doesn't have a first-person 3D experience of the world.
Again false. It is trivial to create such an experience with multi-modality. Just set up an input device which streams that.
> So it can't execute physical experiments, or even understand them.
Here you get confused again. It doesn't follow, based on perceptual modality, that someone can't do or understand experiments. Hellen Keller can be both blind, but also do an experiment.
Beyond just being confused, you also make another false claim. Current LLMs already have the capacity to run experiments and do so. Search terms: tool usage, ReAct loop, AI agents.
> It can understand the texts about it, but it can't visualize it, because it doesn't have a visual experience.
Again, false!
Multi-modal LLMs currently possess the ability to generate images.
> And ultimately our physical world is governed by physical processes. So at the fundamentals of physical reality, the LLMs lack understanding. And therefore will stay dependent on humans educating and correcting it.
Again false. The same sort of reasoning would claim that Hellen Keller couldn't read a book, but braille exists. The ability to acquire information outside an umwelt is a capability that intelligence enables.
https://chatgpt.com/share/683a3c88-62a8-8008-92ef-df16ce2e8a...
You're running into an issue here due to overloading terms. Training data has three different meanings in this conversation depending on which context you are in.
1. The first is the pre-training context in which we're provided a dataset. My words were appropriate in that context.
2. The second is the reinforcement learning setup context in which we don't provide any dataset, but instead provide a reward model. My words were appropriate in that context.
3. The final context is that during the reinforcement learning algorithms operation one of things it does is generate datasets and then learn from them. Here, its true that there exists a dataset in which "C" is defined.
Recall that the important aspect of this discussion has to do with data provenance. We led off with someone claiming that an analog of "C" wasn't provided in the training data by a human explicitly. That means that I only need to establish that "C" doesn't show up in either of the inputs to a learning algorithm. That is case one and that is case two. It is not case three, because upon entering case three the provenance is no longer from humans.
Therefore, the answer to the question but doesn't the reward model for C mean that C is in the training data has the answer: no, it doesn't, because although it appears in case three, it doesn't appear in case one or case two and those were the two cases which were relevant to the question. That is appears in case three is just the mechanism by which the refutation that it could not appear occurs.
> I am not sure if that is an accurate model, but if you think of it as a vectorspace, sure you can generate a lot of vectors from some set of basevectors, but you can never generate a new basevector from others, since they are linearly independent, so there are a bunch of new vectors you can never generate.
Your model of vectors sounds right to me, but your intuitions about it are a little bit off in places.
In machine learning, we introduce non-linearities during training (for example, through activation functions like ReLU or Sigmoid). This breaks the strict linear structure of the model, enabling it to approximate a much wider range of functions. There's a mathematical proof (known as the Universal Approximation Theorem) that shows how this non-linearity allows neural networks to represent virtually any continuous function, regardless of its complexity.
We're not really talking about datasets when we move into a discussion about this. Its closer to a discussion of inductive biases. Inductive bias refers to the assumptions a model makes about the underlying structure, which guide it toward certain types of solutions. If something doesn't map to the structure the inductive bias assumes, it can be possible for the model to be incapable of learning that function successfully.
The last generation of popular architectures used convolutional networks quite often. These baked in an inductive bias about where data that was related to other data was and so made learning some functions difficult or impossible when those assumptions were violated. The current generation of models tends to be built on transformers. Transformers use an attention mechanism that can determine what data to focus on and as a result they are more capable of avoiding the problems that bad inductive bias can create since they can end up figuring out what they are supposed to be paying attention to.
Again, tons of false claims. One is that 'you' have to create the reward model. Another that it has to be human-curated at all. Yet another is that you even need to do that at all: you can instead have the model build a bigger model of itself, train using its existing resources or more of them, then synthesize itself back down. Another way you can get around it is to augment the existing dataset in some way. No other changes except resource usage and yet the resulting model will be better, because more resources went into its construction.
Seriously notice: you keep making false claims again and again and again and again and again. You're not stating true things. You really need to reflect. If almost every sentence you speak on this topic is false, why is it that you think you should be able to persuade me to your views? Why should I believe your views, when you say so many things that are factually inaccurate, rather than my own views?
This is a false premise, because there already exist systems, currently deployed, which are not dependent on human-curated reward models.
Refutations of your point include existing systems which generate a reward model based on some learned AI scoring function, allowing self-bootstrapping toward higher and higher levels.
A different refutation of your point is the existing simulation contexts, for example, by R1, in which coding compilation is used as a reward signal; here the reward model comes from a simulator, not a human.
> So it still depends on human thinking
Since your premise was false your corollary does not follow from it.
> And there won't be a point when human curated reward models are not needed anymore.
This is just a repetition of your previously false statement, not a new one. You're probably becoming increasingly overconfident by restating falsehoods in different words, potentially giving the impression you've made a more substantive argument than you really have.
Hierarchical optimization (fast global + slow local) is a precise, implementable notion of "thinking." Whenever I've seen this pattern implemented, humans, without being told to do so by others in some forced way, seem to converge on the use of verb think to describe the operation. I think you need to blacklist the term think and avoid using it altogether if you want to think clearly about this subject, because you are allowing confusion in your use of language to come between you and understanding the mathematical objects that are under discussion.
> It can produce "new" stuff only by combining the "old" stuff in new ways,
False premise; previously debunked. Here is a refutation for you anyway, but made more extreme. Instead of modeling the language task using a pre-training predictive dataset objective, only train on a provided reward model. Such a setup never technically shows "old" stuff to the AI, because the AI is never shown stuff explicitly. It just always generates new things and then the reward model judges how well it did. Clearly, the fact that it can do generation while knowing nothing, shows that your claim that it can never generate something new -- by definition everything would be new at this point -- is clearly false. Notice that as it continually generates new things and the judgements occur, it will learn concepts.
> But LLM's don't have an understanding of things, they are statistical models that predict what statistically is most likely following the input that you give it.
Try out Jayne's Probability Theory: The Logic Of Science. Within it the various underpinning assumptions that lead to probability theory are shown to be very reasonable and normal and obviously good. Stuff like represent plausibility with real numbers, keep rankings consistent and transitive, reduce to Boolean logic at certainty, and update so you never accept a Dutch-book sure-loss -- which together force the ordinary sum and product rules of probability. Then notice that statistics is in a certain sense just what happens when you apply the rules of probability.
> also having a understanding of certain concepts that allows you to arrive at new points like C, D, E, etc. But LLM's don't have an understanding of things
This is also false. Look into the line of research that tends to go by the name of Circuits. Its been found that models have spaces within their weights that do correspond with concepts. Probably you don't understand what concepts are -- that abstractions and concepts are basically forms of compression that let you treat different things as the same thing -- so a different way to arrive at knowing that this would be true is to consider a dataset with less parameters than there are items in the dataset and notice that the model must successfully compress the dataset in order to complete its objective.
This premise is false. It is fundamentally equivalent to the claim that a language model being trained on a dataset: ["ABA", "ABB"] would be unable to generate, given input "B" the string "BAB" or "BAA".
In 2014, under Obama, the United States Digital Service was established with by Congressional appropriations with the mission to "deliver better government services to the American people through technology and design" and in 2025 it was renamed to DOGE.
Have you considered that it might have been renamed DOGE for the same reason that Tesla car models spell out the word SEXY? The mission has not changed very substantively. The order directs them to modernize federal technology and software to maximize efficiency and productivity.
[1]: https://en.wikipedia.org/wiki/United_States_Digital_Service
[2]: https://www.govinfo.gov/content/pkg/FR-2025-01-29/pdf/2025-0...
Lets call someone working at such a company Bob.
A restatement of your claim is that Bob decided to launch a model to live because of hype rather than because he could justify his promotion by pointing to the millions of dollars in increased revenue his switch produced. Bob of course did not make his decision based on hype. He made his decision because there were evaluation criteria in place for the launch. He was literally not allowed to launch things that didn't improve the system according to the evaluation criteria. As Bob didn't want to be fired for not doing anything at the company, he was forced to use a tool that worked to improve the evaluation according to the criteria that was specified. So he used the tool that worked. Hype might provide motivation to experiment, but it doesn't justify a launch.
I say this as someone whose literally seen transitions from decision trees to deep learning models on < 100 feature models which had multi-million dollar monthly revenue impacts.
Here is a quote from research related to this subject:
> Compared to 2012, it now takes 44 times less compute to train a neural network to the level of AlexNet (by contrast, Moore’s Law would yield an 11x cost improvement over this period). Our results suggest that for AI tasks with high levels of recent investment, algorithmic progress has yielded more gains than classical hardware efficiency.
When you apply the principle of charity you can make their claim increasingly vacuous and eventually true. We're still doing optimization - we're still in the same general structure. The thing is, it becomes absurd when you do that. Its not appropriate to take such a premise seriously. It would be like taking seriously the argument that we haven't had any advancement in software engineering since bubble sort since we're still in the regime of trying to sort numbers when we sort numbers.
Its like, okay, sure, we're still sorting numbers, but it doesn't make the wider point it wants to make and its false even under the regime it wants to make the point under.
This isn't even the only issue that makes this premise wrong. For one, AI research in the 80s wasn't centered around neural networks. Hell even if you move forward to the 90s PAIP puts more emphasis on rule systems with programs like Eliza and Student than it does learning from data. So it isn't as if we're in a stagnation without advance; we moved off other techniques to the ones that worked. For another, it tries to narrow down AI research progress myopically to just particular instances of deep learning, but in reality there are a huge number of relevant advances which just don't happen to be in publicly available chat bots but which are already in the literature and force a broadening. These actually matter to LLMs too, because you can take the output of a game solver as conditioning data for an LLM. This was done in the Cicero paper. And the resulting AI has outperformed humans on conversational games as a consequence. So all those advancements are thereby advances relevant to the discussion, yet myopically removed from the context, despite being counterexamples. And in there we find even greater than 44x level algorithmic improvements. In some cases we find algorithmic improvements so great that they might as well be infinite as previous techniques could never work no matter how long they ran and now approximations can be computed practically.
> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.
> Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.
> When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."
> Please don't fulminate. Please don't sneer, including at the rest of the community.
> Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.
> Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. HN is a community—users should have an identity that others can relate to.
You can read the full guidelines here: https://news.ycombinator.com/newsguidelines.html
I have a textbook, "Artificial Intelligence: A Modern Approach," which covers Language Models in Chapter 23 (page 824) and the Transformer architecture in the following chapter. In any field technical terms emerge to avoid ambiguity. Laymen often adopt less accurate definitions from popular culture. LLMs do qualify as AI, even if not according to the oversimplified "AI" some laymen refer to.
It has been argued for the last several decades that every advance which was an AI advance according to AI researchers and AI textbooks was not in fact AI. This is because the laymen have a stupid definition of what constitutes an AI. It isn't because the field hasn't made any progress, but instead because people outside the field lack the sophistication to make coherent statements when discussing the field because their definitions are incoherent nonsense derived from fiction.
> They are a GREAT statistical parlor trick for people that don't understand statistics though.
The people who believe that LLMs constitute AI in a formal sense of the word aren't statistically illiterate. AIMA covers statistics extensively: chapter 12 is on Quantifying Uncertainty, 13 on Probabilistic Reasoning, 14 on Probabilistic Reasoning Over Time, 15 on Probabilistic Programming, and 20 on Learning Probabilistic Models.
Notably, in some of these chapters probability is proven to be optimal and sensible; far from being a parlor trick it can be shown with mathematical rigor that failing to abide by its strictures is not optimal. The ontological commitments of probability theory are quite reasonable; they're the same commitments logic makes. That we model accordingly isn't a parlor trick, but a reasonable and rational choice with ledger arguments proving that failing to do so would lead to regret.
Of course, anecdotal evidence isn't a very high standard. Thankfully, statistics on this sort of thing are tracked. Statistically, the Tesla self-driving features reduce accidents per mile. They have for years now and as the tech has progressed the reduction has grown as the technology has matured. So statistical evidence also indicates that the person you are asking the question to is also uninformed.
What is probably happening is that it makes for good clickbait to involve Elon and Tesla into discussions. Moreover, successful content online often provokes emotion. The resulting preponderance of negativity, especially about each driving accident Teslas were involved in or caused, probably tricked them into misunderstanding the reality of the Tesla safety record.
Trying to dismiss this sort of reasoning by claiming naivety on the part of those who trust Tesla doesn't work. I have the car. I have physically been driven under an underpass in the car. I have seen the car slow as it travels under an underpass. I have observed for myself that this is no longer happening, because the car now drives more safely in this circumstance.
Not only does it not work, but you have to dismiss a lot of evidence, not just one specific thing. For example, one bit of evidence you would have to dismiss would be the Karpathy CVPR talk in which video evidence of accuracy improvement changes is shown. It is really problematic for your argument that you have to dismiss this evidence too, because previously you were willing to accept evidence from Karpathy. Before, you have a misunderstanding of Karpathy's comment on a podcast and you were willing to credulously accept that misunderstanding because it agreed with your intuitions, but now you have an actual machine learning conference technical talk and you are not willing to accept his comment any longer because it disagrees with your intuition and yet in the latter case you have video and metrics rather than just audio commentary.
Regarding leaving another comment: I did it because I felt my previous comment was too abstract. I wanted to make sure you understood that to not be having hindsight bias your beliefs would need to predict a reduction in safety as a result of the change, but this isn't evidenced. So it is definitely hindsight bias.
Look - I get it. I too thought radar sensor fusion would improve the results. Guess who else thought this? The people you consider stupid and reckless thought this. Do you know why they changed their mind? According to you, because they are stupid and reckless.
I can't stress this enough: you are telling yourself inaccurate stories about what happened. If you just /look/ at the /measured impact/ instead of /wrongly guessing what Karpathy thought/ and /wrongly guessing what Karpathy has said/ and /wrongly guessing what other people were referencing when they talked about what Karpathy said/ then you would realize this.
You are explaining the observed safety record of Tesla being on top for safety by appealing to them being inferior and reckless. You are explaining Tesla causing a greater reduction in accident death than Waymo by them being more inferior and reckless than Waymo.
You are not explaining the evidence with your theory.
Your theory doesn't even reflect reality. Google invested in Tesla and in Waymo. Google did both things. Google didn't make the right call. They made both calls.
https://www.youtube.com/watch?v=g6bOwQdCJrc
For example look at this section: https://youtu.be/g6bOwQdCJrc?t=1370
Also if you look into it you will find Google's self-driving car project ran into similar issues. They upgraded the radar in response to the issues rather than removing it, but there are actual issues.
Vision only was better than radar plus vision in their stack. For real world situations that frequently come up in driving radar wasn't worth it. The theoretical accuracy of the sensor fusion didn't happen in practice because in practice you needed to have vision tell you when sensor fusion wasn't going to be accurate. It isn't naive sensor fusion as in toy math models of sensor fusion. So when there was a disagreement you needed to go with vision. Yet vision was already able to predict both the failure of and the actual result of radar. In theory, you could just have done the more complicated sensor fusion. In practice, the relative advantage of improving other parts of the vision stack far exceeded improving the sensor fusion. Over time one would expect the relative advantage to change as other improvements were made such that this was no longer true. That doesn't mean the time at the decision wasn't correct, it just means people who evaluate it with future changes already in place are suffering from hindsight bias/committing an anachronistic fallacy.