throwaway71271

as this paper shows, it sees they can do tower of hanoi as well, up to a certain point that is.

I think you are right, even if I beleve next token prediction can work, I dont think it can happen in this autoregressive way where we fully collapse the token to feed it back in. Can you imagine how much is lost from each torch.multinomial?

Maybe the way forward is in LCM or go JEPA, therwise, as this Apple paper suggests, we will just keep pushing the "pattern matching" further, maybe we get some sort of phase transition at some point or maybe we have to switch architecture, we will see. It could be that things change when we get physical multimodality and real world experience, I dont know.

For studying? Mainly watching and re-watching Karpathy's 'Zero To Hero'[1] and Stanford's 'Introduction to Convolutional Neural Networks for Visual Recognition'[2], also a lot of transformers from scratch videos like Umar Jamali's videos[3], and I also study backwards to McCulloch and Pitts. Reading the 30 papers https://punkx.org/jackdoe/30.html and so on.

And of course Yannic Kilcher[4], and also listening in on the paper discussions they do on discord.

Practicing a lot with just doing backpropagation by hand and making toy models by hand to get intuition for the signal flow, and building all kinds of smallish systems, e.g. how far can you push whisper, small qwen3, and kokoro to control your computer with voice?

People think that deepseek/mistral/meta etc are democratizing AI, but its actually Karpathy who teaches us :) so we can understand them and make our own.

[1] https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxb...

[2] https://www.youtube.com/watch?v=vT1JzLTH4G4&list=PL3FW7Lu3i5...

[3] https://www.youtube.com/@umarjamilai

[4] https://www.youtube.com/@YannicKilcher

FEAR THE ALL-SEEING BASILISK.

> And, even the dumber LLMs would slot in naturally into such a process

That is what I am struggling with, it is really easy at the moment to slot LLM and make everything worse. Mainly because its output is coming from torch.multinomial with all kinds of speculative decoding and quantizations and etc.

But I am convinced it is possible, just not the way I am doing it right now, thats why I am spending most of my time studying.

You know the meme "seems like will have AGI before we can reliably parse PDFs" :)

So if you are building a system, lets say you ask it to parse a pdf, and you put a judge to evaluate the quality of the output, and then you create a meta judge to improve the prompts of the parser and the pdf judge. The question is, is this going to get better as it is running, and even more, is it going to get better as the models are getting better?

You can build the same system in completely different way, more like 'program synthesis' imagine you dont use llms to parse, but you use them to write parser code, and tests, and then judge to judge the tests, or even escalate to human to verify, then you train your classifier that picks the parser. Now this system is much more likely to improve itself as it is running, and as the models are getting better.

Few months ago Yannic Kilcher gave this example as that it seems that current language models are very constrained mid-sentence, because they most importantly want produce semantically consistent and grammatically correct text, so the entropy mid sentence is very different than the entropy after punctuation. The . dot "frees" the distribution. What does that mean for "generalists" or "specialists" approach when sampling the wrong token can completely derail everything?

If you believe that the models will "think" then you should bet on the prompt and meta prompt approach, if you believe they will always be limited then you should build with program synthesis.

And, honestly, I am totally confused :) So this kind of research is incredibly useful to clear the mist. Also things like https://www.neuronpedia.org/

E.G. Why compliment (you can do this task), guilt (i will be fired if you don't do this task), and threatening (i will harm you if you don't do this task) work with different success rate? Sergey Brin said recently that threatening works best, I cant get my self to do it, so I take his word for it.

I think one of the reason we are confused about what LLMs can do is because they use language. And we look at the "reasoning traces" and the tokens there look human, but what is actually happening is very alien to us, as shown by "Biology of Large Language Models"[1] and "Safety Alignment Should Be Made More Than Just a Few Tokens Deep"[2]

I am struggling a lot to see what the tech can and can not do, particularly designing systems with them, and how to build systems where the whole is bigger than the sum of its parts. And I think this is because I am constantly confused by their capabilities, despite understanding their machinery and how they work, their use of language just seems like magic. I even wrote https://punkx.org/jackdoe/language.html just to remind myself how to think about it.

I think this kind of research is amazing and we have to spend tremendous more effort into understanding how to use the tokens and how to build with them.

[1]: https://transformer-circuits.pub/2025/attribution-graphs/bio... [2]: https://arxiv.org/pdf/2406.05946

Everything is fine

I dream if err, if err dreams me.

Wow! What an amazing story!

I started writing forth few months ago, wrote few interpreters, with jit or with types etc, and its just amazing, I think anyone should do it. TBH I don't think any other exercise has thought me as much about programming as this.

I also notice the "return" of Forth, as it is probably the easiest high level language to make for computers with addressable memory and fetch execute cycle. The parser is just few lines of assembly and of course you can write the parser in the inner interpreter's bytecode, you don't even need assembly :) So hobbyists can just "make it" and make their own tiny operating systems with it. Of course everyone makes their own dialect, but I think thats OK. Things like https://github.com/howerj/lfsr LFSR CPU/VM running Forth, or UXN or duskos/collapseos.

Now you can also use language models to help you onboard into the language, it do some practice programs and rewrite one program in many ways.

So if you are young or old and never tried to Forth, don't miss out, its super fun.

> I disagree with that strongly. The LLM is obviously not a human or a person, but it's not a trivial token predictor, either.

I am sorry, by no means I think it is a trivial token predictor, or a stochastic parrot of some sort. I think it has a world model, and it can do theory of mind to us, but we can not do theory of mind to it. It has planning as visible from the biology of language models paper.

> So again, I disagree about AI-generated tokens having no meaning. But I would agree there is no human connection there

What I argue is that language is uniquely human, and it is how it is because of the human condition. I think we agree more than we disagree. I say that the meaning is 'halved', it is almost as if you are talking to yourself, but the thoughts are coming from the void. This is the sound of one hand slap maybe, a thought that is not your own but it is.

I guess I am saying is that AI is much more like Alien than Artificial, but we read the tokens as if they are deeply human, and it is really hard for people to not think of it as human, purely because it uses language in such profound way.

Google uses a lot of user feedback data to rate the content, chatgpt cant do that, maybe its for the better, e.g. now chatgpt imports 0 star completely unknown libraries from github into my project, it read their code and deemed useful, but there is no way I would've heard about them without it.

Popularity is somewhat proxy for 'good', but not always.

I am not sure. For example now I am writing a book for my daughter, I would like to share it when done, it is not written for ad money, example chapter, just so you know what kind of content I mean: https://punkx.org/projekt0/book/part1/interpreter.html

Is it going to be useful for language models to train on it? I think so, and I don't mind that. As long as they develop better world models and understand human language better.

The problem I have is with humans reading generated tokens. Human language is shared experience, the evaluation and interpretation of the symbols depend both on the author and the reader (even though many times they are the same entity).

When a person with HPPD says 'The sky is black', when the symbols enter your mind they are superimposed with your experience and their experience to create meaning. (HPPD is a disorder from damaged filters on the visual system, it seems that raw information from the eye sensors are entering the brain, and they can see the inside of their eyes when they look at the sky, so it looks black, as if the whole sky is filled with 'floaters)

When you read AI generated content, you are both the judge and executioner, the symbols mean whatever you want them to mean, they have no author (in the human sense).

So, I want to write for other humans to read :) Even if nobody reads it.

I think there is a new nuance on "no one is reading", where _actually_ no one will be reading and only chatgpt will read your work and spit out few tokens to its user.

Now there is a chance of us actually reaching your blog/video etc, like right now on hackernews. Sometimes we will like it or not, sometimes people will share it. Now google and bing prioritize scraping it because it is linked from here, it will be indexed fairly quickly, and chagpt will be able to find it.

Soon, when every open platform is just tokens and everything is generated, we will probably move to gated communities and directories, and it will be very difficult for the chatgpt to discover your content.

And even it can actually find it, I am not sure you want everything you create to be seen through the lens of a language model.

I honestly think that tokens are inhuman, and actually it is harmful for humans to consume tokens.

In gpt2 times I used to read gpt generated text a lot, I was working on a game to guess if the text is AI generated or not, and for weeks while I was working on it I had strange dreams. It went away when I stopped consuming tokens, in gpt4 age this does not happen as I am reading hundreds of times more tokens than back then, but I think it is just more subtle.

Now I use AI to generate thousands of lines of code per day, at minimum sometimes now I just blank out when the AI doesnt spit out the code fast enough, I dont know what am I supposed to write, which libraries it is using what is the goal of this whole function etc, as it is not my code, it is foreign and I honestly dont want to be reading it at all.

This week I took the whole week off work and am just coding without AI and in few days the "blank out" is gone. Well, I did use AI to read 300 page docs of st7796s and write barebones spi driver for example, but I treat it almost as an external library, I give it the docs and example driver and it just works, but it is somewhat external to my thought process.

People argue that all fields have evolved, e.g. there are no more blacksmiths, but I argue that the machinists now are much more sophisticated than the ones in the past, pneumatic hammers allow them to work better and faster, as they use the hammer they get better understanding the material they work with, as in the machine does not take away their experience and ability to learn. I always had 2 days per week where I code without any AI, but now I think I have to change the way I code with it.

AI is for sure making me worse, and lazy. And I am not talking about the "career" here, I am talking about my ability to think.

I wrote few days ago about it: https://punkx.org/jackdoe/misery.html

https://www.decisionproblem.com/paperclips/index2.html go ahead :)

btw thats why I hated the old butterfly keyboards, a tiny crumb gets in and then the probability of a key drops to like 80%, or sometimes it double presses.

I am very happy that apple moved away from the quest of making the keyboard 0 height.

For me its not how fast I can type, my brain(ADHD or whatever this is) needs certain flow of speed; I think of something and want to type it, if I am delayed (e.g. imagine I change from qwerty to dvorak and am still learning) for whatever reason, e.g. key is stuck, my mind goes somewhere else, and then I have to "bring it back".

Also as I am typing one thing I am thinking of another, or even third, so if something goes wrong with the mechanics of the first thing, then whole set of dominoes fall and I have to go back.

Sometimes for a given thought I even type 3rd of 4th word instead of the first if for some reason my keys are not pressed. And then I type a word from the next thought and etc. It even gives me anxiety.

As I am typing the code, I am thinking about the code that is going to use the it, and then the code one layer up, I also think of the machine itself, its wires, cache lines, or I think of how the variable I am typing now is related to its surroundings, I imagine a ladder to the moon each step with its address, and I think how far are things from "me", and so on.

I cant speak of others, but for me keyboard layout and even key feeling/rythm is important just because it allows me to think uninterrupted.

Reading and "accepting/rejecting" so much AI generated code in the last months has made me a bit burned out and have somewhat of an identity crisis.

I have been making more and more time at night to reconnect with the craft.

I think more people in the community might be struggling, so I just wanted to share my way of dealing with it.

Please share how you are dealing with the tokens.

I am super conflicted about this kind of AI. I want artists to create the next amazing season of Solo Leveling, but I dont want to wait 1 year for it.

You could argue that those tools in the hands of skilled craftsman will create amazing things faster, but we all know what will happen is absolute flood of AI slop in every entertainment category.

well there is https://collapseos.org/ :)

R. G. Loeliger Threaded Interpretive Languages Their Design And Implementation[1] is an amazing book, since it was out of print, I printed it on a good 160gsm a4 paper, and I randomly open it every few weeks just to read through it. I strongly recommend it, even if you are not interested in Forth.

I have been programming in all kinds of languages, from assembly to clojure, but in 25 years I never programmed stack languages, I was kind of scared of them, it wasn't until I read the book and made my own Forth I understood what I was missing. Since then I made few interpreters, with jit, or with types, etc, it was super fun, but most of all it allowed me to see a completely new paradigm of programming, kind of the first time you understand eval/apply from 13th page of the LISP 1.5 Programmer's Manual. A language that writes itself and it is written in itself.

If you are making your own Forth, this Brad Rodriguez's article is also really good [2].

[1]: https://archive.org/details/R.G.LoeligerThreadedInterpretive...

[2]: https://www.bradrodriguez.com/papers/moving1.htm

then you will be even more impressed by Rule 110 being turing complete with a specific background pattern.

https://en.wikipedia.org/wiki/Rule_110

    | Name                                 | Semi-private eval | Public eval |
    |--------------------------------------|-------------------|-------------|
    | Jeremy Berman                        | 53.6%             | 58.5%       |
    | Akyürek et al.                       | 47.5%             | 62.8%       |
    | Ryan Greenblatt                      | 43%               | 42%         |
    | OpenAI o1-preview (pass@1)           | 18%               | 21%         |
    | Anthropic Claude 3.5 Sonnet (pass@1) | 14%               | 21%         |
    | OpenAI GPT-4o (pass@1)               | 5%                | 9%          |
    | Google Gemini 1.5 (pass@1)           | 4.5%              | 8%          |

https://arxiv.org/pdf/2412.04604

> This isn't what the article says...

thats not the only thing it says, but it does say it:

> The conclusions to be drawn from the points made throughout this paper are reasonably obvious: We need a new language for teaching novices at secondary school and introductory university level.

It also does states that somehow we know how to teach programming, and the discussion is around finding an incremental improvement given the learnings so far.

I never went to university, but in high school I had two different teachers, and one was hundreds of times better than the other.

I am not really sure how teachers get hired, how they are interviewed and assessed, but the system is clearly not working; particularly in subjects that the learning curve is not a curve (like math, or physics or cs)