jmatthews

I built a smaller scale replication. Most of your claims prove out but the lever is essentially Biderman et als "learn less forget less". My first pass actually made this mistake initially and the findings didn't replicate. After controlling for step density I replicated your findings. Here is a write up:

https://huggingface.co/spaces/dreddnafious/mini-agi-replicat...

At the bottom of my write up I added some ideas to pin down the actual dominant parameters. I also added a pr to fix an issue in the codebase:

PR: https://github.com/volotat/mini-AGI/pull/20

What it fixes: a crash in upstream's GradSNR meter (issue #19), a diagnostic that tracks how much of the gradient is signal versus noise.

This is an interesting approach. First of all, thanks for sharing your work. I've done. I want to say similar work in that I have trained continuous learning models and I have also offloaded parametric knowledge to hard drive people underestimate how difficult that is to do in a functional model. I look forward to digging in deeper.

Without commenting on the EU, particularly, I would be concerned if I were a Canadian about Carney's commitment to globalism when the degenerate cases have become so common.

It's a weird two-front war where people tend to answer the social question and ignore the economic one, but the economic question has not done any favors to the EU or the US for that matter

It's always weird hearing yourself described and recognizing approximately nothing, right?

The hubris to think a group of people could precipitate the rapture is just awe inspiring and the hubris to reduce someone's relationship with the creator to some Wikipedia excerpts is a bit less awe inspiring.

That's great compensation. I'm glad justice was served but couldn't have happened to a bigger piece of crap.

Well done

I learned some of this the hard way. I did the thinking and the distillation, but I had AI write the prose and that's all a lot of people saw, the AI tell-tales.

it's a worthwhile lesson. thank you. There was a great deal of effort on my part, but not in the prose. You've taught me something and I appreciate it.

it's not that you're teaching the AI, it's that you're framing the conversation on a reference material and having a conversation around it. Exploring a problem with referential framing, like a white paper or a dense blog post is a useful cognitive hack. You just have to be careful to pin extraordinary claims to extraordinary evidence.

I'm trying to untangle the "this content isn't valuable" signal from the luddite "Anything with AI is low effort slop"

I appreciate the former and am trying to filter the latter.

I'll give a software example because I'm just better at it, but if you describe Trello and tell me to decompose it on a white board I start thinking in terms of queues and write|edit|save tuples. I don't have to invent queues in my head and I have to wonder if it is possible to proactively assign a series of tranforms or schedules on an fresh input.

I know how to do it and it's all internalized. Even if I've never needed to do it.

that's the toolbox I'm trying to develop in ML. For example. I've studies LSTM and implemented one. What I didn't know was if gating was turing complete and essentially unbound. I didn't know if gates could be arbitrarily complex. Importantly I had no idea how to translate "I need a switch here" to a gate, or if a switch was even possible given the need to be differentiable for backprop.

It's not a trick bud. The github page shows my user name and Claude. The content is intended to be read by an AI agent and explored through a text interface. That is explicit in the readme and the primer itself.

If you think you can generate this artifact with a prompt then show me. This was 2 days of exploration and research.

my apologies it wasn't up to your standards. In fairness to me, that line is exactly what my effort was. I wasn't trying to "learn ML" I am trying to build a mental model that let's me decompose real problem into ML primitives.

It's unclear to me if you think the resource has no value or if it bother you that I wrote it using a coding agent.

I wrote the syllabus and worked through each section. Where my understanding was weak I explored the space, pulled in research, referred the model to other sources, and just generally tried to ground the topic in something I understood.

What resulted was something that helped a lot of subjects click together for me. Especially when to reach for a particular activation function and the section on gating.

This enter survey was motivated by an ML expirent I ran with assosicative memories that just failed horribly. So rather than post mortem that I set about understanding why it failed.

Anyhow, thank you for the feedback. I submitted this in good faith that it may help others.

I've bounced off of many good textbooks. Even Karpathy's YouTube series was too dense for me. I'm trying to come in at a more palatable level.

This was a two day exploration where I provided the syllabus and ran through it with Claude Code, asking questions, trying to anchor it to stuff I understand well. I feel like the artifact has value.

I tried to tackle that under Topology for the problem but it may not be well named https://github.com/dreddnafious/thereisnospoon/blob/main/ml-...

I'm just trying to develop the lens where I can see a problem and know what properties of it are meaningful from an ML standpoint.

Coming from a specific domain where I have a sharpened instinct for how things are haven't really given me the ability to decompose the problem using ML primitives. That's what I'm working on.

thanks Oleg

This is my weekend project. I am building up my pattern recognition in machine learning. By that I mean see X problem, instantly think of Y solution. The primer markdown file is the artifact of that exploration.

read it from top to bottom or better have your favorite language model read it and then explore the space with a strong guided syllabus.

For me getting the AI harness to solve the puzzle has become the puzzle. Just one more level of abstraction. Just like I don't code in assembly language anymore, now I don't name every variable.

AI coding is just a non-deterministic programming language and compiler duo

I find this so hard to get my head around. I am wildly more prolific with agentic coding. It's at minimum a 10x for the first several iterations and when you get into the heavy detail part I am still the choke point.

How many hours do you have mastering git or your IDE or your library of choice for UX?

Even if the round trip efficiency is 25%?

Seasonal storage at competitive prices per megawatt hour is somewhat of a solved problem it just doesn't seem to be getting investment.

In the same way that black hats will always be advantaged versus white hats, frontier models will always be advantaged versus derivatives.

You essentially outline why it should be broken up.

I'm not convinced making the ad tech sector more competitive would prompt that outcome but, "It would disrupt mature products" isn't a compelling argument to allow the existence of a monopoly.

Google is a monopoly, they exert monopoly power and enjoy monopoly pricing.

I think the more likely outcome would be more dynamic products under smaller bannerheads.

I would like to post this respectfully, just for posterity. The anti science insular concepts being assumed as fact, the discredited overpopulation theories, the sky is falling parts of climate change.

I can't credibly participate because from my perspective what is being discussed is a popular sci-fi series that I haven't read. The dogmas and rituals are alien to me.

It is insane how tone deaf most of the comments are, jarring even. I forget how insular so much of the community is and I often make the mistake of equating intelligence, which exists here in spades, with critical thought.

Understand that most of the sentiment expressed here is identical to the pre-digested mass media pablum intended for 100 IQ consumers.

Think of mass media coverage of a subject you're an expert in and how horrifyingly wrong it typically is on so many levels, and try to rectify the two.

I've read the couple of critical responses but on merit, the message is true. Anecdotally, my wife takes hundreds of photos every month that essentially no one will ever see again.

Android has a memories feature that serves them back up to us on occasion. This is a pattern writ large for huge swaths of data.

Differences in governance or allowable access leads to mass duplication and data rot on anything remotely dynamic.

I was that kid, now I have 3 sons and I am homeschooling my oldest because he is similar. The hack is essentially to praise the work, not the outcome. On some level you have to be unfair to your kid to be fair with him.

Any non paywall version?