machinelearning

Training is expensive so I wouldn't necessarily call it "pragmatic".

- the tool's goal is actually to provide a lightweight, practical way to avoid wasting training cycles on bad data.

Evals for robotics are also expensive.

- validation loss is a poor proxy of robot performance because success is underconstrained by imitation learning data

- most robot evals today are either done in sim (which at best serves as a proxy) or by a human scoring success in the real world (which is expensive).

It's great if you have evals and want to backtrack (we're building tools for that too) but you definitely don't want to discover you have bad data after all that effort (learned that the hard way, multiple times).

The metrics the tool scores vary from tedious to impossible for a human to sanity check so there's some non-obvious practical value in automating some of it.

"Most ways of constructing accurate models mathematically don’t actually work" > This is true for almost anything at the limit, we are already able to model spatiotemporal dynamics to some useful degree (see: progress in VLAs, video diffusion, 4D Gaussians)

"We’ve known for well over a decade that you cannot cram real-world spatial dynamics into those models. It is a clear impedance mismatch" > What's the source that this is a physically impossible problem? Not sure what you mean by impedance mismatch but do you mean that it is unsolvable even with better techniques?

Your whole third paragraph could have been said about LLMs and isn't specific enough, so we'll skip that.

I don't really understand the other 2 paragraphs, what's this "dark state-of-the-art computer science" you speak of and what is this "area of research became highly developed over two decades but is not in the public literature" how is "the computer science required is so different from classical AI research"?

> The reason is that it makes the essay easier to read. It's less work to read writing that flows well. How does that help the writer? Because the writer is the first reader

Note that the writer's judgement only serves as an initial proxy for how well the essay reads. This implies that the reader, whoever that is, is the true judge of how well it reads. My point is that that group is ill defined.

If it were sufficient for the writer to be the only judge of how well something reads, surely PG wouldn't feel the need to have other proofread his essays. And surely it is not sufficient for someone who lacks taste to judge their own writing as good.

The way I read that statement is the same as the startup advice of "build what you would yourself want". However you still have to validate that the market exists and is big.

There is really nothing profound in that paragraph anyway, all it is saying is that a writer should edit and proofread their work. That whole paragraph could be deleted honestly. It is obvious table stakes for one to edit their work. What differentiates good from bad is a matter of taste + who is judging it.

So one thing to note is that the essay mentions that it refers specifically to "writing that is used to develop ideas" vs. "writing meant to describe others ideas".

The way I interpret this is that it refers to claims that build on each other to come to a conclusion. So the way to test for truth is to somehow test each claim and the conclusion, which could vary in difficulty based on the kind of claims being made.

As this essay exemplifies, it is difficult to test for truth if you make broad claims that are so imprecise that they can't be verified or don't tell you anything interesting when verified using reasonable assumptions.

The issue with this article is that it is very imprecise.

Are the standards for whether something “sounds bad” based on the average person’s reading or the intended audience.

In its most general form (how the median article sounds to the median person), the argument is pretty vacuous.

Most writing discusses simple ideas and they should sound good (familiar, easy, pleasurable) to the median person.

But the most valuable kind of writing could sound tedious and filled with incomprehensible terminology to the median person but concise and interesting to the intended audience.

The current way the idea is stated doesn’t sound correct because you can convincingly defend all 4 quadrants of the truth table.

> “So here we have the most exciting kind of idea: one that seems both preposterous and true.”

Am I missing something or is the “seems true” part taking too many liberties here?

If anything, as described in the previous few sentences, the premise seems false, not true.

Kind of ironic since the line sounds right but isn’t rigorously right, so it undercuts the main argument.

What's a use case are you interested in re: video?

This is a good problem to solve but the approach is wrong imo.

It has to be done in a hierarchical way to know what you attended to + full context.

If the differential vector is being computed with the same input as the attention vector how do you know how to modify the attention vector correctly

No, you've only discussed the Retrieval part of RAG, not the generation part.

The current workflow is to use the embedding to retrieve documents then dump the text corresponding to the embedding into the LLM context for generation.

Often, the embedding is from a different model from the LLM and it is not compatible with the generation part.

So yea, RAG does not pre-compute the KV for each document.

Nope that’s not how most rag systems work today. I looked at pgvector and couldn’t find anything similar.

Do you have a link? Or maybe you misunderstood what I was taking about

Both RAG and infinite contexts in their current states are hacks.

Both waste compute because you have to re-encode things as text each time and RAG needs a lot of heuristics + a separate embedding model.

Instead, it makes a lot more sense to pre-compute KV for each document, then compute values for each query. Only surfacing values when the attention score is high enough.

The challenge here is to encode global position information in the surfaced values and to get them to work with generation. I suspect it can't be done out of the box but we it will work with training.

This approach has echoes of both infinite context length and RAG but is an intermediate method that can be parallelized and is more efficient than either one.

This is a simple version of the tree search approach that people suspect Q* is

As someone who studied math, I approve of this video.

Summary available here: https://www.askyoutube.ai/share/6533668f27f67cda427ab90f

Its much faster now and a share function has been added for pre-loaded examples

Added

Added

That's reasonable, planning to add it tomorrow

Thanks for the feedback, I haven't done much optimization in regards to the summary.

The screen real-estate of the summary has to be small so it is limited in the amount of information it can display but your example could be a good starting point to see how the key points can be better captured

Glad that it worked well for you! I don't have any general advice that would be useful but that sounds like an interesting project

Fair point, will add this tomorrow

In other words, a ‘hack’

If you're trying to use this and don't get the sign up email, check your spam folder. Gmail seems to auto-categorize the email as spam

The tech that you know has changed. We've moved well beyond programming languages and frameworks. Once you adapt to the frontier, your eyes will be opened to the revolutionary technology being developed right under your nose.

Having had quite a bit of experience on both sides of the equation, you’re absolutely right.

The very act of extrapolating how a person would perform based on a brief snapshot in time seems flawed. It is unreasonable to task the interviewer with making a good decision with barely any data.

I used to think that a well calibrated question bank or interview style could be a reliable signal. But the moment you ask something specific is the moment you don’t ask everything else. A poor performing but good candidate’s strengths could lie in the latter category and a well performing but bad candidate’s strengths could be in the former.

There needs to be a reliable and credible way to learn about a persons strengths and weaknesses based on their track record.

I’d weight credible reference checks, previous projects and credentials more than their actual interview performance.