joefourier

Stereo cameras/camera arrays can have very large resolutions. An ML model is necessary for optimal results with stereo cameras, but it's more grounded in reality and verifiable versus a mono capture. With a large enough camera array, you can get precise, high resolution depth without ML, especially if you accept Gaussian splats, which use gradient descent but I wouldn't call them generative models in the same sense. Large, featureless, textureless planes are still a weakness however, while generative models can guess/interpolate values.

You can use a hybrid method with speckle projection/ToF/LIDAR (the first two behind mainly usable indoors) if you want to stay fully grounded though.

It's never been easier and more accessible to be an enthusiast hacking various stuff together though, from hardware to software. You don't have to rely on subscription services. Open-source software has never been better (see Blender, Kicad, Photopea). You can easily create custom software for your one-off needs using LLMs, and you don't have to rely on a bigco subscription either - pay API prices for open source models, or self-host. Run your own private cloud for your vacuum robot, reverse engineer your smart TV, deploy your own camera firmware...

You don't even need to rely on a single centralised corporate search engine for all your information look-up. 10 years ago, it would have been unimaginable that you could self-host something that could actually replace or be better than Google.

How many people are willing to pay more for America/Western-made products? How many are able to correctly know which companies are honest, and which are finding loopholes (e.g. bringing in underpaid temporary workers from abroad, finding the bare minimum for a product to be qualified as "Made in X", writing "Designed in America").

And how many people will be willing to pay a premium when the locally made products are worse quality than foreign products? It's happening right now with Chinese consumer goods (e.g. robotic vacuum cleaners), EVs, etc. Eventually there's a high possibility you won't be able to buy anything in the product category you want.

There's still tons of pure software engineering problems where iterations are time consuming, or there's no obvious solution, like physics simulations, DSP, computer graphics... I can trivially up the difficulty on any tasks that it can easily solve.

LLMs enable me to iterate quicker, accelerate and parallelise, research, scour the literature and integrate cross-disciplinary knowledge, easily implement more complicated algorithms, and are generally better at avoiding trivial mistakes.

Before LLMs, I can't believe how many hours I wasted debugging computer graphics or computer vision algorithms only to realise the problem was my handling of coordinates, flipping x and y, or one program needing the y axis flipped, or typos when copying the algorithm from equation to code. I have no nostalgia for that era.

But why would you throw away that data at all? These are tiny models compared to domain-specific audio/music/image models, and any capacity spent on recognition is capacity that could have been used for prediction.

Is there a reason you don't throw in something like cross-attention on an embedding of the sequence description/data labels? I've trained numerous generative models for audio and images and it's always the case that the unconditional model performs worse, trains slower, etc. than the conditional one (with the best performance from a hybrid using classifier-free guidance or similar scheme).

Again I don't have experience with the domain of time-series forecasting directly, but I would think there are generalities between autoregressive generative models and time-series forecasting, given that both involve predicting the next sample(s) given N regularly sampled past inputs.

My field is primarily audio/vision ML so I don't have any particular domain expertise, but I don't get how you can lump all "time-series" together in one big bucket. My naive understanding is that the way to improve the accuracy is to gain as much joint information as possible. Like:

> Known-future covariates. Some drivers of sensor behaviour are known in advance, such as production schedules, planned shutdowns, weather forecasts. Timestral can condition on them to improve forecast quality, so a scheduled load increase is expected rather than flagged as a surprise.

To me this should be the entire basis of the system, not a condition to improve forecast quality? Like if you're modelling electricity consumption, if all you have is a window of N previous samples, you're going to get completely beaten by a comparatively simple system that knows the time of day, that there's holidays coming up, the weather forecast, etc.

I would also think that these time-series forecasting models would gain from a sophisticated understanding at least the labels of the covariates? A 5M model has no idea what a "bank holiday" is, it just sees an arbitrary class go from 0 to 1, and won't have any useful inductive biases as to what it means or how to link them to other pieces of information.

E.g., wouldn't you predict €/MWh of electricity far more accurately with access to price of natural gas (itself influenced by many geopolitical factors), breakdown in composition of production of each energy source, predictions about roll-out (requiring a sophisticated LLM capable of understanding news and their implications), etc. What's the use in trying to predict it from 2 variables and a narrow window looking back at just 8192 or 32768 samples?

Or perhaps I misunderstand the use-case and this is more for extremely short term movements?

As a mere software engineer, I found that many human written papers in my field (DSP, ML) aren’t exactly well-written, easily understandable or pleasant to read either, and it’s been the case for at least 15 years. In the past, I would frequently spend an hour or two trying to read a SOTA paper full of convoluted prose and dense pages of equations, only to realise that the algorithm is incredibly simple if visualised properly or even just written down as pseudocode.

Now with LLMs the process is much shorter. They can walk me through the algorithm step by step, translate it to pseudocode, explain equations, and most importantly, I can ask them questions and get clarifications, without worrying about seeming ignorant or exhausting their patience. With a paper, you’re stuck with whatever was written (and often, researchers aren’t good teachers, or intentionally over complicate simple concepts).

The list of counter examples is some of the most popular and well-received posts in the rationalist community which are actually in favour of tradition having value.

"The Secret of Our Success" literally starts with how European explorers starved to death in environments that other humans thrived in because they lacked the cultural knowledge local hunter gatherers had, includes many other examples of how pure reason can be counter-productive to survival, and ends with a warning that the idea that now it's finally different may not be true.

I can get past the AI slop design myself, but the writing style is absolutely grating, especially the short repetitive phrases.

"Punchy phrase. No X." "An open engine. Room to make it your own. "Same scene. Same instant. The same explosion. A different atmosphere." "A few things to know. The short answers."

Why is it always the same? What awful post-training, or synthetic data training pipeline, or RLHF gone wrong, produced this homogenised, low variety slop style? And it's totally possible for LLMs to write actually decent or at least varied prose and I would think Anthropic (or OpenAI) has the resources to fix the issue, they just choose not to.

Marketing copy wasn't this homogenous prior to LLMs, unless my memory fails me.

> because you want a specific choice picked or list of things ranked and sometimes the llm will just output something else entirely!

With structured outputs, that just doesn't happen though?

Understandable, (V)AEs are their own rabbit hole (I've spent way too much time working with them). I think they deserve more attention, there's been a few good papers recently about how to ensure that the latent space is optimal for generation without sacrificing too much reconstruction detail, but it's no easy task.

But how do you plan on scaling to 1080p24 (or higher resolution) video with end to end pixel space training? For a 10 second long video, that's 1920x more tokens than your 512x512 DiT experiments, and I'd be concerned about scaling linear patchification temporally and spatially past a certain point.

VAEs don't have to inhibit fine detail, you can trivially increase the channel count like the Flux VAE did, but if you're not careful, that will make training on that latent space much more difficult.

Older models like the SD1.5 work like you mentioned, in large part due to relying on PatchGan which just ensures that individual patches are plausible in isolation. There's nothing preventing a VAE from rendering tiny text with the correct architecture and training. If pushed to an extreme, it could essentially start guessing the letters themselves and write different, but still legible words instead of nonsense scribbles.

I'm surprised you haven't heard about Natten, it's a few years old now: https://natten.org

It's the equivalent of sliding window attention, a common LLM optimisation. Global attention may be O(n2), but you can avoid that bottleneck by using local attention, even throughout the whole model. This could give you another efficiency boost, or enable you to process larger resolutions with linear instead of quadratic compute scaling.

I wonder how much of that is due to the use of a suboptimal VAE? The same optimisations could be applied to it, and my intuition tells me that your total compute spend would be even more optimal if you retrained the VAE yourself with a better approach (esp. ensuring translation and rotation invariance, + ability to rescale/blur the latents).

You could also have the same advantages of a draft in low resolution latent space, with an easier to learn data distribution that's more robust to perturbation, but instead of 512x512, you could get a 4096x4096 output for the same compute (assuming a 8x VAE).

There's also the advantage of being able to use a high number of diffusion/flow matching steps for the main LDM while the VAE can be single step and much smaller, since it does not have to handle language or significant scene understanding, just perceptual compression. This sounds especially important for a video model where I would be extremely hesitant to train a generative model without relying on interframe compression.

While I don't know if AI is conscious, I'm sure it has an internal world. Isn't that what a latent space literally is? LLMs operate in a high-dimensional space that can be analysed and translated to very abstract, high level concepts, behavioural tendencies, even things recognisable as emotions. In fact any neural network beyond a single-layer perceptron has a sort of "internal world", it's a matter of complexity and degree.

I don't see how that's true, you can absolutely do the same thing on the desktop? No one is forcing you to use the OS's default widgets there either, you can use Qt, Flutter, or even paint every single pixel yourself which is a lot less feasible on the web.

The main reason the web won is that you can ship SaaS without the user having to install anything locally, and there's no expectation of it being able to work offline, so you can offload processing to servers resulting in fewer bugs, compatibility issues, and DRM problems.

How does that make sense? The expertise is what allows me to tell the LLM what to do in the first place, inspect its outputs, and iterate, which is still faster than coding by hand.

> That's why Claude is so cheap for what it does.

The main reason Claude is "cheap" because it has to compete with OpenAI, Deepseek, GLM, Qwen, Muse (who is also massively subsidised their model to get training data), etc. If they increased prices 10x, people would instantly jump ship to the next competitor. There's no moat for Anthropic either, short of constantly training new models in a red queen's race.

I use LLMs every day as a search engine replacement, but I’m concerned about this because there’s a vast gulf of capabilities between different models, or even the same models with different reasoning depths.

Back in the day everyone had essentially the same Google, and there was no concept of paying to get better search results. But now the average family is going to rely on the ad-ridden ChatGPT Free tier which is more likely to be wrong, less in depth, with additional commercial incentives. It’s only a minority of adults now that pays for frontier models, and how many are letting their children have access?

You can see this divide right now with medical advice for instance. If you have access to ChatGPT 6 Pro (only available for the $100/month tier I think), it will research Pubmed etc and give you a careful measured answer that likely reflects the latest medical consensus, while a casual user relying on Google AI might be told to eat toilet paper to relieve constipation: https://www.reddit.com/r/medicine/comments/1wmbn87/a_pt_show...

Ah, are you saying that because most people don’t interact with agents, they aren’t aware that LLMs can have initiative and pursue goals?

I think the line is blurring though, mainstream chat interfaces are adding more and more “agentic” features.

ChatGPT will happily execute code in a sandbox, search the web and design downloadable PDFs purely through the standard OpenAI chat interface. They can also send you emails or do tasks on a repeated schedule.

We are way past that point, any harness can trivially make LLMs start conversations or pursue goals. An encyclopaedia wouldn't have hacked Huggingface on its own.

There's no need for black and white thinking. Javascript and the internet browser are the most common interface sure, but there's still room for specialised desktop software, especially those that require serious performance like anything to do with 3d graphics or real-time audio.

But also, frontier LLMs are enormously expensive and slow. Using Astra for things like simple text classification is not going to scale, and you're likely to end up in the same boat as those people who saw their Vercel bill shoot up to $96k/week when their site got traction, if not worse.

While the idea of improving codebase search and documentation has some merit, any modern coding agent harness does not have the problem of starting each session blankly and giving generic advice. The actual issue that I have experienced is that it will waste tokens re-reading the codebase over and over again to understand your query, although you can mitigate that by proper use of AGENTS.md and documentation.

But more importantly, is it too much effort to write the readme yourself? Or at least to ask the AI to rewrite it so that it doesn't have all the tell-tale signs of AI slop?

> current frontier models

> Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1

The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.

The app not existing would unironically be better in so many instances, though. The web version doesn’t take up 1GB of disk space, install persistent services, send you push notifications by default, and it’s trivial to block ads in comparison.

I’d be very happy if companies didn’t artificially degrade their web version to force installation of an “app” that’s effectively a web browser in disguise.

> It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always performed the best at all tasks. Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence.

Absolutely false. At least when it comes to multimodal inputs, even a simple classifier will outperform the largest LLMs who still hallucinate details or don’t describe audio and images accurately.

And there’s also the issue of cost/inference speed. Running a trillion parameter model for all tasks will be incredibly costly, require a cloud API, while a tiny CNN can be run locally or at a cost multiple orders of magnitude lower.

What about multi-token prediction and speculative diffusion? That’s a different mechanism of prediction, even if it serves only to accelerate decoding.

You cannot just "try all possible optimisations". It takes time, effort, and money that could otherwise be spent elsewhere (especially for training, where each training run is especially costly, and optimisations might be promising early on, but cause the final performance of the model to be worse). You need smart people interested in unglamorous work, and if you're swimming in VC money, it's far more straightforward to just throw more GPUs at the problem.

Anthropic's API has two nines availability and Claude Code is a TUI made with React that can regularly consume more than 1GB of RAM, and the codebase is utter slop. They couldn't fix the flickering bug for over a year!

And yet, Fable and Opus are among the best coding models out there (matched only by GPT5.6 Sol).

It's not about the people there being smart or not, it's about their and the company's priorities, resources and what they choose to focus on.

> Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture?

I'm not sure what you mean? You can see the intelligence of LLMs progress predictably and stably according to scaling laws. LLMs have to encode language in addition to intelligence so there's a minimum bound for them to output sensible text (you can train specialised tiny models to solve basic puzzles without language). Start at around 127M and compare models of increasing parameters and you'll see a clear progression in intelligence.

> It's unintuitive since, to the best of my knowledge, one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems, e.g. naive sorting algorithms suddenly won't beat quicksort if you put more processing to them

How is that a basic tenet? Simple, easier to parallelise algorithms that have lower memory requirements, or can take better advantage of hardware, or don't hit a plateau the more compute you throw at them, can absolutely beat cleverer algorithms. E.g. brute forcing rendering with Monte Carlo path tracing will give you more physically accurate results than ray tracing or rasterisation algorithms that rely on a bundle of hacks to approximate global illumination, transparency smooth shading, etc.