mbowcut2

So, is this just an example of the first-mover disadvantage (or maybe the problem of producing public goods?). The first AI models were orders of magnitude more expensive to create, but now that they're here we can, with techniques like distillation, replicate them at a fraction of the cost. I am not really literate in the law but weren't patents invented to solve problems like this?

Me thinks he doth protest too much.

I agree that the moats are weak for applications, but I think there are possible strategies to capture users. One way is to make it difficult to switch, similar to Apple Music vs Spotify or iPhone vs Android. Although these platforms offer very similar features, switching has high friction. As an example, I can imagine an AI app that has been adapted to your use cases, maybe it has a lot of useful information stored about you and your projects which makes it much more effective, and switching to some other app would mean you have to move all this data or possibly start over from zero.

DeepSeek has demonstrated that there is no technical moat. Model training costs are plummeting, and the margins for APIs will just get slimmer. Plus model capabilities are plateauing. Once model improvement slows down enough, seems to me like the battle is to be fought in the application layer. Whoever can make the killer app will capture the market.

> The vast majority of our work is already automated to the point where most non-manual workers are paid for the formulation of problems (with people), social alignment in their solutions, ownership of decision-making / risk, action under risk, and so on.

I agree. That's why I think the next step is automating trivial physical tasks, i.e. robotics, not automating nontrivial knowledge tasks.

oh, I think the horror element is there.

Absolutely horrifying. Well done.

This was my first thought too. AFAIK each layer encodes different information, and it's not clear that the last layer would be able to communicate well with the first layer without substantial retraining.

Like in a CNN for instance, if you fed later representations back in to the first kernels they wouldn't be able to find anything meaningful because it's not the image anymore, it's some latent representation of the image that the early kernels aren't trained on.

Brand new account + strange overreaction = maybe bot?

This is my main point. Seems like gripes about the quality of published research should be directed toward the publisher.

I understand the complaints, but I don't think the Freakonomics podcast should necessarily be expected to meet the high standard of peer review. Is it really that bad for Levitt to take published research at face value, trusting that gatekeepers upriver have done their due diligence?

Making things easier to google is IMO the least-impressive use of LLMs. I'm still waiting for a Roomba I can have a conversation with.

Numberphile did a video on this a while back. https://youtu.be/mceaM2_zQd8?si=0xcOAoF-Bn1Z8nrO

I guess we need to find a way to incentivize good practice rather than interesting results? Turns out that science is so hard that people cheat.

The biggest difference is when you feed a sequence into a decoder only model, it will only attend to previous tokens when computing hidden states for the current token. So the hidden states for the nth token is only based on tokens <n. This is where you hear the talk about "causal masking", as the attention matrix is masked to achieve this restriction. Encoder architectures on the other hand allow for each position in the sequence to attend to every other position in the sequence.

Encoder architectures have been used for semantic analysis, and feature extraction of sequences, and encoder only for generation (i.e. next token prediction).

THIS. People don't realize the importance of Mamba competing on par with transformers.

Touché. I put the first paragraph in GPTZero, and it says 100% AI generated. Unfortunately, AI detectors are notoriously bad, giving all kinds of false positives, which goes to my initial point that it is largely a gut feeling. But if you look at other articles posted on the site by the same author they also come up as 100% AI generated on GPTZero, and they also fail the gut test.

I’m similar. Never been very over weight but I’ve put on 20ish lbs in the last few years. My wife and I started tracking calories and exercising a month ago and it has been so beneficial. I lost about ten lbs, but even better than losing the weight has been building a better intuition of what foods are calorie rich, higher in protein, etc.

oh I just realized that 'isNew' in the response refers to a global set, not the user set. So, I guess it's doing exactly what I said lol.

This is fun. It would be interesting to build a single graph of concepts that all users contribute to. Then you wouldn't have to run LLM inference on every request, just the novel ones, plus you could publish the complete graph which would be something like an embedding space.

This makes sense. Add in that he was featured on the JRE (which is where I first learned about him) which no doubt gave him a nice audience boost.

I've been long time listener of his show, so I don't necessarily disagree with your points. It just seems like he punches above his weight when it comes to guests.

This is perfect. We need more hobbyist apps. Minimalist design, simple and solid functionality, and no ads in sight. Shake off the shackles of the advertising dystopia!

Next they'll announce GPT-4 is the new CEO.