seamossfet
As a result, I don't think LLMs will reach this state easily.
If there is a TV at a restaurant and the kid sees it, do you tell them to turn it off for you?
If you walk by the electronics section at a walmart do you demand the workers turn off all the TVs before you pass by?
Do you scout every location prior to visiting to make sure you avoid TVs at all costs?
My point is the framing of kid cannot even _hear_ the TV without ruining the child is ridiculous.
The way the post is framed, it makes it seem like you have to completely remove all TVs from the child's life to be consistent with the medical literature which doesn't seem correct.
That's a ridiculous false equivalence and you know it.
Choosing a poor password is nowhere near analogous to the investment and energy requirement of creating completely TV-free spaces for children.
I'm questioning whether that degree of separating is consistent with the medical literature, not whether we should "do something bad because we've always done something bad"
Parenting is hard, and strangers are always brutal judges of you.
Best not to concern yourself with the opinions of strangers.
Like sure, of course, don't let the kid just mindlessly watch TV. But removing the TV entirely from all spaces they could possibly be in is too extreme and the benefits of that are not clear other than expressing the parents' own maniacal anxiety which (in of itself) is not a good thing to teach the child.
I don't think Grassmannian manifolds are necessarily a better representation than simple query/key inner products. It's a valid solution, and it does have advantages in some cases.
The Grassmannian is only preserving the subspace orientation, so we lose information about the actual linear map and basically throw out its basis.
Like everything, this is a trade off. You're trading information for a lower rank representation.
I'm still reading through the paper on the implementation side, but the other caveat I'd be wary of is compute. Does this projection add MACs to the overall model which would make inference slightly slower even with lower rank representations? I don't know. Like with any technique that trades constrained information representation for space/compute you _are_ widening your search space during training. I'd be curious what the impact of this is on relative data needed to train a model using this architecture versus "regular" attention.
QA through entropy lol
Meetings that could have been an email, in some ways, are replacing hallway and water cooler conversations. Are they a good replacement for that? Not really.
The main reason people don't use Huawei hardware is due to regulation. If you're in the US you'd straight up get your hardware seized by customs because they're illegal to buy or bring into the country. EU has similar guidelines I believe.
Nvidia will continue to have its monopoly as long as the powers that be mandate it.
Matters more for 1-bit and 2-bit models.
There's literature on this actually, if you want to get that low the model really should have quantization as a pretraining target otherwise larger models that are quantized after collapse.
In fact, to do this effectively, you need somewhere near 50x chinchilla to do QAT on super tiny targets like 1-bit or even 2-bit. These large models already need an astronomical amount of training data that doing proper QAT that small isn't really feasible unless there are breakthroughs in the architecture.
4-bit models and smaller _can_ perform well but not through just naively quantizing the existing weights of a large model.
It's not that simple, you have to remember at the end of the day these things are just doing next token prediction. If you don't give it the proper tokens to attend to, then your outputs won't be satisfactory.
You can get stellar outputs from LLMs, but it really is a function of how well you manage your input tokens.
"on the fly embedding" and "Sparse + dense reranking" don't really make sense how they're presented and smell like they came from a long claude-driven conversation after multiple cycles of these hybrid compromises across many turns.
Looks interesting, but I'd like to see some reviews on the content first before buying.
A lot of people in the comments do have a software engineering background. People at different skill levels in different backgrounds are going to be using these tools in different ways, and that's going to heavily impact their experiences with these models.
Sure, there are differences between Fable and Sol. But I've even seen people on here saying that they're getting better mileage out of Qwen models they're self hosting.
I think the driver is just as important than the car, when it comes to this sort of stuff.
I feel like to get to Terry's level you need a combination of passion and aptitude for the subject. People that don't want to learn about a topic will always look for shortcuts, which I think represents the vast majority of people. Terry Tao is quite exceptional, and I think exceptional people will still exist even when the "easy" button is bigger than it's ever been.
I don't think chatgpt could have come to this on its own without the amount of steering he did, which just validates the idea that AI is not a replacement for human expertise but an amplifier.
From a data structure and file ergonomics perspective, think of it as similar to Unity or UE4 for drug design. We have a huge variety of assets to manage alongside their relationships to each other, and the project files are local on the user's machine (with a collaboration / sync over the network between scientists working on the same project, hence where something like this would come in for us).
Many of those files are fine with a winning side strategy, but some of them might not be that clean. Take a protein structure defined by an `mmcif` file for example, if we clean the file by removing hydrogen atoms and another scientist repairs a side chain on that same file then we'd need a way to reconcile those differences.
On the agent side, our agents will generate small python scripts that manipulate the proteins, then cache and re-use those scripts as tools when possible. So preserving those scripts alongside the mutated asset and conversation history is something we've been working on.
I have a use case that could use this if it supports handling branching and merging file systems.
It's really hard to do surgical changes with an AI agent, and it's even harder to review those changes. Even if I'm reviewing the specs and the code, the cognitive load on reviews feel like they've ballooned from what used to be a few hours to now taking me days to review these PRs.