News for hackers. Stuff that matters.

A comment on Google Titans architecture, helping AI have long-term memory. Parent comment

Re:Google Titans architecture, helping AI have long-term memory by alyxya on Sunday December 07, 2025 @11:34AM (#46182946)
The hardest part about making a new architecture is that even if it is just better than transformers in every way, it’s very difficult to both prove a significant improvement at scale and gain traction. Until google puts in a lot of resources into training a scaled up version of this architecture, I believe there’s plenty of low hanging fruit with improving existing architectures such that it’ll always take the back seat.
[ Parent | on HN ]
Re:Google Titans architecture, helping AI have long-term memory by UltraSane on Sunday December 07, 2025 @12:11PM (#46183227)
Yes. The path dependence for current attention based LLMs is enormous.
[ Parent | on HN ]
Re:Google Titans architecture, helping AI have long-term memory by patapong on Sunday December 07, 2025 @02:10PM (#46184174)
At the same time, there is now a ton of data for training models to act as useful assistants, and benchmarks to compare different assistant models. The wide availability and ease of obtaining new RLHF training data will make it more feasible to build models on new architectures I think.
[ Parent | on HN ]
Re:Google Titans architecture, helping AI have long-term memory by p1esk on Sunday December 07, 2025 @02:36PM (#46184404)
Until google puts in a lot of resources into training a scaled up version of this architecture

If Google is not willing to scale it up, then why would anyone else?

[ Parent | on HN ]
Re:Google Titans architecture, helping AI have long-term memory by 8note on Sunday December 07, 2025 @08:44PM (#46187379)
chatgpt is an example on why.
[ Parent | on HN ]
Re:Google Titans architecture, helping AI have long-term memory by falcor84 on Monday December 08, 2025 @10:23AM (#46193265)
You think that this might be another ChatGPT/Docker/Hadoop case, where Google comes up with the technology but doesn't care to productize it?
[ Parent | on HN ]
Re:Google Titans architecture, helping AI have long-term memory by tyre on Sunday December 07, 2025 @03:11PM (#46184696)
Google is large enough, well-funded enough, and the opportunity is great enough to run experiments.

You don't necessarily have to prove it out on large foundation models first. Can it beat out a 32b parameter model, for example?

[ Parent | on HN ]
Re:Google Titans architecture, helping AI have long-term memory by swatcoder on Sunday December 07, 2025 @03:48PM (#46185008)
Do you think there might be an approval process to navigate when experiments costs might run seven or eight digits and months of reserved resources?

While they do have lots of money and many people, they don't have infinite money and specifically only have so much hot infrastructure to spread around. You'd expect they have to gradually build up the case that a large scale experiment is likely enough to yield a big enough advantage over what's already claiming those resources.

[ Parent | on HN ]
Re:Google Titans architecture, helping AI have long-term memory by dpe82 on Monday December 08, 2025 @03:01AM (#46189610)
I would imagine they do not want their researchers unnecessarily wasting time fighting for resources - within reason. And at Google, "within reason" can be pretty big.
[ Parent | on HN ]
Re:Google Titans architecture, helping AI have long-term memory by howdareme on Monday December 08, 2025 @05:34AM (#46190731)
I mean looking antigravity, jules & gemini cli, they have have no problem with their developers fighting for resources
[ Parent | on HN ]
Re:Google Titans architecture, helping AI have long-term memory by nl on Monday December 08, 2025 @06:45AM (#46191181)
I mean you'd think so, but...

> In fact, the UL2 20B model (at Google) was trained by leaving the job running accidentally for a month.

https://www.yitay.net/blog/training-great-llms-entirely-from...

[ Parent | on HN ]
Re:Google Titans architecture, helping AI have long-term memory by nickpsecurity on Sunday December 07, 2025 @05:52PM (#46186138)
But, it's companies like Google that made tools like Jax and TPU's saying we can throw together models with cheap, easy scaling. Their paper's math is probably harder to put together than an alpha-level prototype which they need anyway.

So, I think they could default on doing it for small demonstrators.

[ Parent | on HN ]
Re:Google Titans architecture, helping AI have long-term memory by m101 on Sunday December 07, 2025 @07:15PM (#46186853)
Prove it beats models of different architectures trained under identical limited resources?
[ Parent | on HN ]