News for hackers. Stuff that matters.
Re:Ask HN: What's a promising area to work on in 2024 -ideal... by countWSS on Tuesday January 16, 2024 @11:23AM (#39015127)
Any architecture that could compete with LLMs is going to reshape the ML landscape that is now dominated by GPU-heavy algorithms like transformer attention. Its likely there is something more fundamental that could work without much training, like finding exactly what training process changes and\ forming architecture around optimizing that. I don't think people realize how much influence 'dumb stochastic parrots' have now, and the whole progress in identifying what exactly happens inside these networks is still a mystery: there hints that by sheer size they forming structures capable of cognition but only as side effect(finding out what exactly is going there and replicating in simpler terms would yield massive speed-ups).
Re:Ask HN: What's a promising area to work on in 2024 -ideal... by a2code on Tuesday January 16, 2024 @11:48AM (#39015428)
So. Transformers work well due to scale, but fall short without it. Why this happens is not known?