News for hackers. Stuff that matters.
Re:Google Calls in Help from Larry Page and Sergey Brin for ... by westurner on Saturday January 21, 2023 @10:02AM (#34466843)
why woykd you say they are on "red alert"? Sounds like lazy investors pumping or dumping to me.

Google already has very large LLMs online for search and other applications.

A similar take on similar spin: https://twitter.com/westurner/status/1614002846394892288

From April 2022, "Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrough Performance" https://ai.googleblog.com/2022/04/pathways-language-model-pa... :

> In recent years, large neural networks trained for language understanding and generation have achieved impressive results across a wide range of tasks. GPT-3 first showed that large language models (LLMs) can be used for few-shot learning and can achieve impressive results without large-scale task-specific data collection or model parameter updating. More recent LLMs, such as GLaM, LaMDA, Gopher, and Megatron-Turing NLG, achieved state-of-the-art few-shot results on many tasks by scaling model size, using sparsely activated modules, and training on larger datasets from more diverse sources. Yet much work remains in understanding the capabilities that emerge with few-shot learning as we push the limits of model scale.

> […] More recent LLMs, such as GLaM, LaMDA, Gopher, and Megatron-Turing NLG, achieved [SOTA] few-shot results on many tasks by scaling model size, using sparsely activated modules, and training on larger datasets from more diverse sources. Yet much work remains [for few-shot LLMs] /2

DeepMind Dreamerv3 is more advanced than all the LLMs.

Re:Google Calls in Help from Larry Page and Sergey Brin for ... by sinuhe69 on Saturday January 21, 2023 @11:42AM (#34467653)
I don’t think the issue of Google is a lack of advanced AI technology but the transfer and realization of products. So many Google’s products are half-backed, only half-heartedly developed and then be buried at the end. For a giant like Google, they must be stuck in a rut and to get free need all the help they can get. If they continue the business as usual, I afraid they will experience a never-seen before landslide. I believe calling back the founders at this critical time is the right move.
Re:Google Calls in Help from Larry Page and Sergey Brin for ... by westurner on Saturday January 21, 2023 @03:37PM (#34470223)
I disagree with your assessment of their comparative performance.

"Beyond Tabula Rasa: Reincarnating Reinforcement Learning" https://ai.googleblog.com/2022/11/beyond-tabula-rasa-reincar... :

> Furthermore, the inefficiency of tabula rasa RL research can exclude many researchers from tackling computationally-demanding problems. For example, the quintessential benchmark of training a deep RL agent on 50+ Atari 2600 games in ALE for 200M frames (the standard protocol) requires 1,000+ GPU days. As deep RL moves towards more complex and challenging problems, the computational barrier to entry in RL research will likely become even higher.

> To address the inefficiencies of tabula rasa RL, we present “Reincarnating Reinforcement Learning: Reusing Prior Computation To Accelerate Progress” at NeurIPS 2022. Here, we propose an alternative approach to RL research, where prior computational work, such as learned models, policies, logged data, etc., is reused or transferred between design iterations of an RL agent or from one agent to another. While some sub-areas of RL leverage prior computation, most RL agents are still largely trained from scratch. Until now, there has been no broader effort to leverage prior computational work for the training workflow in RL research. We have also released our code and trained agents to enable researchers to build on this work.

Feed-Forward with Prompt Engineering is like RL; which prompt elements should remain given objective or subjective error?