This example could very well have been a factory robot that can use LLM to assist in solving unanticipated challenges, expanding its generalized capabilities beyond a very narrow and specific task set. For instance, the same machinery can have the ability to assemble many different things with minimal additional programming by providing it with specifications, drawn plans, quality control criteria and examples, etc. The LLM can use those to encode instruction to its specialized sub systems that operate with classical techniques to achieve some goal based optimization. The “glue” so to speak to achieve the abstract task is supplied by the LLM. As it performs if it’s doing things wrong reprogramming is as simple as promoting it with the error and examples of better. Through reinforcement techniques the more it performs the tasks and encounters edges and errors the better it’ll get. Compared to current factory floor robotics this would be a remarkable advancement as floors could be reconfigured more rapidly, and the set of specialized machines could be reduced to a more general and commodity set.
Re:Businesses are in for a mighty debt hangover
by soared on Wednesday May 17, 2023 @09:09AM
(#35974736)
Like crypto and VR, sounds pretty cool but it’s not clear how this fuels economic growth.
Re:Businesses are in for a mighty debt hangover
by fnordpiglet on Wednesday May 17, 2023 @01:12PM
(#35978320)
It’s a toy example of solving an abstract task that using classical techniques would have been impossible, but by combining the abstract semantic capabilities of LLMs with classical techniques becomes fully achievable. The problem with classical agents and other techniques is they fail hard when faced with a landscape that was unanticipated or requires a comprehensive semantic awareness beyond the models domain. LLMs provide a space where abstract semantic “reasoning” can happen. It isn’t a deterministic optimizer in any way, no matter the attempts to show it can do basic math tasks etc with larger models and context lengths. But why does it have to? There are tasks, class as classification, description, semantic analysis, and others it does very well at. By combining techniques you can create agents that are much more robust and autonomous in many different situations that wasn’t practical before.
For instance: I uploaded an image of a saloon with cowboys playing cards and asked one of the multimodal models to describe the scene. It did with details about the game and the card table etc. Acting as a goal base agent I told it we need $20. It said it could either play cards or work at the saloon and recommended saloon work as it’s assured. I advised it to play cards and describe how to walk to the card table and to provide an image with the card table labeled and the route annotated with obstacles. It was able to do this. This could be fed into a navigation system. I then uploaded an image of a bunch of card hands and asked it to classify each hand as what it’s holding and the relative strengths. It was accurate. Using a optimizer it could feed the visual card state and play an optimal game. Etc. Each one of these tasks would have been absurdly hard five years ago.