Open source models in general. Meta has for instance released DINO which is a self supervised transformer model. LLMs are also going multi modal (see LLaVA for instance). The name "LLM" has stuck but they should really be called Large Transformer Models. LeCun is working on self supervised visual world models (I-JEPA) which if successful and released could form the basis for killer drones.

For a drone, an LLM derived solution is far too slow, unreliable, heavy and not fit for purpose. Developments in areas like optical flow, better small CNNs for vision, adaptive control and sensor fusion are what's needed. When neural networks are used, they are small, fast, specialized and cheap to train.

A multimodal or segmentation algorithm is not the solution for bee-level path planning, obstacle avoidance or autonomous navigation. Getting LLMs to power a robot for household tasks with low latency to action and in an energy efficient manner is challenging enough, before talking about high-speed, highly maneuverable drones.

Tesla is running these models on 4 year old hardware to control a car in real time (30 fps). You don't need a full 100B model to control a drone, and it doesn't have to be as good as a car to cause a lot of damage. Reportedly both Ukraine and Russia are putting together on the order of a thousand drones a day at this point, Tesla includes the compute to run this in every car they make already today. Hardware is also moving fast, how come people forget about Moore's law and software improvements? To me there's no question that this tech will be in tens of thousands of drones within a few years.

a multimodal llm is a general purpose device to churn sensor inputs into a sequence of close to optimal decisions. the 'language' part is there to reduce the friction of the interface with humans, it's not an inherent limitation of the llm. not too farfetched to imagine a scenario where you point to a guy in a crowd and tell a drone to go get him, and the drone figures out a close to optimal sequence of decisions to make it so.

I think GPT-4V could probably make high level decisions about what actions to take.

Not really practical at the moment of course since you can't put 8 A100s on a drone.

there are rumors that the latest gen drones in ukraine use crude embedded vision ai to increase terminal accuracy. launch and iterate, this will only get more lethal.