milindsoni

I modified the OpenAI Realtime API Console Demo App to add a map to it, and I connected the map events to the Realtime API.

Result... Im able to move around the map and talk to it! Its fascinating how its able to reverse geocode out of memory, Ideally I should add tools to reverse Geocode the locations. Im planning to add more tools to it like internet speed locations and maybe weather

https://github.com/milind-soni/realtime-map-chat

Check out here if you would like to use the code

I built a voice-controlled map where an AI can see exactly where you're looking and describe those locations with the OpenAI Realtime API.

Its a very simple setup where, as you pan and zoom around the map, the AI receives your current coordinates, zoom level, and visible area. Now I can add tools like get the internet speed or weather for that area and I will have a Voice Controlled Map.

I was using transformer.js to generate and use embeddings with small models in the browser itself, its quite useful to implement any kind of semantic search.

Im working on https://udf.ai/ ,Build and host data analysis agents quickly!