I used the Freesound.org api (https://freesound.org/help/developers/) to download a bunch of sounds to MongoDB. Specifically, I selected to include the ID, Name, Tags, Description, Previews, Images, and Analysis in the response. I then looped through all these sounds and combined the response info into a prompt for GPT Davinci, instructing it to create a descriptive paragraph about the sound. I combined Davinci's description with the Freesound.org descriptions and embedded it using Ada. I inserted the embeddings into Pinecone with the audio preview url and ID as metadata. Then I just embed the search query and compare it with the sound embeddings (Pinecone allows you to select what type of similarity, I used cosine) and return the 25 most similar sounds.

Fascinating.

What if instead of reading a sound library, you had a music generator on the backend? What might an integration look like if it used the existing samples for description, as input for a "more like this" kind of feature