Show HN: FluidDATA API – Create your own audio search engine with FluidDATA

Re:Show HN: FluidDATA API – Create your own audio search eng... by mathatoms on Tuesday April 10, 2018 @11:24AM (#16802018)

We were planning on writing up a blog post to go over what our backend looks like. But essentially we have written a crawler to discover audio on the internet and a distributed processing framework to download, extract metadata, and transcribe the audio.

We've iterated through a few storage solutions and have settled on using GlusterFS+zfs running on Storinators. So far we have about 350TB of data indexed in our collection.

[ Parent | on HN ]

Re:Show HN: FluidDATA API – Create your own audio search eng... by dandancanfly on Tuesday April 10, 2018 @11:28AM (#16802050)

That's pretty neat. After you download the audio and process it, do you delete the data, or store it for safe keeping? 350TB is a healthy chunk of data.

[ Parent | on HN ]

Re:Show HN: FluidDATA API – Create your own audio search eng... by mathatoms on Tuesday April 10, 2018 @11:42AM (#16802182)

We have enough storage to hold on to the data. We keep the data around so we can retranscribe files as we update our language models.

[ Parent | on HN ]

Re:Show HN: FluidDATA API – Create your own audio search eng... by chatmasta on Tuesday April 10, 2018 @11:30AM (#16802071)

Wow! Sounds awesome, I would love to read a blog post on that.

Are you co-locating the hardware? What is bandwidth pricing like?

[ Parent | on HN ]

Re:Show HN: FluidDATA API – Create your own audio search eng... by mathatoms on Tuesday April 10, 2018 @11:51AM (#16802268)

Thanks. Our blog is located at https://blog.bitplatter.com.

We are co-locating some of our infrastructure. The backend that does the data processing is running in a rack on our own hardware. The user facing portions are hosted in GCE.

[ Parent | on HN ]