maycotte

You guys are a riot!

So interesting that even though their mathematical representations may differ, both the Jaccard index and the Tanimoto similarity essentially capture the same concept of similarity between sets: the ratio of the size of the intersection to the size of the union. Hence, they result in the same values, given the set operations, and are often used interchangeably.

Does FeatureBase compute on one or the other better?

Running dynamically generated Python code can be inherently risky and could cause serious harm to your local environment. The potential is amazing though. One must just tread carefully in this new world...

Tremor Video is processing data on 20B devices with greater than 30k attributes and is updating them at a rate of greater than 1 million records per second using FeatureBase. Not only is the ingest throughput amazing but reduced the hardware footprint of the main table to ~12TB from ~200TB in other, much slower and much more complex databases like Cloudera, Druid, Vertica, ArangoDB, Redis, and Aerospike.

This post discusses how FeatureBase uses Bit-sliced indexes to significantly reduce the number of bitmaps needed to represent a range of integer values. And by applying range-encoding to the indexes, it is able to perform lightning fast range queries.

To get to true real-time analytics, we can't just get better at scaling up or scaling out. Current technologies now excel at allowing users to spend more in order to get better performance, but the next generation of technologies needs to break efficiency barriers and open up new use cases. Scaling out works to a point, but the more machines you have, the higher the floor is on the latencies you can practically achieve. A more efficient technology is needed.

Why would someone want use FeatureBase vs a traditional RDBMS?

Love that! Early categorical/bitmap storage using notches...punched or not punched!

Bitmaps are amazing and we believe that they will replace all other data formats for analytics.

Today we open sourced about $25m in R&D (more to come)! They take a bit of harnessing, but once you do the results are amazing. We have released an OLAP database called FeatureBase, built entirely on bitmaps (well bitmaps in a b-tree)...

Here is a bit (pun intended) more about our big bet on bitmaps: https://www.featurebase.com/blog/bitmaps-making-real-time-an...

Hey Didip! Thanks for the mention.

Today we made our formal announcement and launched our new website!

Bitmaps are amazing and we believe that they will replace most other file formats for analytics.

Here is a bit (pun intended) more about our big bet on bitmaps: https://www.featurebase.com/blog/bitmaps-making-real-time-an...

Great question. Right now we have pushed hard to make SQL the primary interface to our feature store and can output Pandas and other formats at query time, however we are working on integrated/hosted Jupyter notebooks and excited to continue collaborating with the community on better feature first/oriented endpoints/interfaces. There is so much room to innovate here.

Yes, Pilosa is OSS under Apache 2.

We have been building a feature first data store for seven years and it feels like feature store is about to become one of the more exciting ways to extract value from data. We see feature stores doing much more than becoming just another silo for ML, but instead a way to get a real-time, centralized view of fragmented data that has to either be federated or put in a data lake to to be queried together. I share more of my thoughts in this blog post: https://www.molecula.com/why-moleculas-feature-based-approac... Molecula is based on the OSS platform Pilosa (https://www.pilosa.com/) and both Pilosa and Molecula are transitioning to reposition as feature stores over the coming months. Doing machine-scale analytics and ML on the data itself will be a thing of the past.

Great feedback. We could also do a better job of documenting the most frequent use cases which include high cardinality segmentation, personalization at scale, rapid data exploration/discovery, edge analytics, fast BI, threat detection, bioinformatics and more.