Re:Ask HN: What database should I use for analytics data?
by buremba on Monday November 04, 2019 @04:07AM
(#21439981)
We also developed a product that collects the user event data via SDKs (https://github.com/rakam-io/rakam), depending on the data volume the users pick one of the two deployment types. The first one is based on Postgresql 11 and it takes advantage of partitioned tables (one table for each event time for the configured time period such as month, week, etc.), BRIN indexes, and materialized views.
It's capable of handling 250M events per month without an issue but beyond that we suggest the users to use data-warehouse that supports horizontal scaling and columnar storage engine so we use Snowflake with Snowpipe, Kinesis or Kafka (pluggable) and S3.
I have also tried using Clickhouse which is much cheaper but it lacks tooling for automatic scaling.
It's capable of handling 250M events per month without an issue but beyond that we suggest the users to use data-warehouse that supports horizontal scaling and columnar storage engine so we use Snowflake with Snowpipe, Kinesis or Kafka (pluggable) and S3.
I have also tried using Clickhouse which is much cheaper but it lacks tooling for automatic scaling.