News for hackers. Stuff that matters.

ramiamar

Here are a few sources which helped us grow our company:

- https://www.saastr.com

- http://tomtunguz.com

- The Hard Thing About Hard Things by Ben Horowitz

- Built to Last by Jim Collins & Jerry Porras

- Crossing the Chasm by Geoffrey Moore

You're not dreaming, perhaps you might just be underestimating the complexity of building an analytics stack.

Here's my suggestion:

- Start with picking you data warehouse. 500 GB/mo can exceed regular SQL databases' performance, so the next cheapest option is to choose a cloud based analytical datastore: BigQuery, Snowflake, Redshift, Athena.

- Next, work on your ETL. There are some open source tools, they all require a large amount of maintenance:

  - https://github.com/snowplow/snowplow
  - Talend
  - https://www.singer.io
  - http://www.fluentd.org/
  - Kafka & Kafka Connect
- Retention, cohorts, and other analysis - you would probably need to write your own queries, and schedule them with a cron like tool:

  - cron ( :) )
  - airflow
  - luigi
- For visualization and charting, I know of two good open source tools:

  - Redash.io
  - http://airbnb.io/superset/
One thing you mentioned which might be hard to do is to join across keys from each source. Analytical datastores are not very good at that, as tables are partitioned, and joining tables requires that you plan ahead what you are joining on, and partition your tables accordingly. To support joining records in your ETL process, I'm not sure there are easy ways to implement this (let me know if you find one, we are working on it too :) ). You can look at Kafka Streams or Apache Storm, as a starting point
Grafana for dashboarding, and one of [Graphite, InfluxDB, Prometheus] for collecting, storing, and querying the metrics.