Ask HN: Best way to build a data collection pipeline?

Posted by anacleto on Sunday April 01, 2018 @03:49AM
from the ask-hn dept.

I'm building a predictive analytics SaaS tool. I need to collect a massive amount of data coming from my own customers. Do you have some good advice or know any useful API product to seamlessly collect event data?

( 3 comments | 4 points | on HN )

Re:Ask HN: Best way to build a data collection pipeline? by asavinov on Sunday April 01, 2018 @04:47AM (#16727386)

Bistro Streams is an open source light-weight stream analytics engine: https://github.com/asavinov/bistro

It is a general-purpose and highly configurable data processing engine which can be applied to many workloads and scenarios including data integration, data migration, ETL, big data processing etc.

[ Parent | on HN ]

Re:Ask HN: Best way to build a data collection pipeline? by pastyboy on Sunday April 01, 2018 @04:10AM (#16727299)

Snowplow open source or managed solution ($$$) - give it a look. http://www.snowplowanalytics.com

[ Parent | on HN ]

Re:Ask HN: Best way to build a data collection pipeline? by liberal_098 on Sunday April 01, 2018 @05:01AM (#16727416)

Kafka is probably the best option in this case.

[ Parent | on HN ]