News for hackers. Stuff that matters.

JourneymanCoder

Data transforms on a data pipeline without ETL hell

So I'm building a data pipeline / analytics solutions. The functionality is pretty much what one would expect. Events are received, then stored, then some transformations are performed.We have plenty of internal users who might do transformations with code (e.g. Python) or more standard ETL methods. Problem is we end up with ETL hell when we have a bunch of these transformations we're running,...

( Read More... | no comments | 2 points | on HN )

How to build an analytics system

I need to build a homegrown analytics system that logs user events, which will then be transformed, and used to generate reports or allow for ad-hoc querying.One of the big issues I'm wrestling with is the impact on developers of the web app and mobile app. We already have them in place. So developers would have to go back through all their code to add code to pass in the appropriate event and...

( Read More... | 1 comment | 2 points | on HN )