Building Analytics at 500px

Re:Building Analytics at 500px by potatosareok on Wednesday September 30, 2015 @11:43PM (#10309096)

Thanks a lot for a detailed post. 20gb of log to not seem like much though? Not really sure of 500px scale. Fully conceding that at my company probably are logging too much, I'm wondering what takes ETL pipeline 4hr to run over that. Is just no need to optimize it since there's no benefit to have these metrics real time? Or am I missing some really data heavy part. Again not trying to be negative just wondering.

Luigi seems cool, can anyone comment on it compared Airflow or Spring XD, or is are those just different products.

Periscope looks like Kibana for relational stores, also cool to see.

Thanks again for great post!

[ Parent | on HN ]

Re:Building Analytics at 500px by Wagthesam on Thursday October 01, 2015 @01:18AM (#10309369)

Its not a lot of data, but the biggest constraints to me were cost time to implementation. You could actually get an amazon memory intensive server and do it all in memory, but I didn't have those resources. ETL + a Redshift server in the end cost me around $5000 a year, which is TINY compared to the value we got out of it and the cost that most companies pay.

[ Parent | on HN ]

Re:Building Analytics at 500px by potatosareok on Thursday October 01, 2015 @01:48AM (#10309456)

Makes sense I'm used to physical servers with tons of ram at work. If you don't mind, and I assume you're on Amazon entirely then, what size instances/how many are you using for the ETL(luigi) nodes? And do is there any infrastructure besides s3 (mysql dump, logs) => amazon instances (luigi) => redshift being involved?

[ Parent | on HN ]

Re:Building Analytics at 500px by Wagthesam on Thursday October 01, 2015 @02:11AM (#10309511)

That is all there is. s3, one amazon instance for luigi which holds the mysql read replica, and amazon redshfit. There wasn't any heavy ETL in Luigi. Luigi mostly just extracted/dumped data. All the heavy lifting was in EMR

[ Parent | on HN ]

Re:Building Analytics at 500px by Wagthesam on Thursday October 01, 2015 @03:39AM (#10309767)

Also to add. Logging too much should not be a problem if you are using it. Wish's hadoop log store is over 40tb compressed right now. And its worth every penny.

[ Parent | on HN ]

Re:Building Analytics at 500px by samuell on Friday October 02, 2015 @08:46AM (#10318100)

Luigi creator Erik Bernhardsson has posted some notes on Luigi vs Airflow & more:

http://erikbern.com/2015/07/02/more-luigi-alternatives/

... and Pinball:

http://erikbern.com/2015/03/14/pinterest-open-sources-pinbal...

[ Parent | on HN ]