News for hackers. Stuff that matters.

A comment on Engineers Shouldn’t Write ETL. Parent comment

Re:Engineers Shouldn’t Write ETL by saosebastiao on Friday March 18, 2016 @12:55PM (#11313077)
Lots and lots of SQL (a very good thing IMO). This can come in the form of Oracle, Redshift, or any of the commonly available RDS databases (probably not SQL Server though). This is augmented with a lot of Big Data stuff, which used to be pretty diverse (Pig, Hive, raw Hadoop, etc.) but is moving very quickly towards a Spark-centric platform. There is occasionally some commercial software like Tableau/Microstrategy. Apart from that, it is a whole lot of homegrown technology.

As far as architecture is concerned, every team is different. At least on the retail side, we tend to center around a large data warehouse that is currently transitioning from oracle to redshift, and with daily log parsing ETL jobs from most services to populate tables in that data warehouse. Downstream of the data warehouse is fair game for pretty much anything.

[ Parent | on HN ]
Re:Engineers Shouldn’t Write ETL by arikfr on Friday March 18, 2016 @02:21PM (#11313736)
Are you using Re:dash[1]? Seems like a good fit if you're doing lots of SQL and working with many different databases.

[1] http://redash.io/

[ Parent | on HN ]
Re:Engineers Shouldn’t Write ETL by kakoni on Saturday March 19, 2016 @07:09AM (#11318067)
Do you have somesort of framework for those ETL jobs? How do you handle ETL dependencies/workflow?(=thinking spotify's luigi[1] here)

[1] https://github.com/spotify/luigi

[ Parent | on HN ]
Re:Engineers Shouldn’t Write ETL by drited on Friday March 18, 2016 @01:38PM (#11313450)
Would you know why sql server generally isn't used?
[ Parent | on HN ]
Re:Engineers Shouldn’t Write ETL by sshykes on Friday March 18, 2016 @01:57PM (#11313571)
Probably because:

* SQL Server is proprietary

* SQL Server licenses are expensive

* SQL Server runs only on Windows (or least used to?)

[ Parent | on HN ]
Re:Engineers Shouldn’t Write ETL by hunterjrj on Friday March 18, 2016 @04:40PM (#11314805)
I chuckled a bit because the GP mentioned Oracle. I'm sure you've heard this one:

Customer: "How much does an Oracle database license cost?" Oracle Rep: "Well, how much do you have?"

[ Parent | on HN ]
Re:Engineers Shouldn’t Write ETL by true_religion on Friday March 18, 2016 @05:33PM (#11315299)
That's actually not a joke. Oracle liked to argue you should accept server licences as being a fraction of your budget, so when it went up you'd automatically pay them more.
[ Parent | on HN ]
Re:Engineers Shouldn’t Write ETL by avon on Friday March 18, 2016 @05:06PM (#11315041)
Amazon actually does allow SQL Server. The poster saying it probably didn't was influenced by the fact that Azure, Microsoft's own cloud solution is an AWS competitor.

To your own points:

SQL Server is as proprietary as Oracle SQL Server is cheaper than Oracle SQL Server is being ported to Linux in 2017 :)

[ Parent | on HN ]
Re:Engineers Shouldn’t Write ETL by newjersey on Saturday March 19, 2016 @12:16AM (#11317274)
Amazon.com seems to strongly encourage the companies it acquires to use aws. Apparently, woot.com is fixing to rewrite the website in Java or something just so they do not have to use Windows. Apparently, it makes sense to them.
[ Parent | on HN ]