News for hackers. Stuff that matters.
Re:Engineers Shouldn’t Write ETL by msencenb on Sunday November 04, 2018 @07:28PM (#18378890)
Does anyone have experience with ETL as a service like StitchData (not related to stitchfix)?

The startup I'm employed at needs some data analysis, but it is not big data, simply a way to unify analytics into a queryable database. I'm not looking forward to writing any ETL code, and was hoping someone here had a tool to help.

Re:Engineers Shouldn’t Write ETL by veritas3241 on Sunday November 04, 2018 @08:14PM (#18379074)
We used them for a single data source at GitLab (Zendesk). Worked pretty well! But we quickly hit a road block where what you could get via the UI wasn't all the tables available. We wound up forking it and adding to the tap. Basically all of our extractors and loaders are going to follow the Singer spec for taps and targets - it's a pretty nice model.

Internally, we're using a tool called Meltano which is aimed at solving just your problem. Most of our data warehouse is coming from external business ops tools (Salesforce, Zuora, Zendesk, Marketo, etc.) and we're using dbt for transformations w/ Looker as the BI layer. Definitely check our primary analytics repo [0] as all of our code is out in the open. Feel free to ping me if you have more questions - tmurphy at gitlab.

[0] https://gitlab.com/meltano/analytics/

Re:Engineers Shouldn’t Write ETL by drunkpotato on Sunday November 04, 2018 @09:46PM (#18379495)
At the company I work for, we have integrations with Stitch Data and Fivetran. Both are good and have been responsive to my needs. Neither have been perfect, so when I've noticed a problem I've had to keep on top of them to fix it. I also maintain a few of our own ETL jobs for data sources that aren't supported. I will say that I recommend using an ETL vendor without reservation. The nominal cost is more than made up for in the headaches you'll save yourself in creating and maintaining a homegrown ETL.
Re:Engineers Shouldn’t Write ETL by jpulec on Sunday November 04, 2018 @09:52PM (#18379526)
I've used StitchData at a startup with AWS Redshift. Pair it with something like dbt for transforming your data, and you have a great match. A little pricey, but totally worth it, IMO.
Re:Engineers Shouldn’t Write ETL by mycelium on Sunday November 04, 2018 @08:05PM (#18379043)
I would highly highly recommend ETL as service, after adopting it recently. It substantially changes your relationship with your data sources in a really positive way. And frankly, ETL for common data sources is code that you just don't need to write.

I would say that you should pilot with a few ETL vendors. We currently use Fivetran, they're fine but we've had enough burps that I cannot cold recommend them over other vendors. I cannot for the life of me remember the details, but I think we went with them over Stitch for pricing reasons.

Re:Engineers Shouldn’t Write ETL by georgewfraser on Monday November 05, 2018 @01:58AM (#18380415)
I'm Fivetran's CEO and I just want you to know, whatever "burps" you experienced, these things keep me up at night and the whole team is always striving to make the pipeline "just work". The whole vision of our product is that you should be able to plug in and get a perfect mirror image of all your data sources in your data warehouse. Anytime we fall short of that it drives us crazy.
Re:Engineers Shouldn’t Write ETL by Petefine on Monday November 05, 2018 @04:01PM (#18385940)
Do you have a forum or suggestions tool at all? Fivetran has been amazing for our new datawarehouse and we're very pleased with the service, but there are a few little (non-bug) things that would have made it even easier.
Re:Engineers Shouldn’t Write ETL by fraserharris on Wednesday November 07, 2018 @11:45PM (#18404381)
You can email me: [email protected]
Re:Engineers Shouldn’t Write ETL by jakecodes on Sunday November 04, 2018 @09:15PM (#18379363)
Check out Meltano, a GitLab startup. It’s new and we’re iterating but it sounds like it could solve your exact situation. Feel free to leave issues where we could improve. https://gitlab.com/meltano/meltano
Re:Engineers Shouldn’t Write ETL by Zaheer on Sunday November 04, 2018 @08:53PM (#18379252)
I work on AWS Glue (https://aws.amazon.com/glue/). Depending on how complicated the pipeline you'd like to setup, you may not need to write any code.
Re:Engineers Shouldn’t Write ETL by fraserharris on Sunday November 04, 2018 @07:53PM (#18378998)
I would suggest checking out my employer, Fivetran (fivetran.com). Many tech firms use us to centralize their data for analysis. We have startup pricing for sub-50 employees.
Re:Engineers Shouldn’t Write ETL by NovaX on Sunday November 04, 2018 @09:32PM (#18379432)
Do you know when DB/2 support will be added (Windows, AS/400)? I've been wanting to switch over from AWS DMS and our in-house support.
Re:Engineers Shouldn’t Write ETL by jakestein on Monday November 05, 2018 @11:00AM (#18383329)
We (Stitch) have DB2 in production today https://www.stitchdata.com/integrations/db2/
Re:Engineers Shouldn’t Write ETL by NovaX on Monday November 05, 2018 @11:19AM (#18383503)
Is it log-based cdc? The documentation says only 2 dbs are supported that way. We want to replicate customer databases which have schemas outside of our control, with the goal of migrating them to our products. The other vendor schemas might not have your required columns. There are also many flavors of DB2. So very interested but the details are sparse at best.
Re:Engineers Shouldn’t Write ETL by jakestein on Monday November 05, 2018 @05:07PM (#18386411)
Our DB2 integration is SELECT replication. Our MySQL, Posgtgres, and Oracle integrations are log base. MySQL and Postgres actually have the option of either logs or SELECT.

You're totally right about all of the flavors DB2. Our support team would be the ones to figure out for sure whether or not we can work with your setup, and you reach them at [email protected]

Re:Engineers Shouldn’t Write ETL by fraserharris on Monday November 05, 2018 @02:54AM (#18380623)
We are planning on adding support for DB/2 in Q1/Q2 2019. We are getting SAP HANA out first
Re:Engineers Shouldn’t Write ETL by ben_jones on Sunday November 04, 2018 @07:34PM (#18378920)
No tool but I've had success writing code generators to better interface between various data systems, autogeneratoring various accessors and utility functions.