News for hackers. Stuff that matters.

A comment on Ask HN: Simple, beginner friendly ETL / Data Engineering project ideas?.

Re:Ask HN: Simple, beginner friendly ETL / Data Engineering ... by jakestein on Friday August 17, 2018 @09:39AM (#17782725)
http://Singer.io is an open source ETL project written in Python. The components are small, composable programs that you can run independently, so you should be able to walk before your run.

A good beginner project is to build an adapter to a new data source (known as a "tap" in Singer). Most taps pull data out of business tools like Salesforce or Marketo, but people also build them to pull characters from the Marvel API (https://www.stitchdata.com/blog/tapping-marvel-api/)

Check out the getting started guide (https://github.com/singer-io/getting-started), or jump into the Singer Slack and ask for help (linked from the guide)

[ Parent | on HN ]
Re:Ask HN: Simple, beginner friendly ETL / Data Engineering ... by tomnipotent on Friday August 17, 2018 @06:02PM (#17785779)
And after you use Singer to ingest data, add in dbt [0] to transform the raw data into something more useful - I cannot speak highly enough of this library and how much work it will save you.

Also keep an eye on Gitlab's new Meltano [1] project which IMHO has huge potential.

[0] https://www.getdbt.com/

[1] https://gitlab.com/meltano/meltano/tree/master

[ Parent | on HN ]
Re:Ask HN: Simple, beginner friendly ETL / Data Engineering ... by veritas3241 on Friday August 17, 2018 @11:32PM (#17786704)
Thanks for the shout-out on Meltano! And we love dbt! Great project and great people behind the code. Their next release is going to be awesome - the new docs feature is beautiful.
[ Parent | on HN ]
Re:Ask HN: Simple, beginner friendly ETL / Data Engineering ... by vladsanchez on Friday August 17, 2018 @10:49AM (#17783208)
Fantastic reference! Thanks for sharing. As an ETL specialist I've looked for a composable solution but only found Luigi. I wanted something simpler, more integrated, like IFTTT. This has the feel of Yahoo Pipes. I think Singer is the answer. ;-)
[ Parent | on HN ]
Re:Ask HN: Simple, beginner friendly ETL / Data Engineering ... by dttos on Friday August 17, 2018 @12:41PM (#17784085)
Composableanalytics.com might be the tool you are looking for then....
[ Parent | on HN ]
Re:Ask HN: Simple, beginner friendly ETL / Data Engineering ... by arafalov on Friday August 17, 2018 @10:19AM (#17782964)
An Apache Solr (Search Engine) Target could be quite nice here.
[ Parent | on HN ]
Re:Ask HN: Simple, beginner friendly ETL / Data Engineering ... by zabana on Friday August 17, 2018 @10:12AM (#17782929)
I will look into this, thanks for sharing !
[ Parent | on HN ]