S3 was down - Hackdot

Re:S3 was down by chosenken on Thursday September 14, 2017 @04:12PM (#15252072)

We use airflow mainly for ETL work, backfills, and batch processing. We do a lot of work with clickstream type data, be it taking data from analytics.js and loading it into redshift, or taking analytic data from redshift and loading into Google analytics. We have developed many pipes and connectors for airflow that allow us to connect to many data sources, both at the source and sink ends. I mainly work on the DevOps team, running our infrastructure and working on back end systems, so my knowledge of airflow is more high level. I just keep the system up so it can run ;P FYI I work for a startup out of Cincinnati, OH called Astronomer, you can find us at https://www.astronomer.io

Also, we weren't like all down, we just saw lots of time out issues when reading/writing to S3.

[ Parent | on HN ]

Re:S3 was down by scrollaway on Thursday September 14, 2017 @04:14PM (#15252094)

Your site's ssl cert is not setup correctly :)

Very interesting re redshift => google analytics though, I've never heard of it done in that direction.

Do you think airflow is suitable for one-off/cron task management as well?

[ Parent | on HN ]

Re:S3 was down by chosenken on Thursday September 14, 2017 @04:58PM (#15252458)

Actually yes. Airflow has cron support baked in, when you run a task you can give it a cron schedule. It then takes care of running your tasks when scheduled.

As for the ssl, I messed up my link. We are in the process of moving the site, so some of the redirects are not working perfect at the moment >.>

[ Parent | on HN ]

Re:S3 was down by bpicolo on Thursday September 14, 2017 @04:16PM (#15252111)

I get an SSL warning on first connect to your site. Chrome is unhappy that it's signed for *.hubspot and not astronomer.io

[ Parent | on HN ]