natekupp

hey bserial, I'm part of the team working on Dagster.

While there are many things we're working on, there are 3 goals that got me excited about working on this system:

1. Local development: most modern workflow orchestration systems don't have a good local development story. We want to provide a seamless end-to-end dev experience from your laptop to CI to dev to prod for authoring data workflows.

2. Complexity: the Airflow deployments I've worked on or otherwise encountered have hundreds of DAGs and thousands of tasks scheduled on an hourly or daily cadence. We aim to provide abstractions to better support managing and wrangling that complexity.

3. Testability: Most modern data platforms are poorly tested. Many orchestration systems, like Airflow, tend to hardcode deployment concerns the business logic, e.g. EmrAddStepsOperator. With Dagster, we aim to separate the business logic from environmental concerns to make it easy to swap out an external resource implementation for a mock, dev version, etc.

Hope that makes sense!

We use Pulumi to manage both our GCP and AWS resources, and we really like it.

You might consider using Terraform directly if you want something more mature.

Thumbtack | Software Engineer, SRE, many others | San Francisco, CA | ONSITE

Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.

We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.

We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!

http://www.thumbtack.com/jobs http://www.thumbtack.com/engineering Please reach out to jessica [at] thumbtack.com with any questions.

We use DynamoDB quite a bit at Thumbtack. Our biggest issue is backups - just wrote a short note about our experiences with DynamoDB here: https://medium.com/@natekupp/dynamodb-and-backups-16dba0dbcd...

The original Science paper is here: http://science.sciencemag.org/content/356/6342/1046.full

The section "Relativistic deflections by foreground stars" walks through the math which enables this, which I found really interesting.

Have you considered just applying through the jobs pages of the larger tech companies? Apple, Google, Facebook, LinkedIn are all on the peninsula and more accessible than startups in SF.

We're also doing this at Thumbtack. We run all of our Spark jobs in job-scoped Cloud Dataproc clusters. We wrote a custom Airflow operator which launches a cluster, schedules a job on that cluster, and shuts down the cluster upon job completion. Since Google can bring up Spark clusters in < 90s and bills minutely, this works really well for us, simplifying our infrastructure and eliminating resource contention issues.

Thumbtack | Software Engineer, SRE, Data Scientist, many others | San Francisco, CA | ONSITE

Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.

We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.

We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!

http://www.thumbtack.com/jobs http://www.thumbtack.com/engineering Please reach out to jessica [at] thumbtack.com with any questions.

I'm constantly surprised by how much work Amazon expects its customers to do themselves. The work that Segment has done here should be a service provided by AWS directly, continuously updating cost data in a Redshift database without any customer work required.

We just migrated our data infrastructure to GCP. One of the big motivators was experiences like this with AWS. We've got near-realtime GCP cost dashboards in BigQuery, and the only meaningful work on our end to make that happen was writing the SQL queries.

Thumbtack | Software Engineer, SRE, Data Scientist, many others | San Francisco, CA | ONSITE

Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.

We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.

We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!

http://www.thumbtack.com/jobs http://www.thumbtack.com/engineering Please reach out to jessica [at] thumbtack.com with any questions.

Thumbtack | Software Engineer, SRE, Data Scientist, many others | San Francisco, CA | ONSITE

Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.

We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.

We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!

http://www.thumbtack.com/jobs http://www.thumbtack.com/engineering Please reach out to jessica [at] thumbtack.com with any questions.

Thumbtack | https://thumbtack.com | San Francisco, CA | ONSITE

Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.

We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.

We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!

http://www.thumbtack.com/jobs http://www.thumbtack.com/engineering Please reach out to jessica [at] thumbtack.com with any questions.

Sure, please reach out to jessica [at] thumbtack.com!

Thumbtack | https://thumbtack.com | San Francisco, CA | ONSITE

Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.

We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.

We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS/GCP. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!

Thumbtack | https://thumbtack.com | San Francisco, CA | ONSITE

Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.

We are a friendly, ambitious team of 100+ engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.

We're looking for engineers and SREs interested in working with Go, Scala/Spark, PHP, Angular, iOS, Android, and AWS. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!

Thumbtack | https://thumbtack.com | San Francisco, CA | ONSITE

Thumbtack is a local services marketplace that connects millions of customers with the right professionals for anything they need done.

We are a friendly, ambitious team of 100 engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital. Together, we are disrupting a $700B market in the US alone where word of mouth is still the status quo.

We're looking for engineers and SREs interested in working with Go, Scala/Spark/Hadoop, PHP, Angular, iOS, Android, and AWS. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!

Thumbtack | San Francisco, CA | Full-Time | Onsite

Our mission is to help people accomplish personal projects by matching their needs to the best service professionals in their area. From wedding photographers and DJs to home contractors or French tutors, Thumbtack can help.

We are a friendly, ambitious team of 80 engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital.

We're looking for engineers and SREs interested in working with Go, Scala/Spark/Hadoop, PHP, Angular, iOS, Android, and AWS. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!

http://www.thumbtack.com/jobs http://www.thumbtack.com/engineering

Please reach out to jessica [at] thumbtack.com with any questions.

Thumbtack | https://thumbtack.com | San Francisco, CA | ONSITE

Our mission is to help people accomplish personal projects by matching their needs to the best service professionals in their area. From wedding photographers and DJs to home contractors or French tutors, Thumbtack can help.

We are a friendly, ambitious team of ~70 engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital.

We're looking for engineers and SREs interested in working with Go, Scala/Spark/Hadoop, PHP, Angular, iOS, Android, and AWS. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!

http://www.thumbtack.com/jobs http://www.thumbtack.com/engineering

Please reach out to jessica [at] thumbtack.com with any questions.

Thumbtack | https://thumbtack.com | San Francisco, CA | ONSITE

Our mission is to help people accomplish personal projects by matching their needs to the best service professionals in their area. From wedding photographers and DJs to home contractors or French tutors, Thumbtack can help.

We are a friendly, ambitious team of ~60 engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital.

We're looking for engineers and SREs interested in working with Go, Scala/Spark/Hadoop, PHP, Angular, iOS, Android, and AWS. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!

http://www.thumbtack.com/jobs http://www.thumbtack.com/engineering

Please reach out to jessica [at] thumbtack.com with any questions.

thanks for flagging, I'll look into it!

hey all, feel free to reach out to me either on this thread, or directly at nate[at]thumbtack.com if I can answer any questions!

+1 to this, we kick off our Sqoop jobs using Airflow - http://airbnb.io/projects/airflow/

Airflow is very similar to Luigi; we've been using in in production to schedule all of our workflows for ~4 months now and it's worked out really well for us.

Thumbtack | https://thumbtack.com | San Francisco, CA | ONSITE

Our mission is to help people accomplish personal projects by matching their needs to the best service professionals in their area. From wedding photographers and DJs to home contractors or French tutors, Thumbtack can help.

We are a friendly, ambitious team of ~60 engineers in a bright SoMa office with daily home-cooked food, backed by Sequoia and Google Capital.

We're looking for engineers and SREs interested in working with Go, Scala/Spark/Hadoop, PHP, Angular, iOS, Android, and AWS. We're also looking for data scientists interested in predictive modeling, machine learning, and experimental design and analysis. Join us!

http://www.thumbtack.com/jobs http://www.thumbtack.com/engineering

Please reach out to jessica [at] thumbtack.com with any questions.