jklukas

A question on Twitter (https://twitter.com/sayhar/status/671867539480305664) asked about how Simple decided to do ETL in Redshift.

I think the main benefit is flexibility. Since transformations are happening in the same system where the data already lives, there's no I/O overhead and the performance is pretty fast. Our full suite of nightly ETL jobs runs in about 2 hours, and that's without any effort to run jobs in parallel. There's also minimal schema maintenance involved; the nightly jobs completely drop the existing tables, and in most cases the table creation is handled automatically as part of a CREATE TABLE AS statement. That means we're pretty free to add, drop, or rename columns without needing to worry about migrations or keeping the schema in sync with what some external tool is doing.

That said, no one on our team has deep familiarity with ETL tools, so there may be other good options that we simply haven't investigated.

Being able to adapt dynamically to schema changes would be great. If you do develop a solution for that, please do publicize it. Simple uses segment.com for some communication of analytics info to third parties, and Segment folks have built some capabilities to infer schemas from incoming messages which they then apply to history tables in Redshift. We're hoping to learn more from Segment about how they make that happen.

What you described with documentation efforts is kind of scenario I fear. We could spend significant effort putting together a documentation system that simply doesn't meet a real need. It's good to know other folks are also struggling with that, though.

I wanted to mention some things that are still struggles with our current platform at Simple.

Maintaining our Redshift schema can be time-consuming, especially since there can be upstream changes that affect the schema of the messages we consume, and we often aren't notified. We'll just be silently dropping the new columns, or loads into Redshift will start to fail. Postgres 9.5 is supposed to include DDL messages in logical decoding, which will help for our pipeline from Postgres databases, but we'd love to develop a more robust system for automatically updating the Redshift schema to accommodate changes in the incoming data format. Naive solutions would likely be susceptible to unwanted behavior coming from spurious messages. We could end up with a large number of mostly empty columns in some cases.

We've also had trouble maintaining knowledge about the schema. We'd like to have some sort of "data dictionary" that would be available for folks to understand where the data in Redshift is coming from and what each of the columns mean. This might end up being a standalone document, but there's concern about it staying up to date with changes in the schema. We've toyed with the idea of maintaining this sort of documentation as Javadoc-style comments directly in the repository where we define our Redshift schema and migrations, but we haven't found any existing tooling to render documentation from comments in SQL source files.