Re:Data Branching for Batch Job Systems
by ragulpr on Friday January 24, 2025 @02:40PM
(#42816300)
This pattern is really meaningful conceptually but the tricky thing is to not create a mess in the process.
If it's too easy to branch people will do so and the (knowledge) economics scale disappears (and we'll have a mess).
If the common data definition is too hard to branch from no experiments will happen (slow).
I think most tech for this seem to make it too easy, and in the process injecting a bunch of dependencies that makes it slow and harder to access. May have changed since I last looked.
I found that the simple pattern of versioned paths/table names as `s3://mybucket/mystage/version=42/` or `my_table_v42` puts a high enough evolutionary cost on branching (as consumers need to explicitly adapt) while it also doesn't have the costs associated with using special tech (legacy/lock in/dependencies).
It's also searchable on github/slack/etc if done right.
If it's too easy to branch people will do so and the (knowledge) economics scale disappears (and we'll have a mess).
If the common data definition is too hard to branch from no experiments will happen (slow).
I think most tech for this seem to make it too easy, and in the process injecting a bunch of dependencies that makes it slow and harder to access. May have changed since I last looked.
I found that the simple pattern of versioned paths/table names as `s3://mybucket/mystage/version=42/` or `my_table_v42` puts a high enough evolutionary cost on branching (as consumers need to explicitly adapt) while it also doesn't have the costs associated with using special tech (legacy/lock in/dependencies).
It's also searchable on github/slack/etc if done right.