News for hackers. Stuff that matters.
Re:Launch HN: Castled Data (YC W22) – Open-Source Reverse ETL by amcaskill on Tuesday January 25, 2022 @01:13PM (#30075562)
Fantastic! Congratulations on the launch.

Is there a way to version control the sync configurations? Any thoughts on putting that in the roadmap?

I'd love to be able to put my 'Castled config' in the same repo as my dbt project, for example.

Re:Launch HN: Castled Data (YC W22) – Open-Source Reverse ETL by aruntdharan on Tuesday January 25, 2022 @01:24PM (#30075750)
You mean the warehouse/app credentials, when you say sync configurations? If so, yes, that seems like a great idea. Infact I think your warehouse credentials are already there in your dbt repo in a specific format. Castled can directly read those credentials from there.
Re:Launch HN: Castled Data (YC W22) – Open-Source Reverse ETL by amcaskill on Tuesday January 25, 2022 @01:43PM (#30076044)
That is not what I meant, but also pretty interesting.

I actually mean the 'definition' of the syncs themselves.

I am picturing JSON or YAML that describes the source fields, their mapping to the destination fields, and any other meta about the sync: frequency, number of retries, whatever else that you could configure in the UI

So when I go and update my dbt model to modify one of the tables that I am syncing from, I can make the corresponding changes to my Castled settings file, and release it all as one atomic update to my data infrastructure.

It might be a small number of people who would want something like that, but it's definitely something I would have been excited about when I was running a data team.

Re:Launch HN: Castled Data (YC W22) – Open-Source Reverse ETL by aruntdharan on Tuesday January 25, 2022 @02:11PM (#30076506)
We can definitely consider that. But I feel its a lot of config and can be error prone. For instance, source-destination field mapping configs might be complex and have various issues like data type mismatches, typos in field names etc and a user interface is better suited to guide you through the entire process.

But I see value in exporting the config to a github repo after the pipeline is created and thereafter future edits can be done via the github repo. Does that make sense?

Re:Launch HN: Castled Data (YC W22) – Open-Source Reverse ETL by amcaskill on Tuesday January 25, 2022 @02:16PM (#30076578)
Yes 100% -- you could also imagine just syncing from the UI to a repo, rather than trying to make the config human-editable. Toggle into a branch in the UI, make edits, and have those committed to the repo by the tool.

Looks awesome, I am rooting for you guys!

Re:Launch HN: Castled Data (YC W22) – Open-Source Reverse ETL by aruntdharan on Tuesday January 25, 2022 @02:21PM (#30076644)
Thanks for the input!
Re:Launch HN: Castled Data (YC W22) – Open-Source Reverse ETL by mason55 on Tuesday January 25, 2022 @05:39PM (#30079131)
> It might be a small number of people who would want something like that, but it's definitely something I would have been excited about when I was running a data team.

Yeah this one certainly depends on the target customer. For me, any tool that didn't have source control integration for configuration would be a non-starter. But it's quite possible that the target audience for this tool doesn't even understand the term "source control".

Re:Launch HN: Castled Data (YC W22) – Open-Source Reverse ETL by bleonard on Tuesday January 25, 2022 @09:19PM (#30080872)
Congrats to the Castled team on the launch.

At Grouparoo, this is a primary use case. We have a UI that engineers use locally. This helps gets things right. It outputs a JSON configuration that is checked in. When that is deployed, it does all the syncing.