Re:SQL should be your default choice for data engineering pi...
by lucid_thinker on Monday January 30, 2023 @10:08AM
(#34580536)
What solutions exist to manage data movement efficiently and effectively across multiple databases?
[ Parent
| on HN ]
Re:SQL should be your default choice for data engineering pi...
by tremon on Monday January 30, 2023 @11:20AM
(#34581851)
There's AWS Glue and Azure Data Factory, integration tools like Qlik Replicate or FiveTran Replication, or ConnX solutions if you have enterprise-scale money to burn.
We don't use any of them though, we have custom C# ingestion code that can talk to most databases and Odata API's. .Net's lazy collections make it quite easy to create a consumer/producer pipeline for streaming large datasets without too much overhead.