News for hackers. Stuff that matters.
Re:SQL should be your default choice for data engineering pi... by tremon on Monday January 30, 2023 @11:20AM (#34581851)
There's AWS Glue and Azure Data Factory, integration tools like Qlik Replicate or FiveTran Replication, or ConnX solutions if you have enterprise-scale money to burn.

We don't use any of them though, we have custom C# ingestion code that can talk to most databases and Odata API's. .Net's lazy collections make it quite easy to create a consumer/producer pipeline for streaming large datasets without too much overhead.

There are no comments on this comment yet.