Building a High Performance Data Integration Framework in Go

Re:Building a High Performance Data Integration Framework in Go by yevpats on Monday November 28, 2022 @04:52PM (#33780361)

Great question! We specifically started with connectors/plugins to cloud infrastructure providers and other infrastructure vendors - This is why most of our users migrated and/or compared us to the tools specified here (https://www.cloudquery.io/docs/cq-vs-others/overview).

There are no connectors for AWS, GCP, Azure in FiveTran, Airbyte, Meltano but as we extend the number of our source plugins I believe this question will come up more and we will add those products as well to our comparison list.

The protocol is just standard/plain gRPC (https://www.cloudquery.io/docs/developers/architecture). Protobuf is defined here - https://github.com/cloudquery/plugin-sdk/tree/main/internal/...

Basically just the normalized, verified data + metadata, types

[ Parent | on HN ]

Re:Building a High Performance Data Integration Framework in Go by elchief on Monday November 28, 2022 @04:57PM (#33780431)

understood :) thanks for the reply

[ Parent | on HN ]

Re:Building a High Performance Data Integration Framework in Go by tomrod on Monday November 28, 2022 @06:45PM (#33781571)

Will you adopt the singer interfaces? That's what Meltano does, and it's chefs kiss.

[ Parent | on HN ]

Re:Building a High Performance Data Integration Framework in Go by yevpats on Tuesday November 29, 2022 @02:39AM (#33784699)

Interesting. We will take a look at this even though we usually build our own connectors due the huge concurrency/speed advantages when using CloudQuery SDK.

[ Parent | on HN ]