Re:Building a High Performance Data Integration Framework in Go
by yevpats on Monday November 28, 2022 @04:52PM
(#33780361)
Great question! We specifically started with connectors/plugins to cloud infrastructure providers and other infrastructure vendors - This is why most of our users migrated and/or compared us to the tools specified here (https://www.cloudquery.io/docs/cq-vs-others/overview).
There are no connectors for AWS, GCP, Azure in FiveTran, Airbyte, Meltano but as we extend the number of our source plugins I believe this question will come up more and we will add those products as well to our comparison list.
The protocol is just standard/plain gRPC (https://www.cloudquery.io/docs/developers/architecture). Protobuf is defined here - https://github.com/cloudquery/plugin-sdk/tree/main/internal/...
Basically just the normalized, verified data + metadata, types
Re:Building a High Performance Data Integration Framework in Go
by elchief on Monday November 28, 2022 @04:57PM
(#33780431)
understood :) thanks for the reply
Re:Building a High Performance Data Integration Framework in Go
by tomrod on Monday November 28, 2022 @06:45PM
(#33781571)
Will you adopt the singer interfaces? That's what Meltano does, and it's chefs kiss.
Re:Building a High Performance Data Integration Framework in Go
by yevpats on Tuesday November 29, 2022 @02:39AM
(#33784699)
Interesting. We will take a look at this even though we usually build our own connectors due the huge concurrency/speed advantages when using CloudQuery SDK.
Re:Building a High Performance Data Integration Framework in Go
by sghosh2 on Monday November 28, 2022 @10:06PM
(#33783219)
Much more of a focus on security and infrastructure than other EL tools. Like these security policies for example - https://www.cloudquery.io/docs/core-concepts/policies
Also, It'd be nice to know what the transfer protocol is like. What format is used to transfer between a Source and Destination?