Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by mooreds on Friday March 27, 2020 @01:10PM
(#22704686)
Looks like the S3 connector was released in 2017. So if you wanted this functionality before then, secor was a good option: https://www.confluent.io/blog/apache-kafka-to-amazon-s3-exac...
It also supports going to other cloud object stores (other than S3).
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by chrisjc on Friday March 27, 2020 @01:31PM
(#22704893)
Yup, there was definitely a reason to consider Secor before 2017. Part of the reason I'm familiar with Secor in the first place.
Just to be clear, there are dozens of Kafka Connect Sinks and Sources, and finding Kafka Connect Sinks for other blob storage solutions should be easy. If you can't the KC framework makes it pretty easy to develop your own.
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by memorythought on Friday March 27, 2020 @01:29PM
(#22704880)
I don't think there's a good connect sink for GCS available under a non commercial license
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by chrisjc on Friday March 27, 2020 @01:40PM
(#22704966)
https://docs.confluent.io/current/connect/kafka-connect-gcs/...
> You can use this connector for a 30-day trial period without a license key.
Ugh, you're right...
https://docs.confluent.io/current/connect/kafka-connect-s3/i...