Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by mooreds on Friday March 27, 2020 @12:35PM
(#22704354)
Are you still using kafka? Does Kinesis Firehose read from Kafka?
[ Parent
| on HN ]
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by chrisjc on Friday March 27, 2020 @12:50PM
(#22704497)
No. If you want to use Firehose, you need to transfer data from Kafka to Kinesis first. You can probably use one of many Kafka Connect Sinks for this. Eg: https://github.com/awslabs/kinesis-kafka-connector
At that point, might as well set up a Kafka Connect cluster with a Kafka Connect Sink, and dump to S3 yourself.
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by otterley on Friday March 27, 2020 @01:18PM
(#22704767)
Why not go whole-hog and use Kinesis Streams and eliminate Kafka altogether?
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by chrisjc on Friday March 27, 2020 @01:27PM
(#22704853)
Not sure if you're being factitious, but it sounded like parent's data was already in Kafka.
But you're right, no reason not to start with kinesis in the first place, especially with all the AWS options around it including Kinesis Analytics (Streaming "SQL"), Flink (supported by AWS now) or plain old EMR (Spark, Flink, etc...).
There is a lot of work that goes into maintaining your own Kafka ecosystem, even if you use Confluent, Lenses, etc...
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by lflux on Friday March 27, 2020 @01:32PM
(#22704902)
Good question. Kafka is cheaper for things that don't need to go into redshift, and there's a fair amount of engineering effort that would go in to switching those existing systems over.
Re:Secor: A service persisting Kafka logs to S3, GCS and Azu...
by lflux on Friday March 27, 2020 @01:25PM
(#22704837)
We've phased out Kafka for the parts that feed Redshift, since Kinesis Firehose is a far better experience. We still have Kafka for our application logs and performance monitoring
Kinesis Firehose doesn't read from Kafka, we switched our code to publish to Kinesis instead.