Re:Apache Flink
by marcinzm on Wednesday January 30, 2019 @08:13AM
(#19034734)
>I believe it drastically reduces adoption of these tools because many of us avoid what appears to be bandwagon technologies as we don't want to consciously add layers of future technical debt when a majority of these projects will be abandoned.
Doesn't seem to be impacting adoption of the major ones from what I can see (Spark, Kafka, Flink). Assuming you're at a scale where you need these technologies. Which most companies are not so it's a good thing they don't implement them just for kicks or as "future proofing." If you have a business problem that the main frameworks can't solve then trying out some of the other ones may be worth the cost. There's also ongoing support for older frameworks (Storm) so it's not like your code becomes useless.
The approach open source is taking to resolve this issue is to embrace the diversity but create unified APIs on top of it. Apache Beam and Arrow for defining data workflows and data exchange formats respectively. There's also always SQL which works with a lot of the more data warehouse solutions out there (with some tweaks per solution unfortunately).
Doesn't seem to be impacting adoption of the major ones from what I can see (Spark, Kafka, Flink). Assuming you're at a scale where you need these technologies. Which most companies are not so it's a good thing they don't implement them just for kicks or as "future proofing." If you have a business problem that the main frameworks can't solve then trying out some of the other ones may be worth the cost. There's also ongoing support for older frameworks (Storm) so it's not like your code becomes useless.
The approach open source is taking to resolve this issue is to embrace the diversity but create unified APIs on top of it. Apache Beam and Arrow for defining data workflows and data exchange formats respectively. There's also always SQL which works with a lot of the more data warehouse solutions out there (with some tweaks per solution unfortunately).