chenquan

Today, with great excitement, we proudly introduce a revolutionary update to the ArkFlow stream processing engine: full support for Python processors! This is not just a simple feature iteration; it marks a solid step towards more intelligent, user-friendly, and powerful real-time data processing. We are keenly aware of Python's immense influence and its vast ecosystem in the fields of artificial intelligence and machine learning. Now, with ArkFlow, you can seamlessly integrate all of this into high-performance, real-time stream processing pipelines.

In the recently merged PR, ArkFlow supports Python processors, which allows us to call Python scripts directly in ArkFlow. Using the Pyarrow library allows users to write data processing logic more freely. At the same time, relying on Python's powerful AI ecosystem, ArkFlow will also be able to call any Python machine learning/deep learning library (TensorFlow, PyTorch, etc.) and large models. We will have more detailed articles to introduce relevant use cases in the future.

Hello, I am the founder of this project and I am very happy that a friend has shared it.

ArkFlow is positioned as a lightweight distributed stream processing engine that integrates streaming batches. With the help of datafusion's huge ecosystem and ArkFlow's scalable capabilities, we hope to build a huge data processing ecosystem to help the community simplify the threshold for data processing, because we always believe that flowing data can generate greater value.

Finally, thanks to everyone for their attention.

I'm writing a lightweight high-performance stream processing engine：https://github.com/arkflow-rs/arkflow

docs:https://arkflow-rs.com/

I’m grateful for your assistance.

Hi, then I'm curious what do you think people care most about in data processing and database workloads?

Yeah, I'm looking forward to having further conversations once I get to know tremor.

Welcome to follow anytime.

I don't know it exists.

Hi, friend. How did you do it specifically?

> Stability and safety are the least of the concerns in data processing and database workloads.

I'm curious how you came to this conclusion?

Hi! Brother, I think I will seriously consider what you said and I am honored to communicate with you.

Good job, this is a rich reference.

Hi,brother! I'm still thinking, but it's certainly not now.

Why do you have to beat a native performance-oriented streaming engine written in C or C++?

Currently, most of the mainstream stream processing engines are written in Java. Sorry, I may not add qualifiers to make you misunderstandings.

Software does not have silver bullets, so does programming languages, and each has its own strengths. I also like to use go and Java to develop software.

Yes, whether it is DataFusion, Arroyo, or Bentos, these open source products have made me profit a lot.

Yes, RPCN (redpanda connect)/old benthos is very cool and can solve most of the scenes. Let me tell you quietly that I am using it too.

Arroyo has been designed with more comprehensive consideration.

I think stability, reliability and high performance are the foundation of infrastructure.

Thank you for your valuable experience, I will seriously think about what you said.

Sorry, please forgive me for not knowing Arroyo completely.

Welcome to follow the latest news from ArkFlow at any time and even participate.

Sorry, not yet, but this is the direction ArkFlow is working hard. Rust's own potential will also guide ArkFlow in this direction.

Yes, Arroyo is entirely based on DataFusion, but ArkFlow is not exactly. In the future, ArkFlow will establish a plug-in ecosystem, allowing anyone to process data through plug-ins, not limited to DataFusion.

Thanks for your recommendation.

I think a stream processing engine written in rust will have better performance, lower latency, more stable services, lower memory footprint, and cost savings. At the same time, ArkFlow is based on DataFusion implementation, which will put ArkFlow on a strong open source community.

Yes, they are similar. ArkFlow is mainly based on DataFusion. Bento actually comes from Benthos. Currently, the ArkFlow project is in the early stages and no performance comparison test has been conducted, but I believe that ArkFlow will outperform them in the long run.

Benthos: https://github.com/redpanda-data/benthos

DataFusion: https://github.com/apache/datafusion

High Performance: Built on Rust and Tokio async runtime, offering excellent performance and low latency Multiple Data Sources: Support for Kafka, MQTT, HTTP, files, and other input/output sources Powerful Processing Capabilities: Built-in SQL queries, JSON processing, Protobuf encoding/decoding, batch processing, and other processors Extensible: Modular design, easy to extend with new input, output, and processor components