Re:Building a streaming SQL engine with Arrow and DataFusion
by jnewhouse on Monday March 18, 2024 @01:25PM
(#39747225)
Yep, pretty much. Right now filesystem^ sources are finite, scanning the target path at operator startup time and processing all matching files. This processing is done by opening an asynchronous reader, courtesy of the object_store crate.
^We call these Filesystem Sources/Sinks to match terminology present in other streaming systems, but I'm not in love with it.
^We call these Filesystem Sources/Sinks to match terminology present in other streaming systems, but I'm not in love with it.