News for hackers. Stuff that matters.

nicholastmosher

Re:Aggregate streaming data in real-time with WebAssembly on Tuesday August 24, 2021 @04:06PM (#28294234)
No worries, it wasn't mentioned in this post in particular :)

Security is certainly one of the reasons to use WASM, the ability to run it in a sandbox means that untrusted user code can be uploaded to Fluvio's Streaming Processing Units and do the processing inline, rather than on the client side. This can save big on network bandwidth, especially with a dataset where filtering whittles down a lot on volume.

Other reasons include that WASM is a fast and portable bytecode format and that there is very good tooling and support for compiling Rust to WASM as well as embedding WASM runtimes in Rust, which works well for Fluvio as it's written in Rust.

Here's another post with a bit more detail about some of the design and motivational factors if you're interested: https://www.infinyon.com/blog/2021/06/introducing-fluvio/#fl...

Re:Aggregate streaming data in real-time with WebAssembly on Tuesday August 24, 2021 @03:09PM (#28293447)
Yes, this is very similar to "reduce" from the functional programming world, in fact it is equivalent to the "fold" pattern. The main difference is that fold is slightly more flexible since your accumulator may have a different type than your stream elements.

In Rusty pseudocode, reduce requires a function with two inputs and an output of the same type:

fn reduce<T>(f: Fn(T, T) -> T)

Whereas fold may use one type for the accumulator and another type for the elements, but requires an initial accumulator value to be given explicitly:

fn fold<A, T>(init: A, f: Fn(A, T) -> A)

The Aggregate SmartStreams discussed in the blog follow this fold pattern, applied to a distributed persistent log as the stream and using WebAssembly modules as the functions.