scottlf

I don't recall which of us is planning on attending DataEngConf SF, but I'm asking around now.

EDIT: Ah, it will be Vid Jain, the CEO of Wallaroo Labs.

No disagreement from me.

It's definitely not easy to keep queues shallow end-to-end. In my experiences with Erlang systems, queue management is an area where the BEAM VM's runtime is definitely not actively helpful enough. One anecdote: within the last few months, the weak scheme that the BEAM uses for runtime back-pressure was removed because it isn't effective enough to justify the complexity of its code.

When a system does have the ability to keep queues shallow end-to-end, then I think it's a good base to build on, adding additional features to allow deeper queues where and when we want them.

Regarding Wallaroo: Today, you can use Kafka as a data source, allowing Kafka to be your deep-as-you-wish buffer upstream and/or downstream of Wallaroo. Tomorrow (a.k.a. vapor, though we've discussed the feature internally) it's certainly feasible to add an option to (for example) `TCPSource` that would add a large buffer at the entry to a Wallaroo cluster ... with flexibility to queue in RAM, disk, or elsewhere. It would also make failure recovery more complex, and it's one reason why we've deferred implementing it.

Howdy. I'm the author of both parts of this 2-page series on overload mitigation. Let me know if I've overlooked an important technique for handling too-big workloads or if you have questions (e.g., how we've stitched together TCP's sliding window protocol flow control with Wallaroo's internal flow control).

-Scott

I, too, have a bit of experience with backpressure implementations in both Erlang and Pony. Erlang's "penalize the sender" doesn't work in many cases, so I'm not surprised that it's being removed. Erlang's remote distribution implementation & messaging semantics are Mostly Great but is definitely not Perfect.

1. Head-of-line blocking caused by congestion on the single TCP connection used for message transmission between any two Erlang nodes. This can cause major problems for apps control vs. data plane design, such as Riak and Lasp. Work on Partisan (https://github.com/lasp-lang/partisan) appears to be a substantial improvement.

2. If the single remote distribution TCP connection between two nodes is broken, then the first Erlang process to send a message to the remote node is, hrrm, well, borrowed/co-opted by the BEAM VM to connect the new TCP session. IIRC that process is marked unrunnable by the scheduler until the connection is set up or perhaps there's an error. If that process is really important to the functioning of your app, for example, an important system manager in the control plane of the app, then you have a very difficult-to-diagnose latency Heisenbug to cope with.

-Scott

Hi, everyone. I'm the author of this article. My apologies in advance for not covering non-DTrace tools available to Linux. The intent was "long article" and not "comprehensive book of everything about tracing". Please take a look at the list of references at the end of the article for lots of eBPF, uprobes, and SystemTap good stuff.

Good morning. The world has continued to turn, and now it's my turn to say hello and to offer a chance to answer questions from HN.

-Scott

Howdy. Erlang's modules are involved in code namespace for the compiler & runtime (i.e., the 'M' of the 'MFA' triple of Module + Function name + Arity that names a function) as well being the unit/scope/bound for hot code loading (i.e., you must load or unload an entire module at a time).

I've worked in Erlang-land far longer than I've lived in any OOP-land, so I'm not sure what you mean by enterprise'y OOP. Coincidentally, Pony's rules for "Packages" are something I just smacked my ignorant head against a few hours ago. The subsections of https://tutorial.ponylang.org/packages/ in the tutorial can probably answer at least some of your question: specifically "Package System" and "Use Statement".

Good luck trying to convince your virtual server hosting service provider to provide R2D2 or similar.