jeffail

Personally, I'm happy to see them evolving it with my juices left intact. When we transitioned benthos.dev to https://docs.redpanda.com/redpanda-connect we gained a much larger (and more competent) docs team, but as a consequence had to trim a lot of the old personality. I don't regret the transition but I do take a bit of comfort seeing the WarpStream peeps continue to have fun with it.

It's like shedding an almost intact full body skin and finding comfort in watching colleagues take it bowling, having a great time.

Sure, if they change the MIT license of the core engine then you could fork it at that point. What they're doing right now is taking on a much larger maintainence burden than potentially necessary and fragmenting the ecosystem at the same time.

You're also at the same risk if you choose to use their fork.

It absolutely doesn't need a fork. The entire project is designed specifically to allow vendors and users to have their own ecosystem of plugins and they can all compile and integrate seamlessly. I'll be explaining live in 30 mins: https://www.youtube.com/watch?v=X8nVdUuWZ80

Hey it's Ash (the maintainer being talked about in the blog). I'm not one for fork drama and I haven't had a chance to fully read the blog so I don't have a lot to say. However, this is a full fork of the entire codebase, which means plugin authors will need to choose one project or the other and are locked in, and is entirely unnecessary on both a technical and legal perspective.

If they'd instead chose to fork the plugins themselves (the only parts where the licenses changed, all except two are Apache V2) then all users can pick and choose which ones they include in their projects, and it doesn't fragment the ecosystem at all. Your plugins would compile in my project, and mine would compile in yours.

The part they're choosing to fork here, which will cause this rift in the community, is still MIT licensed: https://github.com/redpanda-data/benthos. If they simply chose to continue using this MIT part we can all live happily together in a utopian society fully saturated with plugged blobbery.

Edit: I'm bit a baby brained so I forgot that I'm literally streaming live in 30 minutes in order to explain all the changes in detail for those out of the loop: https://www.youtube.com/watch?v=X8nVdUuWZ80

Sorry to hear that, unfortunately you can't please everyone whilst also having fun, and my open source work is unapologetically fueled by fun.

Thanks for sharing, this is awesome to hear!

Yeah that's my bad, the site assumes you already know what benthos is as it's an early stages UI for it. The best place to start is https://www.benthos.dev, or if you like dumb videos: https://youtu.be/88DSzCFV4Ng

Hey! The visuals are all via https://reactflow.dev/, which has been pretty awesome to use.

If I want to achieve something then I find the joy in pursuing it, at which point I will naturally do it when I'm capable and will do other things when I'm not.

Letting my attention float around freely (within limits obviously) often looks like I have a low attention span. However, with a birds eye view on what I'm able to get done I look like I'm hyper focused.

And if I can't find the joy in a long term pursuit then it's very unlikely to be worth my time.

Hey everyone, this is a video I put together summarising a decades worth of stream processing delivery guarantee misconceptions and bugs that I've seen frequently.

I'm not trying to scare anyone away from stream processing, in fact a lot of the issues outlined also apply to automated batch processing systems as well. Personally, I think that being realistic and pragmatic about failure conditions makes these systems less intimidating.

I hope one day https://www.benthos.dev will feature on one of these logo collages and I can finally retire to hobby watchmaking or something.

Anecdotally I'd say professions that often double up as a hobby and don't have a significant material or energy cost to exercise will be more susceptible to burn out. If you're _really_ into software engineering and struggle to pace yourself then burn out will often stop you sooner than finger fatigue or your computing costs.

I'm not saying workplace pressure etc isn't a factor, but those are also similar factors in pretty much all careers.

Maintaining an OS project definitely gives you a crash course in (and boosted respect for) non-engineering roles at tech companies.

However, I'd say if you're maintaining something and don't enjoy one or more of these hats then keep in mind that totally dominating a problem space doesn't need to be the end goal. There is such a thing as a lifestyle OS project and you can still make a success out of it, gain a loyal (perhaps modest) community, and enjoy taking it slow.

I'm massively rooting for this. I really wanted to use Gitter as the official chat for my projects, but the experience was just so painful that I ended up fragmenting the community across Discord and a gated Slack. I would love to have a forever home where I can feel comfortable getting everyone on board.

That makes more sense, thanks for clarifying.

Still, assuming there are no other edge cases there, it doesn't address the other problem where a hypothetical consumer of the output topic is reading an at-least-once feed of your exactly-once topic. In order for that not to be the case then the consumer must also be idempotent, in which case what value was gained from the original deduplication?

Yeah but the first time it was read the key was stored in RocksDB, so the second time it gets consumed after the crash:

"If the message already exists in RocksDB, the worker simply will not publish it to the output topic and update the offset of the input partition, acknowledging that it has processed the message."

It gets dropped as if it were a duplicate, oops!

Edit: I reread the relevant section...

"If a message was found in the output topic, but not RocksDB (or vice-versa) the dedupe worker will make the necessary repairs to keep the database and RocksDB in-sync. In essence, we’re using the output topic as both our write-ahead-log, and our end source of truth, with RocksDB checkpointing and verifying it."

Not sure entirely what they mean by "(or vice-versa)", if the message exists in RocksDB but not in the output topic how you distinguish between a real duplicate and a crash artifact?

If the last message of a crash happens to be a real duplicate and this recovery mechanism reintroduces it into the pipeline then you have a duplicate.

Either way, it's not an exactly-once feed. At best (assuming message loss isn't possible) it's an at-least-once feed that usually appears to be exactly-once.

"What’s more, we want to ensure the information about which events we’ve seen is written durably so we can recover from a crash, and that we never produce duplicate messages in our output."

Your processor is described as writing from Kafka to Kafka and using a persisted RocksDB instance to check message identifiers. How then do you ensure messages aren't dropped if your processor crashes or gets killed after checking against RocksDB but before the message is flushed to the Kafka broker?

Also is your producer writing to Kafka not at-least-once? If so then even if it removes all duplicates in its processing stage the feed written to your output topic could still contain duplicates.

By contrast deduplicating on consumption avoids that problem entirely by attempting to build an idempotent consumer, which results in an exactly-once. Although in this case they have identified edge cases of duplicates they're comfortable with.

"amounts to 1.5% of its worldwide turnover in 2017"

I imagine that's a significant sum but I'm struggling to get my head around it. If so then good for the ICO I suppose. I remember reading endless comments a few years back speculating GDPR would never have any bite.

Processor implementations are able to carry their own state, or share state across worker threads or deployments using their own mechanisms, but Benthos doesn't provide any tooling for that. None of the processors you get out of the box need computational state. You currently get an ALO stream (provided you use ALO protocols), vertical & horizontal scaling as per your config, and any glue you need between services.

It would be a nice stretch goal to have standard tooling within Benthos to share distributed state, perhaps with some ability to do exactly-once processing, but that's not the focus of the project right now.

All Benthos processors are stateless, except for a few such as batch, combine, etc, which do basic tasks such as building up batches from consecutive messages. It's possible to create sliding window processors on top of Benthos, but there aren't any general tools for doing that yet.

Yeah just boring. Benthos is mostly a collection of standard processors for doing boring stuff.

Hey, the HTTP output has a fixed number of retries, after which you could either have some mechanism in place to fall back on or by default it will simply continue the retries again whilst blocking upstream. You might also be interested in running Benthos in streams mode: https://github.com/Jeffail/benthos/tree/master/docs/streams

Streams mode lets you run as many isolated stream pipelines as you want in the same process, which in your case could be a simple queue -> webhook bridge. You can manage these pipelines either statically in config files, or dynamically through a REST API.

Hey, Benthos doesn't yet provide any general tooling for exactly-once processing like Wallaroo does, that's possibly a goal for the future.

My main focus has been providing general purpose stateless processors. So you can build a your own stream processor focusing on what makes it unique, and then the moment it's compiled and packaged it can read and write to anything, and convert any kind of payload to anything else just through configuration.

Hey everyone, author here. We use Benthos as a general stream swiss army knife for all the dull tasks, but you can also use it as a framework for writing your own stream processors in Go.

It doesn't provide any tools for idempotent calculations yet, just at-least-once delivery guarantees. But when you use it as a framework you get the benefit of all the configurable processors in your new service.