destel

Just be careful with model personalization. Gemini once learned that I'm an engineer and now "translates" all conversations into engineering language, even if we talk about some philosophical stuff

I had similar bad results with gpt 4o/5 when they got these image generation capabilities.

Don’t know what’s the reason: my bad prompting or these models being tuned to work with photos/illustrations only

Some examples are mind blowing. It’s interesting if it can generate web/app designs

You're not late. Just yesterday I've configured new reply notifications via RSS.

Typically function "f" does two things. 1. Performs calculations (this can be parallelized). 2. Writes results somewhere (this must happen sequentially and in the correct order).

Here's the typical OrderedLoop usage example from the article:

  OrderedLoop(in, out, n, func(a A, canWrite <-chan struct{}) {
   // [Do processing here]
   
   // Everything above this line is executed concurrently,
   // everything below it is executed sequentially and in order
   <-canWrite
   
   // [Write results somewhere]
  })

Without the "canWrite" the entire body of the "f" will be executed either concurrently or sequentially. With it - we can have this dual behavior, similar to critical sections protected by mutexes.

It's worth mentioning that OrderedLoop is a low-level primitive. User-level functions like OrderedMap, OrderedFilter and others do not need the "canWrite" channel to be exposed.

Thanks. This replyTo pattern is very similar to promises in other languages.

UPD.

I've just made a small but important clarification to the article. While in many cases it's easier and even preferred to calculate all results, accumulate them somewhere, then sort; this article focuses on memory bound algorithms that support infinite streams and backpressure.

I've just made a small but important clarification to the article. While in many cases it's easier and even preferred to calculate all results, accumulate them somewhere, then sort; this article focuses on memory bound algorithms that support infinite streams and backpressure.

Wow, that’s some seriously sophisticated stuff - it’s not that often you see a heap used in typical production code (outside of libraries)!

Your first example definitely gives me merge-sort vibes - a really clean way to keep things ordered across multiple sources. The second and third scenarios are a bit beyond what I’ve tackled so far, but super interesting to read about.

This also reminded me of a WIP PR I drafted for rill (probably too niche, so I’m not sure I’ll ever merge it). It implements a channel buffer that behaves like a heap - basically a fixed-size priority queue where re-prioritization only happens for items that pile up due to backpressure. Maybe some of that code could be useful for your future use cases: https://github.com/destel/rill/pull/50

I haven’t used Java for about a decade, so I’m not very familiar with streams api.

Your snippet looks good and concise.

One thing I haven’t emphasized enough in the article is that all algorithms there are designed to work with potentially infinite streams

Hi everyone, I’m the author of the article. Happy to answer any questions or discuss concurrency patterns in Go. Curious how others tackle such problems.

Hi everyone. I built a tiny web game where the goal is to pick the lowest number that nobody else picked. It runs once per day — if someone picks the same number as you, you’re both eliminated.

Since the mechanics are simple, I focused on feel: vintage-poster styling, small animations, and a snappy number-picking interaction.

By day I’m a Go backend developer, so this pushed me into much more frontend/design than usual.

I’d love feedback on both the gameplay and UX. I’m also genuinely curious: how large can the smallest winning number get as the player count grows?

Stack: Nuxt, Supabase and Cloudflare workers

Author here. While the article uses database updates as the example, the same pattern works great for any high-frequency API calls or operations that could benefit from batching. The goal was to make the solution completely transparent - developers just call a normal-looking function, and the batching happens automatically under the hood. I've used several variations of this pattern in production, and it's been particularly effective at reducing costs and latency.

Author here. Recently started a personal dev blog - happy to answer any questions.

Thank you very much for the feedback. I thought about something similar some time ago. Buffer of size one, then measure the average time each item spends in the buffer. But for debugging your approach is simpler and more practical.

Thank you. I "stealed" the name from scala. They have the similar value+error type. Maybe in context it rill the better name could have been "Item"

I have to be honest - I haven't ever heard about it. Just checked and found it's very mature and popular, though it seems to have had no commits in the last 2 years.

After quick scanning, I'd highlight these 2 differences: - Rill is type safe and uses generics - Rill does not try to hide channels, it just provides primitives to transform them

I have some benchmarks in the project's wiki on Github. I can confirm your point: Rill's main bottleneck is channel operations, the library itself adds negligible overhead on top. Of course, for multi-stage pipelines the number of channel operations grows.

To summarize, I believe Rill's performance would be fine for any problem solvable with Go channels. I've used it for a wide variety of use cases - while I'm not sure I can call them high-throughput, I've had pipelines transferring hundreds of GBs of data with no performance issues.

Thank you for the feedback. I agree with your point.

The current solution is to make pipeline stages context-aware (which is often happens automatically) and cancel the context before returning. This is the responsibility of the user and can lead to problems you described.

I haven't yet found a better solution to this. On the other hand, exact same thing happens if your function spawns a goroutine and returns. That goroutine would run until done, unless it's context aware.

Regarding the "Delay" and "infiniteBuffer" functions - these are part of the work on adding support for feedback loops to Rill. I haven't yet found a reliable and user friendly way to do it, so this work is on hold for now.

Thank you! I took a quick look at mgmt, it's quite a large and complex project. I'd need to better understand your DAG-based concurrency patterns to say if Rill would be a good fit. Could you share some examples of the concurrent runners you're thinking about? This would help me understand if/how Rill might be useful for your case.

Thank you! There are two pieces of motivation here. The first one is removing boilerplate of spawning goroutines that read from one channel and write to another. Such code also needs wait/err group to properly close the output channel. I wanted to abstract this away as "channel transformation" with user controlled concurrency.

Another part is to provide solutions for some real problems I had. Most notably batching, error handling (in multi stage pipelines) and order preservation. I thought that they are generic enough to be the part of general purpose library.