achille-roussel on Hacker News
https://github.com/firetiger-oss/go-duckfs
I made this because I kept running into issues and limitations when trying to embed DuckDB in Go applications: because DuckDB and its extensions do their own I/O in C++ land, integrating with the rest of my applications to get authentication or instrumentation working was really challenging. With go-duckfs, all this happens on the Go side, it's like a sandbox for all I/O that happen during...
It's a really nice first-hand tale from the perspective of a junior engineer getting into the workforce, while the industry is going through the revolution of coding agents.
Worth the read!
The knee-jerk reaction is often to add a queue! Sure, it helps for a while. But queues introduce their own complexity: handling failures, managing retries, creating visibility... It's a band-aid, not a cure, and we've been wrestling with this problem for too long!
In this blog post, we'll break down:
- Why queues fall short when building truly resilient integrations - The core principles behind building scalable, fault-tolerant async workflows - Practical techniques that go beyond the limitations of queues
If you're done with fragile systems and want to level up your integration game, this one's for you!
Take a look at the "Local Testing" section in the README of dispatch-py https://github.com/stealthrocket/dispatch-py?tab=readme-ov-f...
TLDR: python -m dispatch.test http://127.0.0.1:8000
This is just a mock implementation but it lets you test your code and experiment with the solution.
I invite you to check out part 2 of this series, which focuses more on the Python implementation https://stealthrocket.tech/blog/distributed-coroutines-in-py...
Stay tuned for more content as well, we couldn’t fit everything in one post!
With Dispatch, you just write code, as if you would write a local program, and our scheduler can manage the distribution and state of the execution. Testing the code especially gets really simple.
Having a hosted scheduler also means that you don't need to deploy any extra infrastructure like RabbitMQ or Redis.
Our take is that distributed coroutines can be brought to general-purpose programming languages like Python so that engineering teams can adopt these features incrementally in their existing applications instead of adding a whole new language or framework.
In my opinion, the value is in being able to reuse the software tools and processes you're familiar with, major shifts are rarely the right call.
A good starting point to answer your other questions is the "getting started" section in our documentation https://docs.dispatch.run/dispatch/getting-started
If you have more questions, join our Discord! We're always happy to chat and get feedback on what's difficult to grasp.
This is why these solutions have often seen the creation of a new programming language as a necessary step to avoid those issues altogether. We think it's important to bring those capabilities to mainstream languages, we can't always go back from scratch with a new language.
Regarding where the state is stored, we keep everything in S3 and have plans to allow users to provide their own object store if they need to keep the application state in a storage medium that they own.
Dispatch can also handle the "one-off" jobs you describe, where you don't need to track the coroutine state. In a way, it's a subset/special case of the distributed coroutine (just like functions are a special case of coroutines with no yield point).
A load balancer can help distribute new jobs across a fleet, but even the shortest of jobs can become "long running" when it hits timeouts, rate limits, and other transient errors. You quickly need a scheduler to effectively orchestrate the retries without DDoS-ing your systems, and need to keep track of the state to carry jobs to completion.
Combine a scheduler (like Dispatch) with a primitive like distributed coroutines, and you've got a powerful foundation to create distributed applications of all kinds without seeing complexity skyrocket.
Say your coroutine is performing a transactional operation with multiple steps, but for some reason the second step starts getting rate limits from an upstream API, you need to enter a retry loop to reschedule the operation a bit later. What if your program stops then? Without capturing the state somewhere, it remains volatile in memory and the operation would be lost when the program stops.
This scenario occurs in a wide range of systems, from simple async jobs for email delivery to complex workflows with fan-out and fan-in.
> Isn't it expensive to serialize coroutines and transfer them between machines?
We only capture the local scope of the call stack, so the amount of data that we need to record is pretty small and stays in the order of magnitude of the data being processed (e.g., the request payload).
> I wonder what observability is like. If a coroutine crashes, what its stacktrace will look like?
We maintain stack traces across await points of coroutine calls, even if the concurrent operations are scheduled on different program instances. When an error occurs, the stack traces show the entire code path that led to the error.
There's definitely a lot of opportunities for creating powerful introspection tools, but since Dispatch is just code, you can also use your typical metrics, logs, and traces to instrument your distributed coroutines.
People have often tried to use Temporal to solve similar problems. We think Dispatch is a very different solution because we don’t require you to architect your system in terms of a rigid framework of activities and workflows... You just write regular code with simple functions, enabling incremental adoption into existing applications and making the code easy to test.
A friend of mine once described Dispatch as "Temporal without the galaxy brain", it's a pretty good way of phrasing it in my opinion :)
Serializing file handles doesn't work, but in our experience, programs rarely run into constructs where this becomes a problem, and when it happens, there are mitigation measures that are usually easy to implement (small restructure of the program, capturing resource metadata to reconstruct them later, etc...).
We have a few features on the roadmap to help mitigate the security implications as well, including allowing users to store their program state in a S3 bucket that they own. Our scheduler can operate with only metadata about the program, so splitting the two can be an effective model to mitigate those risks.
> That's great. Is it persisted to retry even after a reboot?
Absolutely! Our scheduler service manages the state on behalf of the application, even if the program restarts, the operation persists until completion.
> What informs the end user, in interactive situations, and what does it tell them?
Since the Dispatch SDK integrates directly in the code of the application, it's compatible with typical system observability tools. Say you're using metrics or traces to track operations, you can emit this telemetry on your dispatch functions. That being said, we know how important it is to have a built-in solution for this!
> What if the operation is delayed so much that it no longer makes sense; can we cancel it?
Different ways to solve this, one is to simply raise a permanent error so the operation stops, but in some cases it makes more sense for the system to expose controls so the operator can act on group of similar operations for example. Those are features on our roadmap at this time, but definitely agree on the importance!
> How do we identify a particular one, out of thousands going through the same function: by some of the arguments, like in memoization?
Searching through the state or history of the systems is also a key feature that a platform like Dispatch needs to provide to be "complete". We're still in early development so we haven't gotten to add this feature yet, but definitely excited to delivering it!
A major barrier to Erlang is adopting a new programming language, development tooling, etc...
What we aim to do with Dispatch is bring those distributed programming superpowers to mainstream programming languages that are developers are using.
A major conviction we have is that it should be possible to adopt Dispatch incrementally in an existing application.
What we hope to deliver with Dispatch is a model that is a lot more incremental in its adoption and much simpler than what existing solutions have been able to achieve so far, both in terms of development UX and operational complexity.
Give it a try, and let us know what you think of it!
Today marks an important milestone for us with the launch of the developer preview of Dispatch.
Along with a serverless scheduler, we released an open-source Python SDK, providing developers with a simple programming model to integrate with Dispatch. This is the first of many steps on our mission to make distributed application development radically simpler.
To all developers tired of wrestling with infrastructure and eager to experience better tools to develop distributed applications, check out our blog post and take Dispatch for a ride!
Today marks an important milestone for us with the launch of the developer preview of Dispatch.
Along with a serverless scheduler, we released an open-source Python SDK, providing developers with a simple programming model to integrate with Dispatch.
This is the first of many steps on our mission to make distributed application development radically simpler.
To all developers tired of wrestling with infrastructure and eager to experience better tools to develop distributed applications, check out our blog post and take Dispatch for a ride!