Re:Show HN: Capillaries: Distributed data processing with Go...
by kleineshertz on Monday May 15, 2023 @09:28PM
(#35956763)
Temporal is a different ecosystem (and a much more ambitious solution), but one of the principles is the same: users want a platform that solves scalability issues and lets them focus on biz logic and customer value.
Re:Show HN: Capillaries: Distributed data processing with Go...
by lorendsr on Tuesday May 16, 2023 @01:25PM
(#35965000)
Temporal also shares the principle of being tolerant to database and processing node failures
Re:Show HN: Capillaries: Distributed data processing with Go...
by lorendsr on Tuesday May 16, 2023 @01:58PM
(#35965412)
Temporal has different database options: Cassandra, Postgres, MySQL, SQLite.
> source data in CSV files; - Capillaries script (JSON file) that defines the workflow and the transforms; - Python code that performs complex calculations (only if needed).
Temporal is more general purpose: source data anywhere, and you write code to define workflows and transforms instead of JSON, and the code can be in Go/Java/Python/JS/TS/.NET
Re:Show HN: Capillaries: Distributed data processing with Go...
by kleineshertz on Tuesday May 16, 2023 @02:56PM
(#35966081)
Right. And this is where our paths go separate ways. As I see it, Capillaries users are not necessarily tech companies and they do not have much appetite for writing and maintaining a lot of code. All they want is to run, say, 50 kinds of workflows on a regular basis and to keep those workflow definitions very formalized and stable. It would be hard to sell the idea of maintaining 50 different codebases to their management.
As for more complex calculations: in Capillaries, Python is not a programming platform, it's just a scripting engine.
Re:Show HN: Capillaries: Distributed data processing with Go...
by lorendsr on Wednesday May 17, 2023 @02:19AM
(#35971856)
There are definitely ease of use benefits to more tailored solutions. If workflow definitions are really simple and don't change much, JSON might be easy. Most things I prefer the DX of writing the logic in code. And it wouldn't be 50 different codebases—it would usually be a single codebase with 50 different functions.
Next up, TemporiarriesLite™
Go + SQLite using https://litestream.io for single-instance, low-power systems or server-less, seldom-used apps that need distributed backups and statefulness.
Jokes aside, these at-least-once operation state managers are really nice and help us avoid adding SQS / NATS / etc.. queues littered all over the place. The focus on data processing by Capillaries is nice. Looking forward to trying it out.