News for hackers. Stuff that matters.

codingmoh

The sticking point for me, though, is side effects. Once you need to call an external API—maybe to insert vector embeddings, send records to a SaaS service, or update some non-SQL store—you lose the comfortable ACID guarantees and pure SQL elegance. Even if you stage data in a DuckDB table, you still have to process each row or batch with an imperative approach. That’s where I start feeling the friction. SQL is brilliant for purely data-driven transformations; it doesn’t inherently solve "call this remote side-effect function in small batches, handle partial failures, and keep the pipeline consistent.

Can we unify those worlds? If your project, Sqlflow, manages to let folks stay mostly in SQL—while also elegantly handling side effects—that might be a huge step forward. For strictly data-focused workflows, I’m 100% on board that SQL alone is often the best "DSL" around. The complexity creeps in when we go from "write results to a table" to "call an external system" (possibly with partial commits, retries, or streaming needs). That’s usually where we end up rolling bespoke logic. If Sqlflow can bridge that gap, it’d be awesome. I’ll check it out—thanks for sharing.

Thanks for sharing that perspective. I do think jq is great for slicing up JSON before handing it off to Python—especially if your transformations are primarily stateless and can be expressed as simple map/filters. And yes, a lot of frameworks can turn into "magic black boxes" that complicate what should be simple.

That said, my gripe is that when you move beyond "transform this data" and step into "transform and then push to some API, handle partial success, retries, etc.," the line-by-line or chunk-by-chunk side-effects logic in Python can get gnarly fast. That’s the part I wish there was a standard, declarative approach for—something that doesn’t require a big workflow orchestrator (like Dagster) but also doesn’t devolve into tons of imperative glue code.

I agree no tool is a silver bullet. In many cases, plain, well-structured Python is enough. But once you need concurrency, chunk-based error handling, or incremental streaming with side effects, you end up coding your own partial solution anyway—and that’s where I start wishing for a more composable pattern in the standard Python ecosystem. Until then, jq + Python is definitely a solid approach for the simpler jobs!

Thanks for laying that out. You raise great points: Pandas (or Polars) chaining is indeed closer to what we want than raw imperative loops, and it handles a lot of the transformations. But it still assumes an in-memory, columnar approach, which doesn’t always gel with row-by-row side effects like pushing data to an external API. Also, real-world systems do require robust error handling and partial retries, which is exactly the scenario we’d love a functional pipeline to address by design, rather than patching in big try/except blocks. Tools like DAG orchestrators solve that at a workflow level, but not so much at the record/stream level.

So that’s the core frustration: yes, we can hack it in Python with a pipeline style, yes we can load data into DataFrames and transform it, but as soon as we do side-effectful calls (like collection.add(...) per record or batch), we have to drop out of the nice .pipe() flow. For me, it’s less about syntax and more about a missing standard approach to composable streams + side effects. Elixir Broadway, or even a DSL in Lisp/Clojure, addresses that more natively.

“Just use batch jobs / just use SQL transactions” works only if your external system supports transactions or if you keep your data in a single database. But many people use specialized systems (e.g. vector DBs, text embeddings) that don’t have an ACID-like model. Tools like Elixir Broadway do exactly that for streaming: concurrency, batching, backpressure, built-in retry, etc. That’s part of the reason we’re exploring new paradigms, rather than simply reading into Pandas or writing a standard “for line in file” script.

In short, I’m not saying Pandas can’t be coerced into something close to a pipeline. It’s more that we lack a standard, built-in, composable “stream of records -> transform -> side-effect-> handle errors” solution. Once you leave the purely tabular, in-memory paradigm, or need partial success handling, you quickly outgrow the typical DataFrame approach. That’s why I find frameworks like Elixir Broadway, or even an RxJS-like pipeline, so appealing — they make chunk-based side effects a first-class citizen.

So the question is: do we keep building one-off solutions in Python, or do we push for a more standardized, composable approach that can handle both transformations and side effects gracefully?

I think you’re right to point out the inertia in the labor market and the structural energy required for paradigm shifts.

What worries me most, though, is that because of exactly the reasons you’ve described — and now, with the rise of GenAI that makes generating boilerplate imperative code even easier — we might be permanently missing the opportunity to invent a better, more elegant solution.

Instead of challenging the underlying paradigm, we’re accelerating its replication — just with fewer keystrokes. GenAI is helping us pave cow paths, not design highways.

My fear is that the convenience of generating "good enough" code at scale will solidify suboptimal workflows, and the space for reimagining them might quietly close forever.

Ask HN: Why don't we have a functional DSL for data+embedding+API pipelines?

I’ve been working on a pretty common problem: - I have structured data in JSONL files (in.jsonl, out.jsonl) - I match lines by a key - I transform them into (text, embedding) pairs - I optionally filter/map them - I batch them (into chunks of 50) - I push each batch into an external system (e.g. vector DB, Chroma) That’s it. Sounds trivial. But it turns into ugly imperative...

( Read More... | 21 comments | 6 points | on HN )
Re: on Sunday March 30, 2025 @04:54PM (#43527539)
thanks!
Re: on Sunday March 30, 2025 @08:58AM (#43523808)
Hey, thanks for your feedback. That is great input, because I just realized that users might prefer to have a command printed out (inserted into clipboard) for them, instead of it being executed immediately. LocalLLMs are on my todo list! Also, what I want to work on next, is to make it possible to pipe errors / logs into the application, so it could help you troubleshoot and debug faster
Thank you so much!
I've made a very recent (1hr ago) Show HN post and it's not visible. Should be that one: https://news.ycombinator.com/item?id=43519517

Show HN: Built a Simple AI Code Reviewer in 3 Hours

https://ai-code-reviewer.replit.app:443/

Hey HN,I spent a couple of hours this afternoon building a basic AI-powered code reviewer using Replit. Based on your custom instructions, it reviews code on existing pull requests and can even comment in your name.It’s still pretty simple, but it works! I’m honestly surprised at how quickly I got it up and running (took around 2-3 hours). Thought it’d be fun to share here in case anyone’s...

( Read More... | no comments | on HN )
I saw your very recent comment on EMRs /EHRs and that you would like to build a new one.

we are actually in the process of building one, we have a doctor on board and we also have our first users.

Our goal is to create the most fantastic user experience for primary Care practitioners.

Our existing MVP uses AI to to speed up clinical documentation process using speech text to speech. Transcribing the conversation, summarizing it, generating a doctor's letter. We try to identify what actions the doctor recommended to the patients while talking to them. For instance, regenerate a laboratory referral in real time.

We are writing our yc application atm.

I can't see your email in your profile, but you should be able to see mine. I would love to talk to you.

Please drop me a message if you are still interested in building a new EMR.

Ask HN: How do you fix obscure bugs in large projects?

What are some effective strategies for finding answers to obscure errors in your software?I know that following clean code/architecture principles and adhering to organization-wide standards can help you navigate and understand the product/code faster, making it easier to identify the source of errorsStill, working on a new and large codebase/project, it often feels like the only way to find...

( Read More... | no comments | on HN )

Ask HN: What's your go-to bug tracking tool, and why do you prefer it?

What's your go-to bug tracking tool, and why do you prefer it? Are there any key features that you feel are missing from the tool you're currently using?I've been using Sentry for the past few years and have found it to be the best bug tracking tool in terms of user experience. I think, what I like most about, is definitely UX and the versatility of integrations it provides.

( Read More... | 3 comments | 4 points | on HN )

Show HN: Bugbusters.ai automated bugfixing using gpt-3

https://www.bugbusters.ai/

Hi everyone on HackerNews,We are just launching Bugbusters. Bugbusters is a GitHub-Bot that writes bugfixes for errors detected by application monitoring software like sentry. It examines an error's monitoring data, such as stack traces, in combination with the source code and git commit history to generate a potential fix. The bugfix will be submitted via a Pull Request that will also include...

( Read More... | 4 comments | 19 points | on HN )
Re:Ask HN: What problem do you wish someone would solve? on Monday February 07, 2022 @02:32PM (#30249798)
>I need a company to get extremely serious about desktop Linux and charge me $100/yr for something that actually works

Could you briefly describe, what you'd be looking for?