JavaScript Tooling for Data Pipelines

Posted by floyergilmour on Friday March 08, 2024 @10:13AM
from the self-post dept.

JS fullstack devs out there, what tooling are you using for creating data pipelines or for orchestrating data jobs? do you have to wander into SQL/Python land for this? or do you hack something together your self? As someone new to the JS community I have struggled to find a tool or platform for this. In Python land you have tings like Airflow and Dagster for example

( 2 comments | on HN )

Re:JavaScript Tooling for Data Pipelines by cotera-grant on Friday March 08, 2024 @11:23AM (#39642597)

We're a "data team as a service" type company and we actually just open sourced our TypeScript library for doing data pipelines. We're still pretty early in the open sourcing / docs process but we've used it in production for over a year now and are really happy with the stuff we've built on top of it.

A big bonus for us as fellow full stack JS devs is that the full compiler and type checker and SQL gen runs in the browser which we use to build cool stuff

https://getnasty.dev/ https://news.ycombinator.com/item?id=39641713

[ Parent | on HN ]

Re:JavaScript Tooling for Data Pipelines by floyergilmour on Saturday March 09, 2024 @05:12AM (#39650701)

Super cool!!! Is it almost like Drizzle but for BQ/Snowflake?

I don't understand from the website if NASTY also is meant to just run typescript code to do (lets say webscraping, update some postgres DB, check for missing billing stripe billing etc etc)

[ Parent | on HN ]