News for hackers. Stuff that matters.
Re:Show HN: Interactive map for architecting big data pipelines by jdoliner on Tuesday June 27, 2017 @04:37AM (#14643410)
If you're looking for something that doesn't constrain you to a particular language take a look at Pachyderm. It's built around containers so you can run any code you want. I designed it with JVM-phobes like you (and me) in mind.

https://github.com/pachyderm/pachyderm

Re:Show HN: Interactive map for architecting big data pipelines by lobster_johnson on Tuesday June 27, 2017 @11:54AM (#14646551)
Cool, thanks! I haven't looked closely at it, but the "version controlled" part is something I don't need/want. Does it get in the way at all? I'm mostly looking for incremental, semi-real-time streaming processing, not something where you shoot off a big job on a dataset and get back results.
Re:Show HN: Interactive map for architecting big data pipelines by jdoliner on Tuesday June 27, 2017 @06:42PM (#14650221)
The version control semantics of the system are pretty crucial for some of the features you describe wanting. Pachyderm supports incremental operations on stream-like datasets. But what's going on under the hood is that the dataset is being version controlled and thus the system can tell which data has changed and only process the new data. Hope that helps, I'd be happy to chat more about your specific use case. Shoot me an email at [email protected]