* JavaScript data processing framework for in-browser data processing
* Python framework like pandas
* Big data processing framework like Spark
* Database management system
* Data integration system like typical ETL and BI
* Stream analytics like Kafka Streams
* IoT (light weigh) stream processing engine
* Something else?
I would be very thankful for any suggestion from people who know the market and (acute) needs of the customers. What is the best niche for this kind of technology?
So if I were you, I would do Python DSL, because it addresses my immediate needs of increasing productivity. :)
Maybe you could use calcite[1] as an engine or as a base kind of like arrow.
If you do python, maybe look in pyjnius[2].
There's a lot of things that already have connectors. I would continue along your MVP route allowing folks to do basic things with your framework first then you can improve it as you go. SQL databases aren't a bad initial target. Most folks can do SQL.
[1]: https://calcite.apache.org/ [2]: https://github.com/kivy/pyjnius
There's a ton of things I'd be missing to start looking at this.
1. Backend agnostic: let me run on different backends like flink/spark
2. Give me off heap memory please. Let me play dangerous and use pointers directly to optimize interactions with transforms. We wrote our own GC among other things for our tensor lib due to the GC bottlenecks and copying and the like. That's personally what I kind of like about tablesaw.
A library for 1 off adhoc analysis in memory isn't a bad start though, especially since most folks don't actually have that large of problems.