- would You actually be interested to buy this service?
- what sort of visualizations do You actually make? Do they need to be interactive? SVG? Size? How do You use them?
- what exactly do You do with data before its plotted except selecting columns? Is there aggregation or any kind of processing?
- how often is this actually used because You say 'sometimes every few months', does it mean its like a quarterly report?
- what other well established tools have You used other than Excel?
- how big is Your largest data? Size, rows, columns
- if it applies with small amount of megabytes, is there a reason beside simplicity why You can't use PivotChart in Excel? Or Excel in general? Or R/Python to generate it?
I am data scientist who regularly plots quite large data sets, and I like speed :) Its totally doable to build a service that You can run locally, load a CSV, read like 1% of data, play with it.. when you get what You want, load rest and wait a bit and get the visualization You want.
But depending on visualization requirements there may be many paths solutions.
Count sketches, Reservior sampling and similar methods come to mind.
I think I have an email address in my profile; feel free to send me something. I am fairly certain that your needs can be satisfied with existing Unix tools. Then again, the reason I worked on the problem in the 1990s was to free up engineer time so they could do more valuable things. A gui and other tools could be worth paying for if the bosses have that mindset.
a) try to plot their data alone and spend time on hacking the stuff together. This takes time as the guys doing it aren't accustomed doing it daily. This happens accross all kind of divisions.
b) ask another team (with data scientists) for their support. Maybe the engineer has to write a ticket, or the person who should be doing it has other tasks, is in vacation, not willing, not replying to the request etc.
Either way hours are easily spent on solving this seemingly simple task. The amount of time spent is simply staggering.
Unix would also be my personal choice. But getting the right to put a unix machine into the network for a single user is extremely difficult. Windows, Internet Explorer and temporary admin rights are the work environment that almost everyone has to use. That's why I think a web based solutions is the only viable option.
Is there a way I can reach out to you?
Thanks!
We want to plot big data (up to terabytes). Columns should be selectable by gui and nameable. The Data then should be be added to database with an ID. Everything should be usable without use of a scripting language.
Right now the terabytes of data have to be loaded in to ram just to see the first few lines and determine what the columns stand for. Now I know that there are editors that can load data partially but these have to be reinstalled which requires admin rights etc. This is a huge burden in a big company! The process of simply plotting, selecting and storing data takes a huge amount of time. The solution should be web based because no admin rights are availabe.
Often I am impressed how many tools and hacks exist simply to get one thing done: visualize measurement data. Excel is not enough because even the import of dot vs comma vs tab etc takes too much time and everytime has to be relearned. Engineers have to plot the data sometimes every few months and then you have a new excel version that autocorrects measurement data to dates or whatever.
In my opinion this would solve an obscene amount of work. Right now every engineer is hacking together some scripts that are extremely inflexible. When just csv-type data has to be handled.
Edit: this also applies to smaller amounts of data of megabytes. How can we plot them more robust than excel and then select x and y axis? I am pretty sure that we would love to buy a product that solves these issues.