Re:Ask HN: Good tool to visualize data?
by apu on Sunday November 08, 2009 @08:52PM
(#930212)
I've been building up a javascript library to do this, as visualizizing results/data takes up more than half my research time.
Unfortunately, this library is nowhere close to ready to be released, but I can describe roughly the architecture, in case it's useful to you.
I take all my data and write a python script which will dump it to a .JSON file. This includes the raw data/results, names of different fields, groups of fields to toggle on/off together, and what kinds of visualizations I want to use with a given set of data (view as numbers, as bar graph, as line graph, as linear time-series data, as color-coded images, etc.)
Then I have a standard html + javascript file in which I simply load in this JSON file. Because I've prespecified the format of the JSON file (I.e., what fields it has and how the data is stored), I only need to customize a few functions to display results. Things like filtering data, searching and sorting I get "for free", since they're in the library.
The main advantages:
- Interactive browsing of data in various formats, all in the web browser, with no plugins etc. required.
- Can be viewed locally or across the network
- No need for a server to be running
- Processing done on each local computer, as opposed to on some server
- HTML 5 is now good enough (just barely) to offer all the kinds of interaction I require
- Most of the code is in my standard library for visualizations, and so the time to create a new visualization for a new set of data is quite small.
Drawbacks:
- HTML 5 is still slow
- Can't do very advanced stuff yet, without writing a lot of custom code. On the otherhand, using jQuery + jQuery UI, it's very easy to make things draggable, for example, to compare various things side-by-side or even on top of each other (with transparancy)
- No server, so can't "save" complicated settings or parameters (although cookies help).
Anyway, in case you end up building something of your own, hopefully some of these ideas might help.
Unfortunately, this library is nowhere close to ready to be released, but I can describe roughly the architecture, in case it's useful to you.
I take all my data and write a python script which will dump it to a .JSON file. This includes the raw data/results, names of different fields, groups of fields to toggle on/off together, and what kinds of visualizations I want to use with a given set of data (view as numbers, as bar graph, as line graph, as linear time-series data, as color-coded images, etc.)
Then I have a standard html + javascript file in which I simply load in this JSON file. Because I've prespecified the format of the JSON file (I.e., what fields it has and how the data is stored), I only need to customize a few functions to display results. Things like filtering data, searching and sorting I get "for free", since they're in the library.
The main advantages:
- Interactive browsing of data in various formats, all in the web browser, with no plugins etc. required.
- Can be viewed locally or across the network
- No need for a server to be running
- Processing done on each local computer, as opposed to on some server
- HTML 5 is now good enough (just barely) to offer all the kinds of interaction I require
- Most of the code is in my standard library for visualizations, and so the time to create a new visualization for a new set of data is quite small.
Drawbacks:
- HTML 5 is still slow
- Can't do very advanced stuff yet, without writing a lot of custom code. On the otherhand, using jQuery + jQuery UI, it's very easy to make things draggable, for example, to compare various things side-by-side or even on top of each other (with transparancy)
- No server, so can't "save" complicated settings or parameters (although cookies help).
Anyway, in case you end up building something of your own, hopefully some of these ideas might help.