I believe my http://easymorph.com would also count as competition. It loads CSV files and queries databases, filters, pivots/unpivots, cleanses, aggregates, merges. Free edition. Windows only (yeah, Mac owners, I know).

Looks fantastic. I've been complaining for years that there wasn't any program like this, and even started many times working on it. Thumbs up!

Thank you!

> Sad to see it's an Electron app, but I don't want to start this flamewar all over again

Then stop mentioning it. Just don't use anything electron and don't mention it. I will use my RStudio and VS Code.

I think it is ok to express displeasure at the state of something without just silently doing something else and hoping that the entire ecosystem catches up to you. For example, I take every opportunity (like right now, for example) to shit on node.js and npm for that very reason. There is so much evangelism on one side, it is nice to hear the opposing view points.

> I think it is ok to express displeasure at the state of something without just silently doing something else and hoping that the entire ecosystem catches up to you.

It is, but it's not okay to bring up a common flame war topic and then say "but I don't want to start a flame war". If you don't want to start one, just don't start one. Starting it, and then saying you didn't want to, is disingenuous at best.

It's a legitimate thing to mention as being displeased about. The very fact that it's such a hot topic demonstrates that, since it indicates that it's a subject over which smart people can disagree.

The "I don't want to start a flamewar" bit is also a reasonable addendum. It's a terse way of saying, "I don't like this feature, but, people who disagree with me, please don't get up in arms about this, you're free to like this feature if you want."

Frankly, if there's anyone trying to start a flamewar, it's not the original poster; it's people who are edging toward caping up in response to OP's (quite mild) comment.

Ok, so what are some legit criticisms of node.js and npm? I'm no fanboy, but every time I try to ask this, the answers are along the lines of "if you install packages in a different order, you get different results." Which is fair, but everybody installs using `npm i`, so that's mostly a moot point.

Is there any substance here?

I think it's fair to talk about this, as long as we go out of our way not to reduce it to flamewar territory. The app is built on Electron, after all.

Maybe avoid mentioning memory usage, since that point has been done to death.

What counts as a legit criticism? So much of our field is taste-driven. And so much criticism was aimed at old iterations of something, gets carried on in flamewars, but no longer applies to the latest version (e.g. "Java is slow").

All the JS-the-language criticisms apply, those are probably the biggest at this point. The classic "node.js is cancer" is still somewhat relevant though I never thought it made its point well enough on how forced-asynchronous is at least as bad as the forced OOP and checked exceptions we put up with in e.g. Java again. Callback hell was long a common complaint, until people found Promises / the waterfall structure, but then the complaint is debugging hell. The new await stuff in node 8 sounds nice at least.

For a long time npm didn't get stuff over https by default, I think it does for a year now? The only other criticisms I can think of are just shoddy engineering that turns into drama (I remember seeing something about their colorized output would slow everything down quite a lot) or more taste-level things like how seemingly trivial concepts like leftpad are core libraries (that go through multiple versions because it couldn't have just been done right the first time) and the drama involved when something everyone depends on has an issue.

Disclaimer, I haven't done production node since ~2013. It was actually mostly enjoyable, and I wouldn't mind doing it again, despite on the taste-level I think the whole node ecosystem is just worse than many other options (when you have a choice) for many specific problems. But life is a continuous lesson in Worse is Better.

One drawback of Node.js is that it's single threaded. Moreover, even if somehow it became multithreaded today, JS is absolutely not ready for that. (Btw. yes, async I/O is nice, but threads are useful for more than just I/O.)

NPM is quite basic, it pretty much just downloads tarballs recursively from a webserver according to some version spec. The CommonJS module format is fairly loose (and kind of outlived its usefulnes) and so there's no agreed upon package structure and everyone just does whatever they feel like doing, which leads to a huge overall mess. The whole ecosystem is fragmented and very few things are really agreed-upon.

> NPM is quite basic, it pretty much just downloads tarballs recursively from a webserver according to some version spec

This is about as valuable as "Rails is a webserver that responds to some HTTP verbs". Common.js isn't even a part of npm (the correct capitalization) or node.

> Common.js isn't even a part of npm (the correct capitalization) or node.

I never said it was, it's not part of it per se, but it's the 'blessed' module/package system on Node.js (the correct capitalization). Which is really sad, because CommonJS is retarded (a file has to actually be executed in order to get it's require & export characteristics and they might arbitrarily change in runtime). It's sad that Node.js as well as the JS ecosystem in general is fairly hostile towards ES6 modules - basically any current support of ES6 modules is just translating then / dumbing them down back to CJS. Hopefully this'll change in the future as ES6 gets more adopted and CommonJS will be left to rot (a fate it deserves and should've already met ages ago).

A mature framework should not have this kind of problems https://github.com/nodejs/node/issues/12115

I don't want to start a flamewar, either, but I'm glad that the top poster mentioned that it's an electron app. Having that knowledge ahead of time allows me to manage my expectations.

All great feedback, thanks! Some quick replies: - The omission of '<>' from the operator list was just an embarrassing mistake on my part. #facepalm #WILLFIX - Also agree that a drop-down list of checkboxes (ideally a searchable one!) with an 'in' operator for low cardinality columns would be great. But the UI for that is a bit involved, and probably also need to provide some user control over when to do this since gathering distinct values can be an expensive operation for large data sets. - Totally agree that it would be useful to provide more interesting column stats. Thanks for the suggestions and refs to other tools -- very helpful.

You can check RStudio's data viewer for reference https://support.rstudio.com/hc/en-us/articles/205175388-Usin...

It's a modified dataTable(so search is builtin. I didn't see search in Tad yet), with selectize for categorical variable filters(which is a searchable dropdown list)

Another good to have feature is to show column index. We often need to manipulate the columns in code, a column index is helpful.

Based on column index, you can also select a subset of columns faster with numeric input -- over 100 columns are normal, using checkbox to select is too cumbersome.

Good job, curious about the performance when doing multiple AND, OR clauses on 1m+ rows? You'd need compound indices or?

Absolutely all the analysis (including filtering) works by generating SQL queries and that SQLite evaluates. Tad doesn't generate any table indices, and I'd be reticent about adding that. However you should be able to Tad as a sqlite viewer from command line (sqlite://foo/bar.sqlite/tablename) and it should make use of indices on your existing SQLite table if you have them.

Good summary, thanks for the tldr;

+1 for the comment about Electronic/Javascript, also don't want to start a flame war heh.

exploratory.io looks really neat, it's a shame about the pricing and I'm not quite sure if it's a native app once you purchase or if it's an in-browser online-only app, because if it's online only 1) I'd be worried about uploading data to a 3rd party and 2) The internet in Australia is _terrible_ so uploading / downloading larger data sets might be painful.

From their FAQ (https://exploratory.io/faq)

- I have very sensitive data. Is my data safe?

Any data you import into Exploratory Desktop always stays on your PC and never leave your PC unless you explicitly publish (share) it to Exploratory Cloud (exploratory.io). If you decided to publish the data to Exploratory Cloud for sharing or scheduling, you can share it in a private way so that so that only you and others you have invited can view it. The data is also stored in encrypted. Please take a look at our Privacy Policy for more details.

- Where exactly my data is stored after importing?

All the data you import into Exploratory Desktop is saved as a binary form (R's Rdata format) inside your repository, which is located under '/.exploratory' on your PC.

Genuinely curious, what can be used instead of Electron for cross-platform desktop app?

Tk - It can look a bit dated, but it's easy to use, comes bundled with Python or Tcl, and bindings in most popular languages. Can run on Windows, Linux, macOS.

Qt - The IDE is great. It doesn't look native. Either dynamically linked, or commercially licensed. Runs on Windows, Mac OS X, Linux, Android, iOS, and a range of embedded hardware.

wxWidgets - Native backends, so it always looks like it belongs. Bindings in a ton of languages. Can be both simple or complex, depending on what you need it to do. Runs on Windows, Mac OS X, Linux, and in-progress for Android and iOS.

JavaFX - Java's replacement for Spring. Runs anywhere Java does. Fairly flexible, and easy to use.

Kivy - A Python framework. Mainly aimed at touch-compatibility. Runs on Windows, Linux, macOS, iOS, and Android.

LCL - A Lazarus framework. (Think Pascal). Really easy to use, with great drag'n'drop and the like in the IDE. Runs on Windows, macOS and Linux.

nuklear - Fairly easy to use, with bindings in a lot of languages. Sometimes requires a bit more work getting it to run on some platforms.

This is not all, but the ones I find easy to use (as easy or easier than Electron), and easy to set up and deploy.

Thanks for such a detailed answer. Can you elaborate why those would be better than Electron?

It depends on what you want from the framework.

Electron doesn't look native, which a lot of consumers don't like. Others don't care. So, depending on your audience, it can be an extra hurdle that some like wxWidgets don't have.

Electron is difficult to get performant. You need to really think and test your perf. Qt, Tk, and a few of the others are much faster, much easier. And if you put in the same effort you need to put in for a fast Electron program, you can end up with blisteringly fast speeds.

Electron bundles are large to download. Some places this doesn't matter, others it does. (Think Australia, Africa and the like where tiny download caps exist). IIRC Qt, the biggest dependency, is about half the size of Electron. Tk and wxWidget are small, nuklear is just a header. It's tiny.

Electron is not good with touchscreens (or "harder to get right"), but more and more people have them. Kivy is great for that usecase.

Electron is "just code". Qt Creator and Lazarus' IDE are phenomenal for putting GUIs together. You'll be surprised how little code you need.

Electron is primarily JavaScript. Some people like that, others don't. If you want an easy language, you can use Python, or Nim. Want more control C or C++. Value both time and control? Go with Java.

Electron has its place.

If you have a really tight time-to-market window, or this is just a tiny personal project, and JavaScript is your "goto" language, then awesome. Use it.

But, the cross-platform GUI world is big. You have a dozen or so mature, stable, proven frameworks.

So if your project takes off, or you have the time to do things right, evaluate if any of these libraries, including Electron, fit your needs.

How's Java giving user more control than Nim? The latter allows for hardware access like C/C++

I said C/++ was more control, Java was a middleground between the two. Nim is easier, but its memory usage and performance are harder to be sure about, because it hasn't been around as long.

With Java you can get it through JNI.

But if you're going that route, you'll be making your life a lot harder in the cross-platform department.

Other people have mentioned Qt, which is great, but I'd like to point out there it also support QML which is probably better your you.

* It is backed by Qt and rendered by OpenGL, so it performs very well.

* It has very easy syntax, especially automatic variable binding, so there is very little boilerplate for event handingl.

* It supports JavaScript. You can code the logic in either JS or C++, depending on whether you need the performance or not, and it can be easily split between the two.

* They actually support two sets of pre-made controls, one that looks like desktop UI (http://doc.qt.io/qt-5/qtquickcontrols-index.html) and one that looks like Android or iOS (https://doc.qt.io/qt-5/qtquickcontrols2-index.html). Or you can design your own UI from rectangles and other shape. Or combine all three.

Qt is an option. Supported on Windows, Mac, and Linux.

Tk is crossplatform, free and open source: https://en.wikipedia.org/wiki/Tk_(software)

It might look a bit dated on some platforms, but it will be lean and fast, unlike Electron.

I can understand the appeal of writing a non-internet app that uses html/css/javascript. But if you want to do that, why not just run a server locally and use a normal web page? Seems better than running a custom chrome instance.

A bit old-school, but many use JavaFX, especially for enterprise apps, even if people on HN may bash on Java for whatever reason.

"Many" is untrue. I'm a huge Java fan but FX never picked: https://www.codenameone.com/blog/should-oracle-spring-clean-...

Enterprises still use Swing and some have touched FX as Swing is in maintenance mode and they don't have a choice.

Why is it sad that it's an Electron app? Can you articulate a reason why that's an automatic negative? Would you rather software, such as this, not exist than be Electron-based?

Not that we'd want to start a flame war or anything.

exploratory.io looking promising. Thanks for the links and since it's an electron app then there's hope for a web version :)