News for hackers. Stuff that matters.
Re:Wizard for Mac - a tool for statistics and data analysis by EvanMiller on Wednesday September 26, 2012 @01:47AM (#4574325)
Sorry for the poor audio quality in the video. Some random thoughts that may be of interest:

1. I've been in grad school for economics the last four years. The program basically embodies all the things I've learned about econometrics, i.e. applied statistics. Wizard does about 1% of the things that R does, but it does them very, very well.

2. Wizard is fast. All the tight loops are written in C. Wizard will use all the cores on your Mac as well as the CPU's vector capabilities. This was achieved with a combination of Grand Central Dispatch and OpenCL. I highly recommend both technologies. For my research I am using Wizard on data sets with ~3 million rows, and it works like a champ.

3. The main weaknesses of the program is that it does not deal with time-series data particularly well, and you can't manually enter data. These shortcomings will be addressed in future releases.

4. The program is closed-source. I might make some of the libraries open-source. Or I might not. I haven't decided.

5. The non-Pro version will be released sometime next week.

6. I have absolutely no regrets about not writing Wizard using JavaScript and HTML5.

7. [Edit] You can read a review of the pre-release version of the program here: http://www.macstats.org/reviews/wizard.html

Re:Wizard for Mac - a tool for statistics and data analysis by calinet6 on Wednesday September 26, 2012 @08:43AM (#4575265)
This is very cool. The tutorial (the Wizard Wizard, if you will) was very well done and I feel like the program makes a lot of basic stats analysis very easy and visual. Nice work.

In your opinion how relevant would this be to survey analysis for psych and social science research? Does it cover most of the analysis required or are there shortcomings that, say, SPSS would better fill? I'm not a stats guy, just a programmer, but it would be great to work it into our recommendations for tools for researchers on Mac (shameless plug - http://www.socialsci.com).

Re:Wizard for Mac - a tool for statistics and data analysis by arrrg on Wednesday September 26, 2012 @04:33PM (#4577871)
Coming from a social science perspective, this tool seems to cover a lot of bases. I haven’t used it for actual research (obviously), so it’s easy for me to overlook shortcomings, but a lot of the important stuff seems to be there. What I didn’t really see (but maybe I missed it) were good tools to edit and clean up data (which often is a lot of tedious work and at least as important as actually calculating the statistical tests) as well as factor and cluster analysis (especially factor analysis is something that is used quite frequently when looking at survey data) – but I think you can do most of the stuff social scientists do with this tool.

I do not think it is a replacement for other tools (SPSS, R) but I do think it’s excellent at what it does. I’m quite impressed, actually. Firing up SPSS is just no fun (also crazy expensive), so I will definitely look into getting this. (R is fun, but for some reason I’m very slow with it.) It’s definitely very accessible.

Re:Wizard for Mac - a tool for statistics and data analysis by pkaler on Wednesday September 26, 2012 @02:28AM (#4574421)
I took a Stats course in university. Basically, this one: http://www.stat.sfu.ca/content/dam/sfu/stat/courses/outlines...

Can you recommend a way for myself and others in the same position to brush on their stats?

Re:Wizard for Mac - a tool for statistics and data analysis by arrrg on Wednesday September 26, 2012 @05:07PM (#4578048)
I have always wondered whether there are grad students or scientists frustrated enough by the statistical tools they have and knowledgeable enough about programming to make something like this. (Only the first part applies to myself.) Now I know.

Great job! This looks really great. I would love to see factor analysis added. That’s one bread and butter thing I often need which I couldn’t find. Maybe also post-hoc tests if ANOVA is significant, but maybe that’s outside of the scope of this app. (Oh, and another thing I’m missing in the summary view is the number of cases. Maybe I’m just blind, but when I, for example show a t-test there doesn’t seem to be a way to display how many cases there are included in each category.)

Re:Wizard for Mac - a tool for statistics and data analysis by notlisted on Wednesday September 26, 2012 @11:12AM (#4576027)
Love it.

Questions:

- how many rows of data can it handle (limited to excel 65k?)

- can I manually (and permanently) correct any detected data anomalies?

- can I manually (and permanently) add columns to the data?

- can I export numerical values of graphs? (formats?)

- can I export the graphs? (formats?)

- can I export reports (data sub set, models, e.g. graphics for strong republicans in all regions)

- can I create values based on functions of data elements (all fields containing '@gmail')

Thanks.

PS agree with the > $50 issue, I'd buy something for $50 just like that, but if it's over, I start thinking about it.

Re:Wizard for Mac - a tool for statistics and data analysis by rgbrgb on Wednesday September 26, 2012 @02:30PM (#4577192)
>>> For my research I am using Wizard on data sets with ~3 million rows, and it works like a champ.
Re:Wizard for Mac - a tool for statistics and data analysis by notlisted on Wednesday September 26, 2012 @03:16PM (#4577452)
Hmmmm, suddenly the $79 sounds a lot more reasonable... (providing data is easily exported).

I have no need fo SPSS etc (beyond my pay grade and comprehension) but I've been frustrated with the limitations of Excel (love pivot tables) and I have no $$$ for http://www.tableausoftware.com/

Note: my main interest is actually CLEANSING DATABASES (finding outliers/anomalies, and edit the data set, hence my questions re: changing data)

Evan has created another lovely app by the way (http://magicmaps.evanmiller.org/) which makes me more confident that it will all work properly.

Re:Wizard for Mac - a tool for statistics and data analysis by neilxdsouza on Wednesday September 26, 2012 @02:41AM (#4574446)
For point 2. Do you compile to C or is it running as an interpreter? I too have written something similar, but I compile to C++ and yes the speed is amazing, and memory consumption minimal, because, you only keep 1 record in memory (if using more cores then this will not be true).

If you are interested in a solution for entering data, let me know. My solution is open source.

Re:Wizard for Mac - a tool for statistics and data analysis by EvanMiller on Wednesday September 26, 2012 @10:05AM (#4575655)
The program uses the Objective-C run-time, but the core routines are written in C.
Re:Wizard for Mac - a tool for statistics and data analysis by johncoltrane on Wednesday September 26, 2012 @03:08AM (#4574506)
Very impressive video. The trial download is broken ATM.
Re:Wizard for Mac - a tool for statistics and data analysis by jonpaul on Wednesday September 26, 2012 @01:23AM (#4574261)
For the love of God... please put your video on Dropbox or S3. I don't think your site can handle it. At least I'm having issues with it.
Re:Wizard for Mac - a tool for statistics and data analysis by dfc on Wednesday September 26, 2012 @01:29AM (#4574277)
There are nicer ways to make recommendations/requests than "for the love of god...do X."
Re:Wizard for Mac - a tool for statistics and data analysis by jonpaul on Wednesday September 26, 2012 @01:44AM (#4574317)
You are correct, I just got a bit excited about the product and really wanted to watch the video. I'm sorry if I came off a bit too aggressive.
Re:Wizard for Mac - a tool for statistics and data analysis by EvanMiller on Wednesday September 26, 2012 @01:51AM (#4574335)
Sorry about that. The HN traffic was unexpected. I will upload to Dropbox ASAP.
Re:Wizard for Mac - a tool for statistics and data analysis by EvanMiller on Wednesday September 26, 2012 @08:33AM (#4575239)
For future reference, uploading a video to Dropbox when you are on the HN front page is a bad idea. My account was suspended almost immediately. Switching to YouTube...
Re:Wizard for Mac - a tool for statistics and data analysis by mva on Wednesday September 26, 2012 @02:49AM (#4574461)
Or upload it to youtube and split it up, so it's scrollable (wasn't able to scroll in the 30+ minute video now).
Re:Wizard for Mac - a tool for statistics and data analysis by macmac on Wednesday September 26, 2012 @03:32AM (#4574572)
502 Bad Gateway
Re:Wizard for Mac - a tool for statistics and data analysis by dmix on Wednesday September 26, 2012 @01:13AM (#4574247)
Seeing pricing >$50 for software products always makes me happy as a bootstrapper.

It shows you trust the value your customers will put in your product.

Re:Wizard for Mac - a tool for statistics and data analysis by fwr on Wednesday September 26, 2012 @01:36AM (#4574301)
$80 for something that is "Best for casual (...) users" is not really the price I'd pay for a program I'd use from time to time to casually crunch numbers in.

And not like there's a free alternative out there that just blows Wizard out of the water, right?

Re:Wizard for Mac - a tool for statistics and data analysis by EvanMiller on Wednesday September 26, 2012 @10:03AM (#4575642)
Thanks. I have updated the copy to better reflect the distinction between the two versions.
Re:Wizard for Mac - a tool for statistics and data analysis by Flow on Wednesday September 26, 2012 @06:45AM (#4574969)
There is? A link would make me happy.
Re:Wizard for Mac - a tool for statistics and data analysis by kome on Wednesday September 26, 2012 @08:39AM (#4575254)
Re:Wizard for Mac - a tool for statistics and data analysis by Flow on Thursday September 27, 2012 @03:46AM (#4579780)
Looks good. Thanks! I was aware of https://gephi.org since before, always nice to have more tools in the toolbox. :)
Re:Wizard for Mac - a tool for statistics and data analysis by jonpaul on Wednesday September 26, 2012 @01:47AM (#4574326)
I just watched the video, nice work! I regularly deal with files that have 10,000,000+ records. How well does Wizard scale with large data sets?
Re:Wizard for Mac - a tool for statistics and data analysis by EvanMiller on Wednesday September 26, 2012 @10:07AM (#4575670)
Thanks. I've been testing on 3-4M records on a quad-core laptop, and it works great. I would guess that 10M records will be doable on an 8-12 core Mac Pro, but I don't have access to one to test this particular hypothesis.
Re:Wizard for Mac - a tool for statistics and data analysis by sirwanqutbi on Wednesday September 26, 2012 @03:31AM (#4574568)
502 Bad Gateway. Links down.
Re:Wizard for Mac - a tool for statistics and data analysis by EvanMiller on Wednesday September 26, 2012 @08:34AM (#4575240)
Sorry. It's back up.
Re:Wizard for Mac - a tool for statistics and data analysis by denzil_correa on Wednesday September 26, 2012 @04:19AM (#4574674)
Same for me.
Re:Wizard for Mac - a tool for statistics and data analysis by Gojja on Wednesday September 26, 2012 @08:24AM (#4575211)
Me too
Re:Wizard for Mac - a tool for statistics and data analysis by filip01 on Wednesday September 26, 2012 @10:20AM (#4575755)
Just tried it out. Looks really good. Would be a perfect plug-in for Sequel Pro for example. What other products are there in the same space (easy-to-use statistics tools for Mac)?
Re:Wizard for Mac - a tool for statistics and data analysis by justindocanto on Wednesday September 26, 2012 @06:23AM (#4574929)
Getting a 502 error when i visit
Re:Wizard for Mac - a tool for statistics and data analysis by zmjones on Wednesday September 26, 2012 @05:36PM (#4578184)
Why pie charts dude?
Re:Wizard for Mac - a tool for statistics and data analysis by boggzPit on Wednesday September 26, 2012 @01:55AM (#4574347)
video!!! ;)