Show HN: st – simple statistics from the command line

Re:Show HN: st – simple statistics from the command line by imurray on Wednesday September 04, 2013 @08:13AM (#6326923)

For casual purposes st may be convenient, but it doesn't have state of the art numerical stability:

    my $variance = $count > 1 ? ($sum_square - ($sum**2/$count)) / ($count-1) : undef;

Taking the difference between two similar numbers loses precision, and in extreme cases squaring the raw numbers could cause overflow. For comparison, see the recently posted: http://www.python.org/dev/peps/pep-0450/ and https://en.wikipedia.org/wiki/Algorithms_for_calculating_var...

[ Parent | on HN ]

Re:Show HN: st – simple statistics from the command line by nferraz on Wednesday September 04, 2013 @08:37AM (#6327055)

Thanks! I changed the algorithm to online variance, hope it is more stable:

https://github.com/nferraz/st/commit/d0fb1bf814fc5940c5aae39...

[ Parent | on HN ]

Re:Show HN: st – simple statistics from the command line by Doches on Wednesday September 04, 2013 @08:38AM (#6327067)

If you need numerical stability, I'd use Gary Perlmann's [|stat](http://oldwww.acm.org/perlman/stat/history.html). It's older and somewhat harder to get a copy of, but it's as reliably correct as a piece of software can be...

[ Parent | on HN ]

Re:Show HN: st – simple statistics from the command line by imurray on Wednesday September 04, 2013 @10:14AM (#6327680)

I've just had a look at the |STAT source, and it computes the variance with

        double  M       = Sum/N;                /* mean */
        double  var     = (s2 - M*Sum)/(N-1);   /* variance */

where s2 is the sum of squares. In most reasonable situations, this approach will work fine. It just doesn't take as wide a variety of inputs as is easily possible to achieve with normal floating point doubles. In fairness, the |STAT terms and conditions state:

|STAT PROGRAMS HAVE NOT BEEN VALIDATED FOR LARGE DATASETS, HIGHLY VARIABLE DATA, NOR VERY LARGE NUMBERS.

[ Parent | on HN ]