News for hackers. Stuff that matters.
Re:The Median of a Trillion Numbers by jfoutz on Monday August 24, 2009 @01:34AM (#781562)
Since 10^12 is way way bigger than a 2^32, it's either the average of the numbers, or a strange degenerate case. (10^11 1's followed by 10^11 3s).

just calculate the average, and remember the item closest to the current average.

Re:The Median of a Trillion Numbers by roundsquare on Monday August 24, 2009 @04:43AM (#781765)
Eh? Thats not true.

The only time this would work is if the distribution of the numbers was symmetric (or close to symmetric).

If you have

1 --> 599,999,999,999 times

4 --> 1 time

10 --> 400,000,000,000 times

Your average will be about 4.6 (a little higher) and your algorithm will return 4 as the answer when it should be 1.

Re:The Median of a Trillion Numbers by jfoutz on Monday August 24, 2009 @10:33AM (#782198)
Yeah, I commented right before bed. I thought of a bunch of problems with the average while trying to get to sleep.

I was thinking since the size of an outlier is constrained by fitting in 4 bytes, there must be a way to show the median is in some band of numbers around the average. I didn't think of a particularly useful algorithm though.

Thanks.

Re:The Median of a Trillion Numbers by eru on Monday August 24, 2009 @05:09AM (#781791)
Calculating the average is in O(n). Calculating the median is in O(n).
Re:The Median of a Trillion Numbers by roundsquare on Monday August 24, 2009 @05:23AM (#781808)
Actually, even be worse here. E.g. if all the numbers are positive you will get an overflow pretty quick, so you'll need a better datatype. So if you are dealing with b-bit numbers you'd need O(n * log(bn)) to calculate the average.

(Usually this doesn't matter, but since n >> b it does).

Edit: Notwithstanding my earlier comment that average doesn't work, merely commenting on the run time of average.

Re:The Median of a Trillion Numbers by bravura on Monday August 24, 2009 @02:33PM (#782885)
Or, you could compute the average as follows:

    m <-- m - (1/t) (m - x_t)
This will compute the exact historical average. I like to use 2/t instead of 1/t to get a moving average over a non-stationary distribution, which favors the more recent events.
Re:The Median of a Trillion Numbers by eru on Monday August 24, 2009 @04:17PM (#783209)
Please keep numerical errors in mind. They accumulate.