News for hackers. Stuff that matters.
Re:The "Soft Maximum" function by imurray on Wednesday January 20, 2010 @05:52PM (#1066026)
The soft maximum often comes up when dealing with probabilistic models and in neural networks. I'm surprised that a blog that has been highlighting numerical issues hasn't pointed out how the soft maximum should be computed.

If x is vector of values then the naive code is:

  log(sum(exp(x)))
where exp operates elementwise. However, if even a single item of x is large (1000 say) this will return Inf. If x are log probabilities, where you want the log of the sum of the probabilities (common), the elements of x might all be less than -1000 (also common) and then the function will return -Inf.

Many people have a function called logsumexp() kicking about to compute the softmax robustly. It will do something like:

    y = max(x);
    return y + log(sum(exp(x-y)));
One example, slightly more elaborate, Matlab implementation is in: http://research.microsoft.com/en-us/um/people/minka/software...
Re:The "Soft Maximum" function by dalke on Wednesday January 20, 2010 @08:53PM (#1066394)
See the followup posting