That's interesting. I didn't read the paper closely, but skipping to the pictures, it looks like ReLU, but smoothed out so the derivative is continuous. Intuitively, that seems useful.