News for hackers. Stuff that matters.
Re:Knowledge Is a Region in Weight Space for Fine-Tuned Lang... by rsfern on Thursday February 23, 2023 @07:14AM (#34909453)
For what it’s worth, I usually default to swish activations, which seem to be popular in my corner of graph neural nets (materials and chemistry). Performance is about the same as ReLU, and I like swish because it doesn’t have a hard discontinuity.

There are no comments on this comment yet.