Agree that if reboot fixes it, it's probably not DDoS. In that case, start by limiting simultaneous inbound connection counts to something reasonable and add alerting so you can inspect the system when you hit the limit.

Honestly, I'd start with a limit of 100, and go from there.

If connections just sit idle, put a timeout. If they send a lot of garbage, close and ban that IP for a day, etc.

I've seen a lot of volumetric DDoS against servers I ran, but they never targeted our mail server. (Yeah, we just had one, but it was just doing autoreplies and forwarding)