Besides provider sometimes have strange port rules it's not uncommon for them to forcefully change your IP from time to time, even if there is an open connection. It tends to happen at night and it tends to be a forceful disconnect from your router to the outside world for <5min.

At least I ran into this frequently (multiple times a week, I really need to fix my sleep cycle).