You're probably drying out your skin with hot water and soap. Not cleaning it well enough can do that too. I used to get that problem until I realized two important things.

1. Regular hot water on your hair and beard dries out your skin terribly. Especially during winter. You need to use luke warm showers and probably avoid washing your head every time you shower. Use cold water on your face afterwards and gently dry it off. Which leads into the other point.

2. Stop touching your face so much. Routinely or obsessively touching your face or beard irritates it a lot. That can inflame the skin, causing it to over produce skin, oil, etc.

Also using too much beard oil as well can cause irritation. Your skin doesn't need too much oil. Only what it produces naturally (when you don't wash it away with soap or hot water).