A VPN has upsides and downsides. It obviously protects your server a lot better against directed attacks, but when you lose your laptop or when your computer gets ransomware'd, you can't get access to the server anymore.
Furthermore, code execution vulnerabilities have been found against VPN servers because of their immense complexity and OpenVPN can consume quite a lot of resources for a daemon doing nothing. WireGuard has changed the VPN landscape with its simplicity, but if you fear your server may not be updated all too often (because it's partially managed by a customer, because your colleagues might not care to do so after you leave), leaving a simple solution behind can have its upsides.
I'm not advocating that everyone should enable port knocking on their servers to make them secure or anything, but the "port knocking is always bad" crowd is often very loud despite the fact that there are small little ways port knocking can improve security with very little effort or increased attack surface.
I think the main concern with port knocking is that it's observable. You're effectively sending your password in clear, so if someone can intercept or overhear your traffic then your secret is lost. Cryptographic authentication schemes like SSH itself or VPNs do not have this problem.