Does it have to be a Google voice number?

no

The issues with those is that google will prompt you if you don't use them. I lost my within a week because I didn't see the email.

I keep my numbers at didww.com now where I pay a small fee but they won't just delete my number.

Google and many other services reject voip numbers, though

I have numbers there that have been ported from local providers. There is no way for Google or anyone to determine if a number is a voip number. All google knows is which numbers they provided and this list may also be available to others.

Or there is a simple database lookup way to determine that the number doesn't belong to the local provider:

"Superfast Response Times As we access the entire mobile number portability (MNP) database in every country where number porting is in operation, Velocity can truly live up to its name, providing answers at lightening speed (typically less than 5ms)"

https://tmtid.com/velocity/

That's not accurate - when a telephone company registers a block of numbers (at least in the US), you specify if it's landline, pager, VoIP, cellular, etc.