People prefer algorithms with a clear mathematical basis because they're easier to analyze, so the flaws surface easier and it's clearer what breakthroughs would break them.
Cryptographers have been looking for algorithms that are NP-hard for example, because having a "breakthrough" in them having to require P=NP is a large hurdle. But it's not the end of the story, because it turns out that a problem being NP-hard doesn't mean it isn't easy to crack (worst case vs actual case).
Your comment reads a bit as saying "we shouldn't use maths to compute things". But maths is what computers do...
I'm not even sure what you would imagine the alternative would be. An algorithm that isn't amendable to mathematical analysis?
Even for symmetrical ciphers, which look far less like pure math, analysis proceeds very much along mathematical lines (differential/linear cryptanalysis), or in some cases, via algebra as well: https://en.wikipedia.org/wiki/XSL_attack
To be in NP-complete a problem must both be in NP and every problem in NP must have a polynomial time reduction to it. This means that NP-complete is fully contained in NP and in fact smaller if P!=NP.
For example P is contained in NP and if P!=NP then no problems in P are in NP-complete.
For asymmetric cryptography there is no alternative not based on math. Even snake oil cryptography mostly deals in symmetric crypto like "unbreakable" ciphers and hashes, because you can't make a working asymmetric encryption scheme by mashing together whatever operations your schizophrenia tells you to.
CONVERSION:
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
STEP 1. Write out the message (HAIL ERIS) and put all the vowels at the end (HLRSAIEI)
STEP 2. Reverse order (IEIASRLH)
STEP 3. Convert to numbers (9-5-9-1-19-18-12-8)
STEP 4. Put into numerical order (1-5-8-9-9-12-18-19)
STEP 5. Convert back to letters (AEHIILRS)
They assert 100% unbreakability, which isn't true, but on a sufficiently long message one's anagram checker might give out, and then there is the task of ordering the words so generated...Mind you, decryption by the intended recipient is equally complex.
Personally, I don't have particular worries about the hardness assumptions of asymmetric crypto, and I think of them a bit like I think of bitcoin (hear me out). Yes, it is certain that eventually someone will solve the discrete log problem for any given algebraic structure (either by rendering all crypto that relies on it broken, or (less likely) proving it fundamentally secure), but for now, we know that this is hard (since it has been open for a while), and we're also incentivizing people to make mathematical discoveries.
I'd also claim that the "crypto community" (at least the academic side of it) and the "technology community" are not the same, and (at least to me) often feel opposed. Cryptologists write papers filled to the brim with dense and precise mathematical assumptions and reductions; technologists skim the papers, ignore the assumptions, and implement half-assed, unaudited versions of the systems in question and claim them secure (pardon my cynicism).
As to what the community thinks about mathematical public key crypto, they hail it as the greatest innovation since sliced bread and the herald of modern cryptography. Prior to modernity, cryptography was very ad-hoc and depended on what the author's intuitions; modernity introduced precise definitions of what it meant for a system to be secure and raised the bar. It also relies heavily on the concept of a hardness reduction, i.e. a proof that breaking a cryptogrpahic primitive is at least as hard as solving a yet-unsolved math problem.
Specifically about algebraic problems, I have a (low confidence) intuition that they are unavoidable in public-key crypto precisely because of the need for an algebraic structure relating the public and private keys. With this in mind, I'd rather have algorithms which rely on known hard to solve problems (demonstrated hard by having years of mathematical effort poured into them with minimal result) to those which rely on problems no one has ever bothered to look at.
A final question: you are unhappy with public key crypto that relies on algebra; would you be happier if it relied on some other branch of mathematics? Analysis? Topology (okay, so that's still algebra)? Complexity theory (a secure cryptosystem that relied only on P!=NP would be a holy grail for several reasons, but I don't know of any attempts to find one)? Would you feel safe using a cryptosystem that was secure if and only if the Riemann Hypothesis were true? If the RH were false? The Collatz Conjecture?
I'll also go out on a limb here and also say that I think the technology community has a bias towards thinking something like "math == hard" is true, so gives added weight towards using these same protocols. I know many people here have deep knowledge of both cryptography and software development, so I'd be very interested to hear other people's thoughts on these issues. Can anyone speak about options to math-based public key algorithms, or ways to inject some skepticism into the tech community about these algorithms, so perhaps alternatives can start being implemented? A public key algorithm which doesn't lend itself easily to algebraic analysis would feel much safer to me.