Could you share what kinds of banks will do manual underwriting? I am another person that refuses to get credit cards, but I did run into issues getting a mortgage.

All the banks that I spoke with would concede in person that I am an extremely low risk, but they all intended to sell my mortgage on the secondary market (but still do the servicing themselves). So, they wouldn't / couldn't give a mortgage to someone who doesn't fit the "normal" boxes.

Finding a bank that was willing to not sell the mortgage took a bit.