Honestly I had the exact same problem. The solution for me was still pretty simple though, I just changed how often I would examine/categorize the information. I had much better recall of what transactions I have approved over the past couple of days rather than over the past month and it takes much less time overall.

I use YNAB which is basically the manual version of Mint. It imports exactly nothing for you. You must log into your bank account/credit card and log each transaction and categorize it yourself. This might take me 30 minutes per month. If you wait until the end of the month it will take longer because you'll forget what you did at the start of the month. You need to keep up with it like anything else.

You can import records in bulk from online banking exports, though, which is what I do to get my transactions in the system, and if it sees a payee it's seen before then it'll automatically suggest a category. You do still have to approve every transaction, which I think is useful.

Fellow YNAB user... how often do you reconcile? I try to do it at the end of the month and it can get a little overwhelming.