The way I like to do it is to keep multiple backlogs, broken down roughly by feature group (ie. groups of things that might eventually become a milestone or story of their own). Then when I decide to prioritize a given story, I can go to the relevant backlog and re-triage only the bugs in there, which should hopefully be a relatively manageable number.

Another benefit of sub-categorizing this way is that it makes it easier to resolve bugs as duplicates. When a new bug is filed, it's hard to see that it's a duplicate when you're comparing it against 10000 other bugs, but it's easier when you're comparing it against only 100 other bugs in the same category.

I doubt you'll ever be able to get it down to "at most twice." But it needs to be much easier than ever resotring to "looking through the whole list."