This is almost exactly like Microsoft's response to the ribbon criticism when it first came out. They were so proud of themselves that a handful of those buttons were used 30% of the time or whatever, but ignored the fact that another handful was used <1% of the time yet were still visible by default on the default tab.

Which is to say your justification is very much "75% of our interview process makes sense, it's only this 25% that you almost never needed any any previous job, you don't need at this one, and if you're self-taught you probably never learned in the first place, but we're going to ask you anyway because it makes the interviewer feel smart."

'Its just the algo stuff you usually have to review before hand vs all the others'

There's the problem. If it's something you have to review, it's not something you use that often. Unless the job is going to have you using it often, why is everyone's time being squandered on it?

I guess the problem is you need it infrequently but when you do need it it is really helpful to know.

And easy enough to look up.

But what are you looking up? You have to know what class of problem you have to be able to look it up.

And if the interviewer will be satisfied with "Oh, this looks like an X kind of problem. I'd just google that and go from there", fair enough.

A decent interviewer is probably willing to be cooperative and fill in some details for you. I guess some places are all about trying to play "gotcha" but the hell with them.