Are you using PSFzf or something else? https://github.com/kelleyma49/PSFzf

I used to use PSFzf indeed: it's pretty good and got me started with fzf on PS, but the history reading nor file scanning are particularly fast so currently I'm using some C# code to do those and pipe it into fzf directly. Don't recall exactly but the most basic C# code I could come up with which does 'read history file in reverse and skip duplicates' was like 3 times faster than what PSFzf uses.