I have similar setup (infinite history, remember directory, shell session, ...), and then ctrl-r runs fzf that find the command first in this session, then in the same directory, then in the rest. I already hinted about that setup here:

https://news.ycombinator.com/item?id=22504950

Note, that just logging your directory, time and command in text file is great help, since if you don't remember something, you can just open the file and find what you need.