F3 is common for "Find Next", F5 for "Refresh". I don't want to use two hands for these.
IntelliJ has defined shortcuts for F10, Shift+F10, Ctrl+F10 and Ctrl+Shift+F10. Adding another modifier to all these isn't great.
You're thinking small here. With QMK keyboard firmware, a single key can send shift-f10 or similar. I have a top layer key that acts as ctrl-super and also as a sticky key (I don't have to hold it) because I configured a bunch of key chords in Sway that start with those keys. Mainly focusing specific programs by appid or class. So, I'll hit that key and then b for browser, v for video (mpv), etc. You can even have keys that directly type out parentheses, no shift needed. The default layout for my keyboard had that for both parens and brackets. I found more useful things to put on those keys, though.
I use a Pinky4 which takes some stuff off the outer edges compared to a 60%, but has more on the inner edges, and so I stuck that key somewhere along the upper right edge of the left half. If I were using qwerty, it'd be near t and g keys.
In addition to being able to run QMK on a keyboard, you could also get a macropad, or just a second keyboard (maybe something small like a planck, could go above your main one), and have all sorts of specific binds like this. I would recommend layers instead for anything not used constantly, but a case can be made for both options. I have mouse keys in a separate layer, letting me hold a modifier with my thumb and then use something similar to an arrow key cluster to move my mouse. I even have all 4 scroll directions mapped, which lets me scroll horizontally, unlike my actual mouse.
Software on the computer really affects use-case I think. In qutebrowser (or firefox + tridactyl) I just hit r to refresh a page. I never miss easy access to f5. I can also copy a page's url with yy instead of doing a dance of ctrl-l and then ctrl-c. I can still hit f5 if I have to. Like if using an unusual program or another computer with my keyboard.
It sounds like due to your software choices, you use the function row a lot more than me. There are some larger split keyboards, but it's worth considering changing some of your behavior/software also.
But whatever, you do you :)
I'm also not sure how to use more layers without dedicating more keys to switch between layers?
For more common layer switches, I extensively use dual-use keys. For example, if I tap "F" then I get the letter "f" but if I hold it, it switches to a vim-like layer. HJKL becomes arrow keys, Y becomes Ctrl+C, etc. The point is that you don't have to dedicate more keys to do layer switching. You can have the same key do different things depending on how it's used.
For me, memorizing has been easy due to incremental change. I didn't come up with my new layout on day one then stick to it forever. I'd find a frequent pain point, tweak the layout, and memorize the tweak. It's easy to remember 2 or 3 key changes at a time.
For shortcuts - I just conciously accepted it takes time to learn. I'd say it was about a month before I stopped reaching for the old shortcuts and started to use the layered shortcuts.
It can also help force you into it by not having the old keys mapped - obviously this comes with judgement on a per key basis. I'm not asking you to unmap "A" - but unmapping/not-mapping the F-keys in their usual place coerces you into using your new mapping and you'll just get the muscle memory eventually.
1) I hit the function keys far less often than regular character keys and
2) I rarely press the function keys while typing text since, as I mentioned, I use them to switch workspaces. In particular, I often use them while my one hand is on my mouse (think drag & drop) and, as you say, same-hand chording is out of question.
I have looked at ergonomic keyboards before, and have the same reservations about the number of keys as you. I use the numpad a lot for data entry, and my previous favourite keyboard was a Logitech one that had an additional row above the numpad that contained keys for parentheses (without using shift), an equals sign and a second backspace key. I did seriously consider the Maltron L90 full-width keyboard, but I couldn't source it locally and the price was too steep for me to buy without testing it first.
Having something like that keyboard coupled with a programmable (QMK) controller would be an option I'd seriously consider though, even without a separate numpad.
In your specific case I'd probably map the layer modifier key on one hand, then map the F-keys like a numpad on the other so there is no same-hand chording. Though I strongly suspect you aren't using all 12 keys and instead are using say the first 5 or 6 only.