The scrambled on-screen keyboard / n-th character request only poorly "protects" against cases where the attacker has full root access to the customer's machine (how would it protect against phishing?), and in those cases the attacker is likely already capable of making all kinds of payments, even without access to the online banking account. If security is done properly, an attacker can't make direct use of a banking account anyways: creating a new transfer recipient should require phone confirmation by default.
All you'd have to do is iterate every potential answer into a bloom filter and store that. The math gets a little hairy around the 50 character mark as you'd have 117600 operations to do to construct the bloom filter, and it gets worse if you expect more characters.
Here's how you would construct it.
For every 3 choose n of the password, insert a value of those 3 characters ("abc") + delimit (":") + the positions (1,2,4), You can't just insert the characters because the index of characters is part of the answer.
all you'd need to do is store... a ~1MB bloom filter per client. Huh, that number was bigger than I thought it would be.
Well nevermind. Fun thought though.
If an attacker is running code on your system, you are already lost.
Re brute forcing, some like firstdirect make you type a word as well and they probably block you after 3 fails so it would be quite hard to brute force. Tddirectinvesting only ask a 7 digit account number and 3 digits from the password so you could probably brute force coms of those with a botnet though they only let you transfer money to a pre nominated bank account so it would still be hard to nick people's money.
Actually it's very simple to achieve.
First, those digits will not be randomly placed among all the things you've typed, but they'd follow some specific patterns (the most obvious one being you typing all of part --due to autocomplete-- of the bank's url).
(Of course if you can run a keylogger you can also check what website is loaded on the browser and log that information alongside the keys too, but you don't even need to go that far).
So, we established that the attacker checking the keylogger logs can trivially tell - "now they're typing their banking password".
If they also knew the correct placement that would be handy, but they can do without it too. Just knowing those N characters are from your password (in any order) really improves the possibilities they need to search.
Even if it takes a year, either they are very dedicated to you as a special (large bank account) profile target, so they can wait, or they are logging tens of thousands, via some malware, so it's still worth it to wait.
Such attacks rely on large scales rather than being targeted.
If you target a specific account there are much better attacks out there to do if you target specific individuals or organizations.
Yes this isn't the best method and I've had and still have a lot of objections to it (it requires the password to be stored in a reversible encryption, but that is also sadly a regulatory requirement).
But I've tested it a 10 character password using their random characters random order request method took at the least 413 (that was the lowest in my case, I didn't run a full statistical analysis on it) login requests. This is because that asking for 1st 2nd and 3rd characters, and 2nd 1st and 3rd, and 3rd, 2nd and 1st etc. are all considered "different" authentication requests by the bank.
Your keylogger would have to be also able to read the page and know that the 1st box wants the 4th character and the 2nd box wants the 3rd and the 3rd want's the 7th. This isn't that trivial, and this doesn't scale for an attack that can target 10,000's of users over a short time period.
You need to understand that banks constantly change their web pages, they monitor for bank related malware and some of them even use additional protection like for example randomizing the names of the input fields and even the number of the fields to make effective keylogging with full browser compromise even harder.
You also are incorrect when assuming that if i know the 1st 3 characters of a password it somehow helps me it doesn't because you do not have an authentication mechanism to brute force against there isn't some login page that takes the full password, and 3 incorrect login attempt lock your user and require you to initiate a recovery by phone or by visiting a branch.
This system overall is pretty good at preventing direct attacks against the bank's own system, it's resilient to phishing, brute forcing is not an option, and a keylogger can be active for 1-2 years without effectively getting the password. Effective security isn't black and white, there are is a lot of grey areas that might seem asinine and many of them are but they do work when you have the proper mitigating controls.
But let's ignore all of what we've established so far and go back to your assertion you assert that this attack is effective against high value accounts / individuals. Well that's great, because from the point of view of the bank it says hey look we've put in a control that can effectively protect 99% of our users, let's see what can we do to protect the 1%. That's how you achieve good security, you don't pool everyone into the same group, accounts with an average balance of 5000$ do not have the same risk portfolio as accounts with an average balance of 1M$. Differential security and risk management is how you apply effective security on very large groups, you employ shared controls that cover the basics and add mitigating controls based on the individual risk portfolios for each sub group.