It's by no means perfect. For one thing if I have a choice of 0.78 ROC based on 40 days worth of data and 0.77 based on 10 days worth of data I'd take the later because I know the later one adjusts more quickly to my changing preferences. Also my selection algorithm breaks up the input into (say) 20 clusters and shows me the (say) 15 top results in each cluster and I know I like the results from that better than if I just take the highest ranking results.
It's by no means perfect. For one thing if I have a choice of 0.78 ROC based on 40 days worth of data and 0.77 based on 10 days worth of data I'd take the later because I know the later one adjusts more quickly to my changing preferences. Also my selection algorithm breaks up the input into (say) 20 clusters and shows me the (say) 15 top results in each cluster and I know I like the results from that better than if I just take the highest ranking results.