As a non-native speaker, my understanding of "notorious" so far has been what you described as the neutral form. However, to me, it does have a slight connotation of obsession or "doing it so much as to be annoying".

I am obviously not an authoritative source on how words are understood, but just wanted to add this data point.