Well, why would it downgrade pagerank which is based on web graph connectivity? You're not changing the graph in any way with your click, no?

But Google definitely tracks such short clicks and considers them a bad signal, very important in ranking. Same for Yandex, search for "dwell time" (visit duration) in its code/factors.