The closest mention I can find is this [1]:
> Early monastic rules say little about how time was determined, though the Rule of the Master notes that pairs of monks took turns, in weekly shifts, trying to stay awake while the others slept in order to awaken the abbot on time. Several possibilities for deter- mining that time are likely, however, including the use of calibrated candles or lamps of oil, [...] the recitation of a certain number of psalms, [etc]
However, here is it only mentioned as a possibility, not an actuality, and no sources are provided.
There appear to have been several orders who practiced 24-hour prayer by taking shifts, see for example the Acoemetae [2]. Here we do have Monks staying up all night reciting psalms but I can't find any indication that the prayers themselves were used to keep track of time.
The only reference to prayer-as-timekeeping that I can find seems to be in Medieval recipe books [3], which apparently would say things like boil this fish for as long as it takes to recite the lords prayer three times. Dead-reckoning works well on such short time-scales!
Even as early as the 500s there appear to have been timekeeping methods accurate enough to tell that dawn was approaching. When the skies are clear you can read this from the stars and accurate-enough water clocks have been in Europe since ~ the 500s. So there doesn't seem to be a need to perfect the act of spending hours reciting psalms at the right tempo.
[1]: https://libres.uncg.edu/ir/uncg/f/M_Helms_Before_2004.pdf [2]: https://en.wikipedia.org/wiki/Acoemetae [3]: http://the-history-girls.blogspot.com/2017/08/cooking-on-pra...