mvsin

Something like this does exist, production systems rarely use greedy search but have more holistic search algorithms.

An example is Beam Search:https://www.width.ai/post/what-is-beam-search

Essentially we keep a window of probabilities of predicted tokens to improve the final quality of output.