The main algorithm is described here[0]. It's the most recent videos from the 40 most subscribed channels on the instance.

YouTube's recommendations are still used. They'll appear the same as if you viewed YouTube with a clean browser session, although you can disable them in your preferences if you'd like (under "Show related videos").

0. https://github.com/omarroth/invidious/issues/217#issuecommen...