Afaik he said to not abandon it entirely. He thought we had a sort of new religion in each world, materialism in the west.
Not very far off. In their fervor, the soviets replaced a millennia old in their parts known quantity religion, with an atheist cult of "communism", the all-known "party", and the "new man". That strand of militant atheism operated the same way, and worse, than actual religion.
One could e.g. murder people and let go of compassion, as that was a "christian" thing. What supposedly mattered were the end results (the building of communism). Heck, they even made mummies and pilgrimages of their supreme leaders...
Dostoyevsky had already said pretty much the same (e.g. in the Possessed, and in Crime and Punishment), regarding the new revolutionary fervor in Russia and where Russia was headed, 100 years before Solzenyntsin.
Wasn't Dostoyevsky also religious?
His point was that the kind of revolutionary mania emerging in Russia was bad, and a kind of atheist substitute of religion (which, in its fervor and conviction lacks its controls and subtlety).