You are correct on the number of years between plantings, but really only 2-3 years off are needed to clear the fungus. Alfalfa just lasts longer.
As another comment mentions, some other common crops have the same disease risk, so avoid potatoes and eggplants too many consecutive years.
Fallowing it an option, but a little nitrogen for a home garden shouldn’t be too prohibitive. When you have a lot of acres it can be a meaningful cost saving measure but it’s not huge.
Nitrogen is necessary for proteins so you can get larger yields, but if some other nutrient is missing then nitrogen alone won’t help. K and P also need to be present in good quantities, as well as a mixture of iirc 17 elemental trace elements. I’d use a well-rounded fertilizer like 7-7-7, and if it wasn’t growing then assume there was a disease or pest at play.
So for a given plot you do tomatoes for one year every 4-6 years, do I understand correctly?
Does it matter so much what other crop you plant, in terms of reducing the fungus? Alfalfa is just a good choice for its nitrogen fixing? Could you leave it fallow for example? (just trying to get ideas for my garden)