Because historically people would fast for religious reasons, eg Lent, and a rich meal after that period would be a break-fast. A meal like eggs, pancake, bacon might form a break-fast. Then as ordinary people grew richer they'd take a rich meal of similar form much more often, habitually in the morning. Finally all morning meals became known as breakfast, even the ones that resemble a traditional fasting diet (oatmeal and water, or dry bread, say).

Am I close?

Modern usage of "fast" tends to mean not eating at all. Historic British usage would be most associated with the Lenten fast I'd imagine.