Also the byzantine empire got eventually destroyed by the turks The byzantine government which relied on tax collector->emperor->soldier was less efficient than the simplicity of the knight->king structure of the turks. The more aristocratic Franks and Spaniards had no problems pushing back the moors, because turns out the motley crew of vagrant crusaders were more effective than an easily corruptable bureaucracy.
The key development in the pike square was mobility. Earlier infantry forces using similar weapons had more rigid formations, with a single direction that was supposed to point towards the enemy. That made them vulnerable to outflanking by cavalry / skirmishers / archers. Pike squares drilled rapid turns, which made outflanking them much more difficult.
The Byzantine Empire remained capable and strong until around 1200, when a series of incompetent emperors destroyed it. In particular, one of them had the clever idea of using the Fourth Crusade in a power struggle against another. That resulted in the loss of Constantinople. While the empire eventually got its capital back, it never recovered. The empire that the Ottomans later defeated was only a shadow of its former self.
As for the supposed effectiveness of the Franks and Spaniards: the Reconquista was a series of wars from 718 or 722 until 1492.
Military tactics don't necessarily scale. Knights where probably optimal at the 10k-20k army size range, but there wouldn't be able to scale to the size of say Alexandrian battle (47k vs 120k).
In the bigger picture, the Byzantine Empire lost the battle but ultimately won the war. That marked the end of decline for the empire and started the Komnenian restoration, making the Byzantine Empire a major military power for the next century.
The cavalry charge became a lost art, because people learned how to counter it. Even during its heyday, heavy cavalry was vulnerable to feigned retreat tactics, possibly for psychological reasons. Byzantines used it to defeat Normans at Larissa a year after Dyrrhachium, and Mongols later used it effectively in their invasions of Poland and Hungary. And around the time of the Hundred Years' War, infantry had learned how to hold its ground against cavalry charges.
The cavalry charge became a lost art, because people learned how to counter it
And as for being a lost art, the art was really in the breeding and teaching horses to trample through a formation. I imagine it would have been a useful resource in later periods, it's not as there where no <8k unit battles anymore. It's just that since that wasn't the META for the important battles nobody could justify maintaining such an expensive skill.
As I mentioned, the immediate counter to a cavalry charge was feigned retreat. It worked, because it looked very much like a successful charge. You charge, the enemy breaks formation and flees. You give chase, but you are a bit too eager and proceed a bit too far. Then you find yourself in a trap, exhausted and surrounded.
You mentioned Agincourt, but the Battle of Crécy (1346) was similar but larger.
The Byzantine Empire continued to have professional infantry, because they were still wealthy enough and capable enough to support a standing army. They also had horse archers, which had been a key component of the army in the late Western Roman Empire but had almost disappeared in the West afterwards. And before their decline, their heavy cavalry also consisted of professional soldiers instead of individual warriors.
Professional infantry reappeared in the West in the late middle ages in the form of mercenaries. Then it turned out that pikemen were generally more effective than knights. While a charging horse is big and scary, it's not actually large enough to crash through infantry lines. If the infantry doesn't chicken out, the charging knight has to.