Those that do not understand FIX are doomed to reinvent it badly. I only got an appreciation for the protocol when I implemented a FIX server from scratch. Even the wierd text+field delimiter encoding is not bad (except for performance of course).
Session management, order entry and recovery in FIX is well designed and well documented (although it takes a few readings of the spec to put the pieces together). The biggest issue is the plethora of underdocumented extensions that have been bolted on to satisfy niche needs. If you are lucky you can ignore them most of the time.
Another big issue is a lot of exchanges treat FIX as more of a guideline than an hard spec and you have to workaround the lossage.
The best exchange binary protocols I have worked with (at least here in EU, borrow heavily from FIX.
It certainly kept the vast majority of retail out, but I disagree that this was the main cause of specialization. I think that the number of lone geniuses who could've beat the market using algo trading is a number approaching zero.
It's true that you don't need a PhD, or even higher education, as the math you need can be entirely self taught. However, you need plenty of specialized knowledge to avoid the accidental discharge of the vast number of very expensive footguns that litter the landscape. This knowledge can be gained from literature, but without having someone streetwise as a guide, it's going to be very difficult to discern genuine insight from wishful thinking.
The part that is hard are trading costs and slippage. But assuming that retail investor’s capital is relatively small and no trading costs (Alpaca provides this), an individual can do pretty well.
The parts that are hard for hedge funds is slippage, illiquidity of assets, trading costs, and client demands. As an individual you really don’t have to worry about those things and consequentially things become much easier.
The only downside is that it’s harder to get as much cheap leverage as the big guys. Also some (HFT) strategies are off limits to individuals, but that’s fine, because HFT really isn’t a good way to make money in the first place.
Data is available since 2019-03-30 for majority of the supported exchanges. Yes it's all from public APIs.
The finance sector’s biggest meme for the last decade and a half was that PhD quants were required to program trading algorithms when really a single exchange cost $12,000 a month for real time data, it came with horrible documentation, support, antiquated protocols, FIX, and then you had to figure out how to co-locate your own server to connect to the exchange fast enough to prevent front running
The gatekeeping is what kept this specialized for the most part