Holy crap "modular castles" would be ideal for the Lego Ideas website.
It's nuts that they haven't done it already. The late 80s, early 90s crusaders/falcon knights/black knights sets could be put together in something approaching what I'm envisioning, but it was underdeveloped. The Medieval Market Village and Kingdoms Joust sets were natural jumping-off points for a modular castle system.
One thing I've always really wanted was a modular castle line, akin to the modular town building series, e.g. https://shop.lego.com/en-US/Pet-Shop-10218