Can it be done sustainably? Can a complete ecosystem be established where the feeder fish don't have to be harvested from elsewhere?

I don't know, but usually, nature is better at creating ecosystems than we are.

That doesn't mean it is better at creating food for us.

Sun -> algae -> small fish -> medium fish -> fish we eat.

Very doable, the problem comes in dealing with the waste.

+1

Though I'm struggling to think of even just one example of a human-created ecosystem designed to substitute for a natural one that's been at least as sustainable as those existing in nature.

Yes, people do it with Talapia right now.

Not really, unless you're willing to stretch sustainable to the breaking point.

As I understand it, after a few years the floor of the fish farm is basically a solid toxic layer of fish food and fish shit, and it becomes unsuitable for farming. That's not sustainable.

There is nothing wrong with the process, just the implementation. The bottom layer is not radioactive, it just needs to be processed properly. Since there is not going to be anything there of significance which is not organic there is not reason you could not move fish from pond A to pond B, and then process the bottom layer of the pond to be fertilizer that will help grow more food to feed to the fish, etc. Since you are talking about years of use in a single location it is pretty obvious that you are not actually talking about known sustainable processes but are tagging that label onto something else.

I think "properly processing" a 3 foot thick layer of sludge on the ocean floor might be a little more difficult than you're making it out to be. Or maybe it's easy. I don't know.

But as things stand right now, the process as implemented is completely unsustainable.