As to your examples:
Person X is irrelevant. You claw back the money from fund Y. Fund Y may, in turn, have arrangements with person X requiring them to return any money, but that's actually tangential. (this is like 1st-semester law again, yeay!)
Person A owes <total sum taken out> - <total sum put in> (this is like 10th grade math again. yeay!)
In real-world experience, rather than 10th-grade story problems, the devil is in the details. Dates are important (you're eligible for recovery of losses from date range, you're liable for recovery from other date range), and Person A is likely to be vulnerable to clawback of earlier profits, even if reinvested and later lost. The profits from the scheme are separate transactions from the later losses in the same scheme. In this case, your simple math would be more fair, but probably not how the legal system would work. Yay!
You can see examples of this in capital gains taxes on people who made and lost a lot in cryptocurrency boom/bust cycles. Arbiters may have a lot of discretion in evaluating claims in these big cases, but I'd rather have the law on my side than relying on the discretion of an arbiter.
> that some Madoff investors.
You omitted the word "direct" (why?)
Person X invests in a fund Y that invests with Madoff and gets out soon enough. Person X then invests those gains "directly" with Madoff. Person X loses money with Madoff. How much does Person X deserve (or owe)?
Another example would be Person A who invests $N in the scheme. Takes out $10N principal & profits (as a test that the scheme is authentic). Invests that $10N and even more in the scheme, loses it.
How much do you claw back for (or from?) Person A?