I'm not sure why you would want to restrict calories in your case? Even if the work on fasting from animal models carries over to humans beyond the benefits of reducing metabolic syndrome (which is unclear), being underweight is definitely a risk factor for all cause mortality.

If anything it kinda sounds like your natural 'set point' already has you calorie restricted insofar as it would.be beneficial