We are part of society, so I hope we don't "trained a perfectly rational AI" but train something that operates within constraints that we agree upon as a society.

A "perfectly rational AI" might also kill of disabled people as unproductive, but we as a society will need to introduce bounds and constraints and objectives in line with our human values.