The only ways out of these situations are: a manager that understands what you’re doing; knowing that these “story points” are just indicative and there’s no broken incentive towards gaming them.