I think you have explained it well. Since English is not my first language, I can't even tell that the word 'just' is supposed to be offensive (and probably same with many other speakers of English). However, attitude can easily put off people and that can be done using any set of words.
But your attitude is exactly what many people find objectionable. People who are fit, but who may be dealing with something more challenging than you understand, don’t benefit from input dismissing the challenge. More than that, dismissive and demanding, commanding or ignorantly prescriptive input can be a mental and often emotional tax that undermines actually facing the challenge at issue.
A lot of times people will do much better at facing their particular challenges if you just let them find their way, or even if you just defer to how they want you to help (if they even do). This really isn’t so much about the word “just” other than its exemplary utility in getting this dynamic wrong.