Yes, your fears are justified. Not only will it change how others first interpret you, but it will also change your world view as you will only get to see things from a systems engineer perspective. Systems engineering can be a gateway into software development. But if you can be a developer just do that, don’t use a stepping stone you don’t need.