Also, for environments where saving code in an editor doesn't instantly cause a change in behavior (compiled languages, web programming, working with a running Lisp image) it gives beginners an easy way to learn and troubleshoot the process of deploying their code.
The point is that the first step is getting your environment set up so that you can write[, compile,] and execute code. Printing a string is just a conveniently trivial program.
That's like saying the first step to flying an airplane is turning the ignition. Yes, that's true, but without a firm idea of what's going to be happening after that point you're only going to hurt yourself by turning that key. It's something that comes to head when learning to fly a plane/program.
But programming is a craft, so how is the first step tangential.
It's the first step when learning a new language, which is exactly when Hello World programs are customary.
You need to have your text file, download & install the compiler, figure out how to compile, and finally execute the output -- all of which are non-trivial steps for a first timer. Once you get the plumbing set up, you can then introduce other things about the language.
Just delay compilation until the third week and you'll be able to start the dialogue at objects, "is a", "has a", actions, properties. This is the language we use to deal with the world already and though it's got funny syntax attached, it's not a foreign concept to anyone. Talk about the runtime system and it's expectations later.
(Edit: I think this is also why a lot of people hate the IO monad in Haskell so much. Side effects are actually an advanced topic in pure FP so much of the literature on IO is based around flaky metaphors and "don't worry about how this works" just to get people printing things ASAP. If you just use the REPL and don't worry about implementing it then you can build up to understanding the interaction between IO and the RTS in a sensible way.)