But a lot of serious people worry about the alignment problem when applied to future, actually "intelligent" (in the sense of having powerful planning ability), AIs.

We don't know when they'll be possible, but we do know that we don't currently know how to train them in a manner where they'll behave as we like when far outside the training domain (we cant even clearly define what we'd like!).

This is actually a very complex topic and I wouldn't be able to do it justice in a hackernews comment.