That sounds more like volatile than atomic. At its core, atomic just means that observer may see either pre-update or post-update state but won't see intermediate states.
According to the comments on the bug report, this behavior is inspired by C++'s <atomic> library, which guarantees cross-thread visibility.
Only in the sense that they won't see inconsistent (intermediate) states.
Atomicity shouldn't guarantee order per se - if you want both, you need a mutex.
It is possible to have a model where all atomic operations are relaxed (i.e. unordered) and use explicit barriers for ordering (which is closer to how some, but not all, CPUs work), but it is significantly harder to reason about and prove algorithms correct under it.
edit: wording
Still the C++ spec has wording about making stores visible in a reasonable time. The wording is non-normative as they run out of time (ah!) while trying to come up with a more formal (and implementable) specification.
edit: oh, I see I wrote 'default visibility' instead of 'default ordering'.
However, this is not the case for atomic operations (which are generally specifically intended for communication via shared variables). This optimization of removing the writing thread entirely isn't allowed if atomic operations are used, because those actually do come with a guarantee that other threads will get to see the update in finite time.
Go was incorrectly "optimizing" the atomic operations out entirely. It's a fairly subtle bug, actually, but still likely a big deal for the few programs communicating via shared variables.