But this is not ad-hoc at all. This is how it is deliberately designed so it degrades gracefully depending on what optimizations you choose.
Not all compilers even really degrade. GCC does post-hoc variable tracking (and thus is mostly unaffected by optimization except when values are optiized out completely). For declarations and statements, the info is always on the declaration, so it won't be lost.
Both compiler guarantee that at -O0, all debug info will be kept.
In Clang/LLVM, for example, nodes are annotated with debug information using LLVM's metadata system. However, it is strictly optional for any given transformation to preserve the metadata on IR nodes. So what makes it through after optimizations is kind of a "best effort" affair.