Server operators are more capable of blocking badly behaving bots than users are of blocking adtech companies. Consequently you follow the limits in robots.txt or you risk getting blocked.

On the other hand, if you follow the limits in DNT, you get nothing, and the consequences of not doing so are ... nothing, so companies did not follow that. (See also P3P for another previous attempt)