Is this true for technical books? I can't imagine that a book like Practical Malware Analysis or The Rootkit Arsenal would ever be "wildly popular".

If publishers like O'Reilly were operating on this model then why would they ever publish any books on any niche subject? Surely they would only ever publish books targeted to a mass audience.