I did not claim storage was $5,000/TB. I did say it was substantially higher than the $50/TB you're estimating.
And frankly it's difficult to estimate the log throughput in this scenario. Cisco on debug all can overload the device's CPU; systems like sssd can generate MB of logs for a single login. A fleet of systems with all debugging on could certainly saturate your network especially if you're using a SAN.
For the 20TB case, you can fit that on 1 to 4 drives. It's super cheap. Plus probably a backup hard drive but maybe you don't even need to back it up.
For the 2PB case, you probably want multiple search servers that have all the storage built in. There's definitely cost increases here, but I wouldn't focus too much on it, because that was more of a throwaway. Focus more on the 20TB version.
> That incurs lifecycle costs like training and implementation
Those don't relate much to the amount of storage.
> and then you get asked about retention and GDPR....
It's 90 days. Maybe you throw in a filter. It's not too difficult.
> if these logs are unlikely to be used
The devs are complaining about the search features, it sounds like the logs are being used.
> preventing IT scope creep and the accumulation of cruft that can mature into technical debt
Sure, that's reasonable. But that has nothing to do with the amount of storage.
Unless you want to claim storage costs $5,000/TB for 3 MB/s of I/O “multiple gigabytes per hour” with 90 day retention for a team worth of logging is not stupid on its face. Not to say that is a efficient or smart solution, but certainly not a “look at this insane request by developers” the person I was originally responding to was making it out to be.
Personally, I would probably question the competence of the team if they had that sort of logging rate with manual logging statements, but I am merely pointing out that “multiple gigabytes per hour” for 90 days is not crazy on its face and a plausible business case could be made for it even with a relatively modest engineering team.