As soon as the soft limit is hit, fire off an alert. Have the hard limit set at double or more.
It is also trivial to set up, and does not require me to figure out how to set up an OS alert, or trust that whatever alert process is running. So it is an essentially fail proof alert that works the same on any OS.
What’s a good argument against it?
But this way, I directly get an idea of the increase in storage we will need over the next year in order to plan the next hardware expansion.
They set all applications to "debug" level logs in production and were generating multiple gigabytes of logs per hour.
They wanted 90 days retention, and the ability to do advanced searching through the live log data so they could debug in production (they didn't really use their dev or stage environments, or have a process for documenting and reproducing bugs).
Seems pretty reasonable to me.
I wonder if there's anything as good in the open-source world. The closest thing I can think of is Clickhouse's "new" JSON type, which is backed by columnar storage with dynamic columns [2].
[1] https://www.uber.com/en-BR/blog/reducing-logging-cost-by-two... [2] https://clickhouse.com/docs/en/integrations/data-formats/jso...
The design described there is what Uber should be logging in the first place. Instead they are logging the fully resolved message and then compressing back into the templated form.
However, the compression back into the templated form is a good idea if you have third party logs that you want to store where you can not rewrite the logging to generate the correct form in the first place.
And that was a number hundreds of times higher than what they were replying to, just to make a point.
If the solutions are:
* rewrite that part to add retention, or use better compression, or spend next month deciding which data to keep and which can be removed early
* Wiggle a thing in panel/API giving it more space
The second will win every single time unless there is pushback or it hits the 5% of the developers that actually care to make good architecture not just deliver tickets.
- set up some service that talks to a s3 bucket
- set up that bucket in the same region/datacenter
- send a decent but not insane amount of traffic through there (several hundred Gb per day)
- assume that you won’t get billed any data transfer fees since you’re talking to a bucket in the same data center
- receive massive bill under “EC2-Other” line item for NAT data transfer fees
- realize that AWS routes all traffic through NAT gateway by default even though it’s just turning around and going back into the data center it came from and billing exorbitant fees for that
- come to the conclusion that this is obviously a racket designed to extract money from unsuspecting people because there is almost no situation where you would want to do that by default and discover that hundreds to thousands of other people have been screwed in the exact same way for years (and there is a documented trail of it[1])
1: https://www.lastweekinaws.com/blog/the-aws-managed-nat-gatew...
Even has ha mode.
On a per-month basis, the grossly exaggerated number is in the single thousands. The non-exaggerated number is down in the double digits.
$50/TB is a lowball if you want much of the data to be on SSDs, but taking an analysis server and stuffing in 20TB of SSD (plus RAID, plus room for growth) is a very small cost compared to repeated debugging sessions. Especially because the SSD has to deal with about 0.01 DWPD.
Current single disk solutions are around $25/TB for HDDs and ~$100/TB for NVMe.
At a minimum you're looking at $54k just for raw capacity-- assuming no backup, no chassis, no networking, and no redundancy.
More reasonable estimations would be in excess of $400/TB.
Unless you want to claim storage costs $5,000/TB for 3 MB/s of I/O “multiple gigabytes per hour” with 90 day retention for a team worth of logging is not stupid on its face. Not to say that is a efficient or smart solution, but certainly not a “look at this insane request by developers” the person I was originally responding to was making it out to be.
Personally, I would probably question the competence of the team if they had that sort of logging rate with manual logging statements, but I am merely pointing out that “multiple gigabytes per hour” for 90 days is not crazy on its face and a plausible business case could be made for it even with a relatively modest engineering team.
I did not claim storage was $5,000/TB. I did say it was substantially higher than the $50/TB you're estimating.
And frankly it's difficult to estimate the log throughput in this scenario. Cisco on debug all can overload the device's CPU; systems like sssd can generate MB of logs for a single login. A fleet of systems with all debugging on could certainly saturate your network especially if you're using a SAN.
For the 20TB case, you can fit that on 1 to 4 drives. It's super cheap. Plus probably a backup hard drive but maybe you don't even need to back it up.
For the 2PB case, you probably want multiple search servers that have all the storage built in. There's definitely cost increases here, but I wouldn't focus too much on it, because that was more of a throwaway. Focus more on the 20TB version.
> That incurs lifecycle costs like training and implementation
Those don't relate much to the amount of storage.
> and then you get asked about retention and GDPR....
It's 90 days. Maybe you throw in a filter. It's not too difficult.
> if these logs are unlikely to be used
The devs are complaining about the search features, it sounds like the logs are being used.
> preventing IT scope creep and the accumulation of cruft that can mature into technical debt
Sure, that's reasonable. But that has nothing to do with the amount of storage.
More like $15/TB. $100K for 2 PB of storage with redundancy and backups is quite reasonable.
$300 is moving towards refurb / shucked prices.
Where? For new prices I'm seeing $350 at amazon, $350 at B&H, $280 direct from newegg, $280 at serverpartsdeals.
When you buy a SAN to present a bunch of disks as one thing to the rest of the machines.
Don't get me wrong, I understand that a single-digit kUSD/month is peanuts against developer productivity gains, but I still wouldn't be able to take a developer making that suggestion seriously. I would also seriously question internal processes, GDPR (or equivalent) compliance, and whether the system actually brings benefit or if it is just lazy "but what if" thinking.
End number doesn't matter, what matters is developers thinking about how long data should be stored and what data should be stored.
Not doing that analysis and overprovisioning 4x will just cause disaster in 2 years instead of 6 months.
So what? Just buy more. Storage is cheap.
It's hard to have a discussion here without understanding the scales involved. Is the problem that they're wasting 100 GB or 100 TB? And if the issue is truly that they're wasting 100 TB, then clamp down on it as part of cost reduction efforts. The truth in most organizations is you get rewarded for eliminating mountains of waste, but trying to prevent the waste in the first place brands you as someone difficult to work with who is standing in the way. Why not lean into that?
If you just "I want 20 GB of storage", i'm not going to give it to you.
Storage is cheap in relation to other things. Just have a good reason to why you need it.
So, you are not expecting that your co-workers have good reasons for what they are doing? Maybe the hiring bar at your place is too low then.
I prefer to work at places where my default assumption is that everybody around me is smart and responsible. Lift's lots of worries from my shoulders (and tends to benefit the stock price over time too and thereby my income).
If I wanted unquestioned control, I'd run my own shop. If I want the best product, then I hope that people question my assumptions.
The way this was phrased was more from the angle "who knows what these guys were thinking; if they can't give me a good reason, no way they will get storage space as I don't trust that they make good decisions on their own".
Let's say I work in DevOps and want to optimize cloud costs. In that case, I would challenge the size of everything, the use of higher-costs services, the number of regions, all that - but the team might want more regions and bigger resources to improve latency and performance, and use more high-cost services for developer experience, and ship features without having to think about utilization.
It's a tug of war, and only works when you have forces on both sides to balance out. Being too conservative might stall innovation or make things too slow to save a buck, not being conservative enough might drain funds or make things impossible to scale.
Yeah, any workplace in which the word "war" was used in the context of colleague interaction saw me leave within a few months.
I like to plan those things ahead of time with all stakeholders involved, then we work together instead of against each other.
Unless you work with clones of yourself, there will always be differences in opinions and priorities, and not every feature and bug fix can be a company-wide stakeholder meeting, and you certainly will not get any social points for trying to micro-manage other teams.
But the attitude needs to be "let's put the requirements on the table and see what we can do" instead of "you don't get what you want unless you give me a good reason". The latter comes from an angle of distrust which I'm arguing against. The former comes from an angle of collaborative problem solving.
In a company in which I go to a team relevant to a project and like to engage in a discussion and am met with an attitude of "unless you give us a good reason we'll stop talking to you", the atmosphere is not one that will keep me personally for long. YMMV.
> I believe you are intentionally misunderstanding.
You are free to believe what you like. Opening a reply with such a sentence is pretty sad though. It does not foster a healthy atmosphere, nor does it match reality, I might add.
Your response hitched on a single word ("war") within a common phrase ("tug of war", a game). While it might have been accidental, such answers mislead from the actual discussion (and tends to be used as distractions when no good answer is present).
> Of course there will be differences. That's why you sit down and plan things together, pulling in and coordinating with all _relevant_ stakeholders.
When you discuss new architectures or large projects, this is a given, but this covers only a small portion of company operation - the rest is organic day-to-day work, which slowly but surely distorts initial assumptions. Slowly boiling the frog, so to speak. Think one team making changes that affect request patterns, another team making something that is accidentally quadratic, and a third team suddenly asking for a large number of cloud resources to carry this that should absolutely be challenged.
And at the same time, teams are under different organization units with different budgets, schedules, leaderships and priorities - and most certainly don't care about daily scrum work of other teams.
> In a company in which I go to a team relevant to a project and like to engage in a discussion and am met with an attitude of "unless you give us a good reason we'll stop talking to you", the atmosphere is not one that will keep me personally for long. YMMV.
No one said "we'll stop talking to you", but "you get what can be justified". If you take offense to be challenged and would rather work somewhere else, you do you, but if you can't justify your request I'd argue that you are not doing your job properly in the first place.
Are you going to rearchitect my system for me?
“There’s 10 people here whose time I bill out at $250/hr each, spending an hour discussing whether to buy a $1,000 software license? Why?”
"Some of us don't like jumping into things without looking."
I wager your boss would not be amused with me.
Good to know. FWIW i should also be avoided. It's tempting to use, since most programs use it as a counter, so it /should/ standardize the logfilesizes. But in practice it's very tricky to get a definitive disk space requirement with it.
But, unlike Toyota, we do have disk space alerts.
Sometimes the problem is also entirely political, the management needs to tell client and charge them for more storage and won't accept the change till that happens. Meanwhile clock is ticking...
The problem in this case was twofold:
- admin had the job to implement database backups. He didn't factor in backup size when allocating disk space. So this was wntirely his own fault.
- the database does store certain transactions for a certain period, so this grew initially until it it setteled at a certain level. Because the margin of storage was slim, this caused the problem
Because most of our cases where that happens was either lack of planning or lack of communicating that plan. By far most common one was "neither dev nor client knows the data volume in longer period". Which is fine as long as that's also communicated, but that's also often a problem.
But I'm not denying of course that there are just shitty incompetent ops departments, just for the other customer we had dealing with ops department that had:
* backup storage (which was some remote FTPS server IIRC) provisioned so slow the backup wouldn't copy within 24 hours. And the backup size was below TB. * weeks long delays with any resize.
And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the disk 40 Gb. He could have easily just allocated 100 Gb and call it a day, this way we had to go to him 3 times to scale it up in 10 Gb steps each time, including a the stress of figuring out why things are failing (something that the admin should have seen on his monitoring system).
Admins are my heros, but please if you allocate disk space, just take the biggest expected backup and multiply it by π. And if you need to be stingy with storage for some reason, be stingy, but decide when and where to be stingy — and at least keep an eye on the monitoring and upsize the storage before it is too late.