Steelman possibility: There's 20 people in 10 departments giving him conflicting non-written requests for disk space, his own request for new disks is held up somewhere, and his boss keeps telling him "Just do what you're assigned" which doesn't include preventing a full disk condition.

Note that a few dozen terabytes is also nothing. 12 TiB is about $800 on an SSD and about $250 on a HDD. Plus some overhead for the enclosure and redundancy, of course. It costs on the order of a day's pay for an engineer, at most.

I don't excuse wanton waste of storage because it's easy for sloppy practices to balloon to massive confusion and inefficiency. But that discipline should be enforced by good engineering practices, not by limiting resources.

This is such a uninformative comment. Business/Enterprise storage is far, far more expensive than a drive you pick up at Best Buy or Fry's.

I'd bet the salary of the guy who is making this a pain could buy enough storage that they don't have to worry about it anymore.

Depends on the org. Some orgs are well managed, others don't want to pay for stuff behind the scenes until it bites them in the ass. They don't know what they don't know.

I’ve been getting organic and cage-free hard drives nowadays.

> and cage-free hard drives nowadays.

SOmewhat recently I toyed with an idea of HDDs with an 'interposer' which would be just a dumb ATA2iSCSI interface, with Ethernet or WiFi connectivity.

That would allow you to place those drives literally anywhere and with a minimal footprint.

Sadly it would be too costly for the home usage (I assume about $30/unit at best) and for the enterprise usage... 1Gbit is too slow, 10Gbit is too hot and bulky and then you can't sell a $3 plastic case for $100 'vendor approved with light-path(r)(tm) diagnostic indication' for each drive.

Well, that's a choice your ops people make.

There is absolutely no technical reason to provide all of your storage needs with a single quality of disks. You may do that to get some economies of scale, but if the impact is that high, you should rethink it. You can just as well have two different solutions, one with a huge amount of disk space, little redundancy and low performance, and one with a limited amount of space, plenty of redundancy and high performance.

And yes, I know that goes against every common ops procedure, because disks are so cheap. But the next thing you hear is always that disks are not cheap at all, and you really can't have it both ways.

When you start requiring redundancy, you start needing 2-6x the number of disks, minimum.

Still cheap, but not that cheap.

The problem is that staff time costs an order of magnitude more and most of these things can go wrong. For example, you don’t just want to buy random lots of whatever disks are cheapest because then you have to track recalls and firmware updates for everything.

If you have two tiers available, someone will use the less reliable one because that’s how they fit their budget but then because it’s “in production” they’ll expect the same level of service.

All of these are manageable but what you’re really hearing is that the technical issues are really the tip of the social iceberg most organizations have. One of the reasons people pick AWS isn’t just that it’s usually cheaper than the full cost of rolling your own but that lots of these things don’t affect you: you never fail to provision an EBS volume because the VPs of finance and IT are still arguing about procuring a new rack of disks, people can’t request endless customizations because the options are “take it or leave it”, etc.

Single disks are cheap. Multi-disk storage systems are not. Multi-tiered storage systems are even more expensive. I don't see an inherent conflict there.

So don't waste your money on it then!

Why in the world would you buy one "Enterprise" disk instead of three "consumer" disks?

Well, "enterprise storage" can actually be multiple, redundant copies of the data behind the scenes.

So, internally the storage team might quote say $1000/TB (simple numbers for an example) for a given storage quantity. And behind the scenes they'll likely have at least redundant storage arrays, plus backups and 24/7 monitoring for all of the data, etc.

Yes, but that still refutes the point all the same.

Because if you're providing services to a large business, you're typically not managing servers with physical disks attached. You're using a SAN with fiber connections or NFS mounts. Most SANs require specific drives sold by the vendor with firmware they've tested, mounted to custom sleds etc. You can't just connect a WD Mybook drive.

Mom and Pop businesses using single servers can do all they want in regards to drives, I don't care. I would argue they shouldn't have servers, but if you do have servers, you should at minimum use RAID etc, not a drive plugged into your USB port.

If you are in that situation, then obviously you have to do what you have to do.

Absolutely none of that has to apply to the sort of situation we are talking about here.

The only real reason to spend more per disk is when you know all your disks are going to fail, and extending the lifespan per average disk will definitely save you more than the enterprise markup costs. So you better have dozens or hundreds of disks in the first place.

For any truly valuable data, you should at minimum have a backup in one different physical location. That backup should be include at least one redundant disk. None of that is worth spending a dime on more expensive hardware.

You try telling a VP that their business unit can't function because you decided to purchase the cheaper drive.

All disks fail eventually. Outliers may run longer than the MTBF for that drive model, but they all fail eventually.

And backups are fine for restoring data, but they don't help provide access to that data in a timely fashion. That's why people use SANs, AWS etc.

The cost of a SAN storage array is a nit for a business making cars (like Toyota) or selling insurance (like my company).

It's very unlikely for all of your disks to fail at the same time. Even if they do, that's why you have an offsite backup. The name of the game is redundancy, not longevity.

In practically every use case, two consumer disks will be better than one enterprise disk. Once you start failing enough disks often enough, longevity can be worth the additional cost. Until then, it just isn't.

Actually, it's not uncommon for a batch of disk from a vendor (the same lot #) to have failures.

And again, a backup is fine for deleted data, or fire/ransomware. But for day to day operations, no one is really willing to wait for you to restore from a local backup, much less an offsite backup.

> no one is really willing to wait

And they are willing to wait for you to rebuild your raid array?

Either you lost your data from a disk failure, or you are waiting to lose your data from a disk failure. How are we not on the same page here?

I think if you had worked/managed a SAN, you might realize that a single disk failure is a non-event. I'm not talking about a JBOD or an storage shelf using RAID5, I'm talking about a Netapp or similar system that can easily handle disk failures without interrupting service.

Because the enterprise disks are rated for years of continuous service and have things like firmware which doesn’t lie about whether data has been committed durably for the sake of benchmarks on review sites.

None of this means that you should trust any particular disk enough not to need redundancy, backups, etc. Companies can and do make trade offs based on their needs and management competency and people have been shifting software for a generation to rely less on the hardware – back when Sun announced ZFS, one of the major appeals was that you could drop expensive hardware RAID controller dependencies in favor of cheap boxes of disks – but there isn’t a single global optimum point. A lot of enterprise purchases are driven by being able to satisfy your most demanding users with the same service as everyone else so you can avoid needing your admins to be trained and experienced with dozens of different storage systems. That last part especially extends to testing: for example, does your rack of consumer drives with software redundancy come back up cleanly after a kernel panic or power outage, especially a nasty one like a fluctuating brownout? Depending on your budget, needs, and technical bench depth you might reasonably conclude that the savings are worth the ops work, or that it’s safer to pay an enterprise storage vendor who’ll certify that they’ve done that testing and will have tech support on-sight within an hour, or that you’ll use AWS/Azure/GCP because they do even more of that tedious but important work. All of those can be right, but I’ve typically found that people in the first two categories think they’re doing better than they are and would be paying less for better service in the cloud.

It’s because all consumer disks are garbage. The way it works is that disks are tested. Disks without failures are sold as “Enterprise.” Those that have failures are labeled and sold as “consumer”, not thrown in the bin.

Are you talking consumer prices, enterprise redundant SSDs in a Data Center, or enterprise cloud storage? In my experience the latter two could add another digit to your price.

And all of that needs to be backed up, secured, stored, tested for validity regularly, and so on.

More disk space isn't one drive, but redundancy too And backups. And even network bandwidth.

This thread sort of highlights the problems. DEVs don't get it. Nor does management.

Entry level Enterprise grade HDD storage will run you about $1,000 per TB.

All Flash arrays will get even more $$$$

Price out a Nimble or PureStorage array can see if they will sell you 12TB for $800... I will wait.

The audit department at my current gig says all records must be retained for eternity because of regulatory reasons but cannot direct anyone to the relevant regulation.

on-prem block storage like a Scality Ring is where it's at.

Or there was an array that was thin provisioned at 200% over and every 100GB request playing chicken with a storage over run condition....

“No problem, just put in a ticket”