News for hackers. Stuff that matters.
Re:Backblaze hard drive reliability stats for Q3 2016 by pfarnsworth on Tuesday November 15, 2016 @12:31PM (#12960146)
With hard drive size increasing so quickly but hard drive transfer speeds basically flat, I wonder if there are long-term implications for them with respect to recovery from backup and downtime. For example, if a whole rack goes down, and they are on 32TB drives in the future, for example, could it takes a week or more for their data to get online?
Re:Backblaze hard drive reliability stats for Q3 2016 by dom0 on Tuesday November 15, 2016 @01:15PM (#12960587)
Well, uh, yes.

No one really expects that rotational rust will get much faster, and in fact history shows that, compared to the increase in density, the increase in transfer rates are laughable at best. Between 1990 and today you are probably looking at a 20 000 times increase in density, yet transfer rates only increased around a factor of around 150-200. [In fact, from the early 1960 to today it's only a factor of about 1000. There are quite possibly few performance metrics that increased so slowly as disk transfer rate].

Why is that?

Increasing density does only marginally increase transfer speed: Most density increases are achieved by packing more tracks onto the platter, while storing more sectors per track plays a minor role. But a single R/W head can only read a single track, not parallel tracks, hence speed only increases if you pack more sectors into each track, not by increasing the number of tracks. That's why between today and 10 years ago performance in desktop or server drives only differs a little, compared to the capacity increase to 8+ TB in a 3.5" drive.

More platters also don't help in transfer rate, because the alignment of all heads on the actuator is fixed: at any given time only one platter and one R/W head is used (locked to the track). [More platters can help reduce seek time in certain scenarios though]

More disks on the other hand...

Re:Backblaze hard drive reliability stats for Q3 2016 by hinkley on Tuesday November 15, 2016 @02:16PM (#12961225)
That's not entirely true.

Higher density means more data per track, not just more tracks per disk. You get an entire track per revolution so a track with more data is more MBps. So linear reads on a higher density drive are faster, and semi-linear accesses (ie, reading two files that are next to each other) do get faster.

I remember reading a story about a guy who built a drive array with high capacity 7200 RPM drives that got within 20% of the performance of the 10K RPM setup they had, by partitioning the drives at the same capacity as the 10K equivalent. The head only had half as many tracks to traverse, so worst case access time was better, and the higher density made up for the lower RPMs.

Re:Backblaze hard drive reliability stats for Q3 2016 by baruch on Wednesday November 16, 2016 @01:10AM (#12964893)
Short stroking helps get more performance from the disk, at least in terms of latency. The bandwidth change occurs because the density is fixed and there are more blocks on the outer rings per track compared to the inner rings so in one revolution you can read more tracks and you don't need to change tracks so often.

You parent comment is right though, there are only small changes in bit density on the track in recent years so the bandwidth is not improving by much.

Re:Backblaze hard drive reliability stats for Q3 2016 by hinkley on Wednesday November 16, 2016 @12:03PM (#12968951)
So we went from 1TB to 8TB disks just by packing 3x as many tracks onto drives? I find that hard to believe, and the benchmarks agree with me.

You don't double the write throughout on a disk by doubling the number of tracks. You need more platters and/or sectors per track to do that.

Re:Backblaze hard drive reliability stats for Q3 2016 by dom0 on Wednesday November 16, 2016 @05:16AM (#12965817)
> Higher density means more data per track, not just more tracks per disk.

I literally said that:

> Most density increases are achieved by packing more tracks onto the platter, while storing more sectors per track plays a minor role.

Re:Backblaze hard drive reliability stats for Q3 2016 by hinkley on Wednesday November 16, 2016 @12:07PM (#12968994)
You are using a different definition of 'minor'. I don't think something that accounts for a 30-100% improvement is minor.
Re:Backblaze hard drive reliability stats for Q3 2016 by dom0 on Thursday November 17, 2016 @12:29PM (#12979450)
When capacity increased in the same time frame by 800-2000 % then I kinda call that minor.
Re:Backblaze hard drive reliability stats for Q3 2016 by mahyarm on Tuesday November 15, 2016 @03:27PM (#12961893)
How about the stupid idea of making a minimum block size be a multiple of the number of platters to write to. The block is divided evenly amongst all platters and are always at the same parallel track/sector. That way you can multiply the read/write speeds to be a function of the number of platters.

Instead of 50-100MB/s, you can get 4-8x the speed in large linear transfers, which helps get dead racks back up faster and would work quite well in backblaze's backup model.

Your block sizes will be huge, but I think in backblaze's case that doesn't really matter so much.

Re:Backblaze hard drive reliability stats for Q3 2016 by eye2sky on Tuesday November 15, 2016 @05:11PM (#12962769)
You cannot read from multiple platters in parallel. When you are aligned to read from a certain track on a certain side of a platter, you are not necessarily aligned on all platters. This head mechanism is not accurate enough for that.
Re:Backblaze hard drive reliability stats for Q3 2016 by rzzzt on Wednesday November 16, 2016 @01:36AM (#12964986)
Google's paper on datacenter hard drives addresses this point -- it's easier to do this and other tricks with a set of disks than attempting it within the confines of a single drive.

https://news.ycombinator.com/item?id=11172876

Re:Backblaze hard drive reliability stats for Q3 2016 by kardos on Tuesday November 15, 2016 @01:53PM (#12961013)
Is there a reason you couldn't use a second set of actuators?
Re:Backblaze hard drive reliability stats for Q3 2016 by dom0 on Tuesday November 15, 2016 @02:01PM (#12961101)
There's kinda not enough space: https://upload.wikimedia.org/wikipedia/commons/thumb/7/75/Sa...

You could probably do it if the platter diameter were reduced, which would then accordingly reduce capacity as well. At that point you could just as well use two drives and also get lower overall failure probability. Or you use SSD caches, or memory caches, ...

Re:Backblaze hard drive reliability stats for Q3 2016 by kardos on Tuesday November 15, 2016 @03:08PM (#12961693)
I was thinking of a Frankendrive with 5 actuators, which can do 5 parallel reads/writes, as long as they don't overlap. But the new dimensions would probably be the roadblock.
Re:Backblaze hard drive reliability stats for Q3 2016 by digler999 on Tuesday November 15, 2016 @03:45PM (#12962051)
I was going to ask this too, but start with a simple 2nd voice coil on the opposite side of the spindle, keep the 3.5" form factor and just make it longer. My guess is the motion of one actuator would disrupt the airflow of the other, since we're talking about micrometer-ish (?) mechanical tolerances. This might be able to be mitigated by interleaving which side each arm wrote to. So the left arm's heads address the tops of the platters, the right side addresses the bottoms.

Perhaps another option is to go back to the 1980's hdd designs where the arm moves straight down the radius of the platter. This design might permit multiple heads on the same arm. I'm sure all this stuff has been researched thoroughly.

Either way, this doubles/triples the probability of mechanical failure

Re:Backblaze hard drive reliability stats for Q3 2016 by dom0 on Wednesday November 16, 2016 @05:19AM (#12965829)
> since we're talking about micrometer-ish (?) mechanical tolerances.

Even less than that. Tracks are 100-300 nanometers wide; the head flies about 3-6 nm above the platter.

The actuator mechanism literally locks onto the signal encoded in the magnetic track and follows it as the platter rotates underneath the R/W head.

Re:Backblaze hard drive reliability stats for Q3 2016 by hga on Tuesday November 15, 2016 @07:25PM (#12963593)
Perhaps another option is to go back to the 1980's hdd designs where the arm moves straight down the radius of the platter. This design might permit multiple heads on the same arm.

At my first job, sysadmin and programming on a PDP-11/44, 1-2 fascinating Winchester drives were procured for it. They had clear plastic covers, and you could see everything, the disk, the actuator, which had two heads, and it was by and large square, I'm almost positive it was rotary, not solenoid based straight stroke like many '70-80s drives, CDC's famous line in particular.

Re:Backblaze hard drive reliability stats for Q3 2016 by kalleboo on Wednesday November 16, 2016 @01:00AM (#12964863)
No problem, just bring back the Bigfoot https://en.wikipedia.org/wiki/Quantum_Bigfoot
Re:Backblaze hard drive reliability stats for Q3 2016 by mapt on Tuesday November 15, 2016 @04:28PM (#12962385)
Or more read-write heads?
Re:Backblaze hard drive reliability stats for Q3 2016 by andy4blaze on Tuesday November 15, 2016 @12:40PM (#12960248)
The Backblaze Vault design mitigates that as the "raid array" is scattered across 20 different Storage Pods in twenty different racks. You'd need more than three racks to go down before you would be offline. Backup systems in place make that highly unlikely. Andy at Backblaze.
Re:Backblaze hard drive reliability stats for Q3 2016 by fpp on Tuesday November 15, 2016 @01:22PM (#12960678)
I'll see from your reports that you're migrating 1000s of 2TB drives to 8TB drives - what is actually happening to your old 2TB drives?

Guess you are not throwing those away so what are you doing with them.

Re:Backblaze hard drive reliability stats for Q3 2016 by brianwski on Tuesday November 15, 2016 @01:40PM (#12960883)
Disclaimer: I work at Backblaze. We securely wipe the drives, then we sell them to a "used hard drive reseller".
Re:Backblaze hard drive reliability stats for Q3 2016 by kikoreis on Tuesday November 15, 2016 @06:37PM (#12963316)
Does "secure wipe" mean ATA SE, i.e. hdparm --security-erase, or --security-erase-enhanced?
Re:Backblaze hard drive reliability stats for Q3 2016 by ams6110 on Tuesday November 15, 2016 @07:28PM (#12963601)
For that volume I'd think a degausser would be used.
Re:Backblaze hard drive reliability stats for Q3 2016 by teraflop on Tuesday November 15, 2016 @09:23PM (#12964040)
AFAIK, if you physically destroy the contents of a drive with a strong magnetic field, you also destroy the servo tracks, rendering the drive useless. Modern hard drives can't "low-level format" themselves; they're not mechanically precise enough.
Re:Backblaze hard drive reliability stats for Q3 2016 by deelowe on Wednesday November 16, 2016 @07:17AM (#12966315)
Degaussers render the drive nonfunctional.
Re:Backblaze hard drive reliability stats for Q3 2016 by SteveNuts on Tuesday November 15, 2016 @03:56PM (#12962129)
I know it's a lot easier to wholesale resell them, but it'd be so awesome if you could dedicate a portion to be sold to the homelabbers among us.
Re:Backblaze hard drive reliability stats for Q3 2016 by jerguismi on Wednesday November 16, 2016 @02:17AM (#12965121)
I don't know how that would be more awesome, essentially spending effort towards the upper middle-class hobbyists. I would rather see them sell them directly to the highest bidder and focus the effort on improving their business, or give the drives to charity.
Re:Backblaze hard drive reliability stats for Q3 2016 by theandrewbailey on Tuesday November 15, 2016 @01:45PM (#12960937)
They were reusing their 1TB drives to test new pods with. But I think they would only need a hundred or so for that, unless they die faster in that workload.

https://www.backblaze.com/blog/hard-drive-reliability-q4-201...

Re:Backblaze hard drive reliability stats for Q3 2016 by pfarnsworth on Tuesday November 15, 2016 @12:46PM (#12960292)
I guess my question is more about the long term implications of transfer speeds increasing much slower than hard drives space. And if there's a hidden risk to downtime because all of a sudden, a full rack of 32 or even 64TB drive will take day(s) to transfer as opposed to an hour or so because transfer speeds are so slow.
Re:Backblaze hard drive reliability stats for Q3 2016 by pgeorgi on Tuesday November 15, 2016 @02:35PM (#12961405)
There are implications. ZFS' raid-z2 (which uses 2 disks for redundancy data) was pitched - in 2008 or so - with the idea that disks are now large enough that recovery can take long enough that it's likely that another disk in the raid set breaks before the redundancy is back.
Re:Backblaze hard drive reliability stats for Q3 2016 by andy4blaze on Tuesday November 15, 2016 @01:23PM (#12960698)
I suppose if you had a single 32TB drive that went offline for say a week, and then once it came back online you'd have some type of pent up demand and a "slow" transfer speed. Storage systems in general spread the load across multiple devices so the effect of slow transfer speed is near zero in most backup and archiving applications and maybe more problematic in transactional applications.
Re:Backblaze hard drive reliability stats for Q3 2016 by Retric on Tuesday November 15, 2016 @04:13PM (#12962281)
Transfer increases as square root of drive density. http://www.anandtech.com/show/10541/western-digital-adds-hel... Shows 10GB drives at 249 MB/s sustained sequential transfer rate.

That takes ~11 hours to fill. At 400x the density it would take 20x as long or 9 days. I don't think HDD drives are hitting 400x the density any time soon but if they did it would be a problem.

However, in an array you could take a month to fill a drive to 75% without causing to much trouble. Assuming you had enough drives. That's around a ~80PB limit drive. IMO, the real issue is it would take another month to download all that data. Relegating HDD firmly into archival storage.

PS: I don't think rust is going to get into those density's making this far less of an issue.

Re:Backblaze hard drive reliability stats for Q3 2016 by drzaiusapelord on Tuesday November 15, 2016 @02:55PM (#12961580)
Heck, just a raid rebuild takes days nowadays if you dare use RAID5 still or, less risky, RAID6.

The smart move is to keep several smaller arrays instead of one big one. This lowers risk as well. I dont put in anything bigger than 7 disks into production. Past that I'm just asking for trouble. Its better to have 4 7 disk arrays than one 28 disk array. A drive fail means a quick rebuild and a restore is going to be 1/4 the time.

Re:Backblaze hard drive reliability stats for Q3 2016 by cm2187 on Tuesday November 15, 2016 @03:55PM (#12962118)
Stupid question but why does it take more time to rebuild a 10 disks array than a 6 disks array. I mean modern PCIe can go several GB/s and the disks are in parallel, so it should take the same time to rebuild irrespective of how many disks.
Re:Backblaze hard drive reliability stats for Q3 2016 by fulafel on Tuesday November 15, 2016 @07:35PM (#12963639)
Internet connection speeds have been flat longer so it eves out I guess.