> If the crypto is strong, and the key never exfiltrated, then this may be even better than an explicit overwrite, which on some media (like magnetic drives) may still leave trace hints of the old data that advanced techniques could recover.

The idea that magnetic drives left readable traces behind after an overwrite is a myth (especially of modern drives; if I recall the article I'm thinking of correctly the myth may have had a _slim_ chance of being true with old drives [like 100MB days]).

I'm fairly certain you're correct, but the fact that various secure delete tools overwrite data multiple times probably helps propagate this myth.

I wonder how it is even possible in those 100MB disks.

The idea was that, since the drive head doesn’t pass over precisely the same area on the track every time it writes to the disk, the track has to be wide enough to account for this lack of precision. Thus, a single overwrite pass could leave a portion of the track’s edge unchanged and readable with an electron microscope or something.

I always figured that a write wouldn't change the magnetic field 100%. Say it's 90% effective at changing the magnetic field. There's now four options:

1 overwritten by 0: 10% field strength

0 overwritten by 0: 0% field strength

1 overwritten by 1: 100% field strength

0 overwritten by 1: 90% field strength

But I guess that's not quite true.

I looked into back in the day (but not quite as far back as that), https://alicious.com/secure-drive-data-wiping/, couple of references there.

Low density data, paired with not very accurate head tracking.

I agree, there is only one reasonable method to consider a drive "wiped": encrypt all data written to it with a key; wipe == destroy the key. Drives are basically little computers now, they have full-featured CPUs, OSes, memory, write caches and persistent buffers, they compress data before writing it, lie to you about where the data is (ok fine, it's a legitimate abstraction), and who knows what else. My point is that drives have become way too complex for wiping by "overwriting" "every" "block" to ever be an option again (and yes all three of those words deserve scare-quotes). The answer to this problem has become: always encrypt, lose the key to wipe. Period.

From your comment I'm not sure if you're aware but there are two established standards that do just that:

ATA Secure Erase is an optional security feature implemented by many flash-based disk devices specifically because write degradation and overprovisioning preclude them from being wiped via overwrite as you would magnetic media.

More SSD disk controllers than you may realize have an AES encryption engine and persistent storage for the key. The controller is encrypting/decrypting on the fly transparent to the user/system. A secure erase simply rolls that key (and flags all the cells as empty).

As noted in the sibling comment, there must be trust in the device itself that the old key is unrecoverable (or the data recoverable if the prior key had been compromised).

hdparm on any linux liveusb (e.g. Clonezilla) and an eSATA dock and you can nuke a few disks a minute with two commands:

  hdparm --user-master u --security-set-pass p /dev/sdX
  hdparm --user-master u --security-erase-enhanced p /dev/sdX

hdparm has other arguments that can tell you if this is supported and how long the drive says it'll take to do it.

The other standard is TCG's OPAL. An AES media encryption key (MEK) is generated to encrypt/decrypt all data on the fly. The MEK is encrypted with a user-defined key encrypting key (KEK) and that encrypted MEK is stored on the drive. At boot the user (or some external process) needs to supply the drive with the KEK to decrypt the MEK and begin transparently decrypting/encrypting data on the fly. Splitting the disk into multiple "locking ranges" with independent MEKs and KEKs is also described in the spec enabling secure wipe of just a portion of the disk. Lose the KEK or destroy the MEK and the data is unrecoverable.

I'd say this option still requires trust in the disk itself but since TCG dictates not just the interface but also the implementation and has a corresponding certification process (as well as reliance on an off-device key), I reckon TCG OPAL devices should be more trusted than those that simply claim to implement the ATA Secure Erase command, but I haven't had an opportunity to test any.

Cool, thanks for sharing.

It's interesting that even the drives themselves use this strategy to "erase" data; it doesn't actually overwrite the cells, it just always encrypts all data that goes to cells and rotates the key when it's asked to erase. This strategy is clearly superior, it doesn't cost any wear on the cells to perform, and it's instant.

The multiple locking ranges is pretty neat, I'd never heard of that.

How can one trust it though? Presumably the key has to make its way into the drive to actually decrypt the data. How can one be sure the key was actually deleted and not archived somewhere in the controller’s storage?

Ideally, data is encrypted at the CPU (or coprocessor or TPU chip) and sent to the drive encrypted. The drive never gets the key, its only job is to store encrypted blocks correctly. In the TPU case, the TPU chip controls decryption and the key never leaves the specialized, tamper-resistant chip; wiping means sending a command to the TPU to rotate keys, thus instantly destroying any access to the data.

A coworker once insisted on fondling discarded drives with extremely strong magnets, then taking a power sander to the magnetized part. I wasn't sure if this was overkill or not. Is there any wisdom in this?

Sounds like someone needing a reason to use a power sander

Yup exactly why I needed to buy my Milwaukee right angle die grinder… ask my wife ;)

When I worked for the UC Observatories we physically destroyed most retired drives, just in case of sensitive information. Some personal machines' drives were excepted but there was a good deal of hammering.

Well if they wanted to be 100% certain nothing could be recovered, then that is a very good way to make sure.

what if you just fill the whole disk with random bytes? Wouldn't that take care of it ?

You don't have access to "the whole disk". What the drive presents to the OS as an array of blocks is just an abstraction now. There are at least 3 commonly known ways that "filling" the disk with random bytes could fail to destroy the original data:

* If any part of a block is found to be corrupted for any reason, the block is transparently set aside and never used again, but its contents may still be present to read for someone that is willing to bypass the drive's firmware.

* To prevent uneven wear, ssd capacity is over provisioned which means it has more cells than what it tells the OS. In general, when you "overwrite" a block on your drive, that data likely does not actually overwrite the cell where the data used to be. The drive picks some other unused cell, writes the data there, and does some bookkeeping so it knows that this new cell is what the OS means when it asks for that block again. When the OS "fills the disk" it may not end up overwriting some of those cells due to the firmware's arbitrary wear leveling algorithm.

* Many drives have a local write buffer that uses different, faster persistent storage to temporarily save writes to the drive as it takes its time to write out cells. This is to prevent data loss in the case of a power outage. In case the drive hadn't fully written all the blocks to the main storage when it boots back up again, it finds the data in the write buffer and finishes the job. The write buffer has both of the previous issues except its even more arbitrary and depends on load profile which your random filling routine is not guaranteed to hit.

There is no such thing as securely overwriting blocks anymore, because "blocks" are a fiction presented to your OS by the drive's firmware. There is only one way to be sure the data on the drive is filled with random data: make it random to begin with, and ensure that it stays random by destroying the decryption key.

>* If any part of a block is found to be corrupted for any reason, the block is transparently set aside and never used again, but its contents may still be present to read for someone that is willing to bypass the drive's firmware.

How useful is one block of recovered data?

Blocks can be up to 4mb. You could store all your passwords, private keys, contact/identity info, etc

Really now. Last time I cared to look at any of this, it was all 4K blocks.

You are confusing physical blocks (i.e. flash memory) with logical blocks (i.e. filesystems).

Edit: HDDs typically have 4K blocks though ("advanced-format drives"), but we're not talking about HDDs.

Yeah, I honestly have not cared about how hard drives work in about 10 years since I built my last array. I have better things to do with my life than worry about this stuff. I just want to connect a drive, reformat/partition if necessary, and then go back to work. A single SSD now read/writes faster than a 16 drive array (got close in RAID0, but that's just dumb). Now, if SSDs could just get a decent volume size.

They've got what I consider decent size, it's just that the price can be a bit steep compared to an HDD, at least at quantities of 1. $700 will get you an 8TB SSD.

8TB at blazing speeds to boot in a single device. To get decent speeds from HDD would require at least 8 drives striped together. So not only do you need 8+ HDD drives, you also need the enclosure. That would easily double the $700 for the single SSD. Luckily, with USBC/TB4, these enclosures come with the controllers built-in instead of requiring PCIe board

There could be bits of plain text data left over in over-provisioned storage set up in case a block goes bad (e.g. a flash cell that can no longer erase).

Would that extra cell be used unless there is a need for it? If there is a need for it, then I assume it will be a need for it when you fill the drive too.

But yeah, it depends on the manufacturer of the drive, so you don't want to take the chance, Full encryption is unbeatable.

Flash devices maximize the longevity of the cells by wear leveling. The on-board controller maintains a map between the physical cell layout and the logical layout that's changing with every single write operation.

Extra cells are used both reactively and proactively.

Reactively: Error when erasing or write/verifying a cell? Copy contents to spare cell, flag old cell as bad, update the map, maybe complain if running low on spare cells but otherwise call it a day.

Proactively: Cells are only rated for so many erase operations which directly informs the longevity (total bytes written) of a drive. MLC cells can be written to 3,000 times and SLC 20,000 but the latter is much more expensive. For use cases where greater longevity is required (without spending SLC money), why not put a disk together that has twice as many cells as it needs and wear level across all of them? If you `dd if=/dev/zero of=/dev/sdX bs=64k status=progress` your 32GB drive and it'll complete after 32GB but you have no way of knowing how many more cells that drive has that you didn't touch.

(Also as one of the parents pointed out, disk controllers are getting smart. You don't know if it was transparently compressing data too causing you to touch even fewer cells. ATA Secure Erase is the way to go.)

I’ve always wondered this, instead of writing many passes of gibberish data that takes many hours, could I simply format the drive in seconds, encrypt the whole blank drive with a long generated password, then format and encrypt a second time in seconds and be done?

Or if I’m feeling spicy, do all this then hit it with a hammer until it’s shattered in a million pieces like my dreams of working in tech.

If the first encrption would encrypt the entire drive, then you’ve effectively done a 1-pass erase. It might work, but I would still prefer specialized software — perhaps the encryption method of choice would still leave some data unencrypted or untouched. The hammer method also sounds fine to me.

Not true of older Android phones, however.

If the crypto is strong, wouldnt the leftovers be equivalent to not overwriting?

The only better is less wear on the disk

I’ve read opsec material that no matter how many passes you do it can still be read. Hard to know for sure unless you know the stack in and out like the back of your hand.

1) Modern research suggests that recovering data which has been overwritten once on magnetic media is probably not feasible due to the high density of modern magnetic storage devices and the use of dynamically aligning heads (e.g. that follow track geometry as they read) instead of absolutely positioned heads (stepper motors) that allow for more error in alignment.

2) Nonetheless, there is an appreciable risk of data remaining on the device due to non-volatile caches, remapped disk areas, and other factors that are not always well understood or disclosed by the manufacturer. Manufacturers have also been found to be unreliable in their implementation of ATA security features (e.g. embedded secure erase). As a result, with few exceptions it is U. S. government policy to permanently destroy all storage devices rather than trusting any kind of secure erasure. The typical NSA-approved method is either to degauss (surprisingly tricky to do right) and then crush, or reduce to 2mm particles with a device resembling a large, terrifying blender.

> As a result, with few exceptions it is U. S. government policy to permanently destroy

And as a result, (probably) Iron Mountain makes bucketloads because we don't just encrypt the drives (let's face it, that would be perfectly acceptable for 99% of US government computers - maybe Top Secret stuff deserves proper destruction).

> I’ve read opsec material that no matter how many passes you do it can still be read.

If that were true, we'd have infinite storage capacity.

Giving some charity to the claim, I'm assuming it meant on devices you plan to dispose of.

I don't see how throwing it away improves storage capacity in a way that would invalidate GP's comment

The assumption isn't infinite capacity. It is recovery of the last data.

Is that fundamentally different from the question of whether overwriting data can make it unreadable? If yes then you can eventually make the "last data" unrecoverable by overwriting it. And if no then presumably you can recover anything previously written to the disk, ergo infinite capacity.

Presumably the original comment meant that you can't guarantee the original data is unreadable with any finite number of overwrites, not that you can always read the original data after overwriting. I'm not saying that's true, but it's a logical interpretation that doesn't imply infinite storage.

Even if you could find the original crypto header, You still need to know the password to unlock it. On iphones, the secure enclave will hold the decrypted key while its running but you need to enter the password after a reboot.

Also if someone had the technical capability to do this. I would be infinitely more concerned about them having an exploit which allows them to hack my next phone and grab all the data while its running.

A lot of that opsec material is very dated back from when density was lower and you had a spinning disk. Do you have any that’s evidence-based that such techniques are needed for flash storage? If anything the opsec concerns are different. There’s very little that would force the controller to actually perform erase of specific blocks (eg the flash could blacklist blocks due to errors). To my knowledge that’s the primary concern, not that the data is still somehow recoverable once erased (which used to be true but I’m not sure holds for modern storage).

That’s probably true but the GP is likely drawing the distinction that iOS tries to do an “unrecoverable” delete as best it can whereas presumably the Echo is not doing this (encrypting the data and then deleting the key as part of the reset).

this does not apply to cryptoshredding or any modern storage media. even a single pass on magnetic media renders data unrecoverable.