News for hackers. Stuff that matters.
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by wannacboatmovie on Monday September 02, 2024 @05:33AM (#41423948)
> As usual, I wanted to check if any firmware update is available

Updating HDD firmware is something you do to resolve a very specific problem, not ... just because it's available. People are blind to the fact that updates can and do introduce new bugs.

Re:Upgrading a Toshiba NAS HDD Firmware on Linux by rbanffy on Monday September 02, 2024 @06:01AM (#41424086)
It depends on what is in the changelog. If there are performance or durability improvements, you might want to change the firmware even if you are not facing a significant issue. The downside is the risk to data on the disk - don't do that (or anything, really) on a drive that holds the only copy of something important.

If the drive is being moved to a different array or machine and the data is to be lost anyway, the risk is very low and, if the process is easy (unlike this) it might be a sensible move.

I agree "just because it's new" is a poor justification to risk data.

Re:Upgrading a Toshiba NAS HDD Firmware on Linux by naming_the_user on Monday September 02, 2024 @08:03AM (#41424743)
Back in the day I'd just wing it, hey, firmware update the only harddrive I own, what could go wrong.

Now if it's not on at least two drives within a few hours of something new (photo, video, work done, whatever) it seems like a terrible risk.

Part getting old, part being able to afford to have redundancy, I guess. Storage feels so much cheaper now.

Re:Upgrading a Toshiba NAS HDD Firmware on Linux by vocram on Monday September 02, 2024 @08:40AM (#41424943)
And in part because you remember all the times YOLO went wrong and you bricked stuff / lost data
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by miah_ on Monday September 02, 2024 @08:50AM (#41424968)
HP has a bootable Linux CD that updates all the firmware on their devices, since the early 00's. We used to regularly update firmware across our entire fleet of servers during maintenance cycles. Never had a failure.
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by Asmod4n on Monday September 02, 2024 @10:47AM (#41425783)
You are speaking about enterprise grade hardware which is meant to be updated. Consumer stuff not as much.
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by worewood on Monday September 02, 2024 @11:27AM (#41426249)
More often than not, the changelog just states "performance improvement" or "bug fix", without any detail on what they've done.

For example, I've been bitten before by Dell due to plundervolt updates that removed the undervolting capability under the umbrella of "security fixes".

Re:Upgrading a Toshiba NAS HDD Firmware on Linux by rbanffy on Monday September 02, 2024 @12:38PM (#41426830)
> I've been bitten before by Dell due to plundervolt updates that removed the undervolting capability

That's enterprise computing for you.

Been bitten by that kind of shenanigan more often than I can remember.

Re:Upgrading a Toshiba NAS HDD Firmware on Linux by HeatrayEnjoyer on Monday September 02, 2024 @01:00PM (#41427018)
Happened to a colleague. Not sure if he ever found out how to rollback to the older version.
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by userbinator on Monday September 02, 2024 @03:53PM (#41428289)
Usually BIOSes can be flashed with the machine off, using a "chip-clip" programmer.
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by throwaway48476 on Monday September 02, 2024 @06:44PM (#41429401)
They can but the data on the chip is a combination of machine specific identifiers, config data, and BIOS code. If you chip flash it you have to get an image from some sketchy place and it will nuke your serial number etc. You can also have problems if your RAM is made by a different manufacturer. Chip flashing is a last resort.
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by rbanffy on Monday September 02, 2024 @03:45PM (#41428239)
Usually there isn't, unless there is a sufficiently serious issue with the update.
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by HeatrayEnjoyer on Tuesday September 03, 2024 @08:37AM (#41434368)
Why wouldn't there be a rollback option, especially on enterprise hardware? If it breaks something you need to revert, and quickly
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by prmoustache on Monday September 02, 2024 @06:16AM (#41424146)
Well it is nice to stay current with the news about your harddrive. Wheren't there are some point some SSD from Intel or Samsung that would eventually brick themselves unless you applied some firmware?

I remember having patched some SSDs for this very reason the last time I worked on on premise bare metal systems.

Re:Upgrading a Toshiba NAS HDD Firmware on Linux by EvanAnderson on Monday September 02, 2024 @06:53AM (#41424315)
The SSD firmware failure mode you're describing was a thing: https://www.cisco.com/c/en/us/support/docs/field-notices/705...
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by justinclift on Monday September 02, 2024 @08:59AM (#41425034)
> Wheren't there are some point some SSD [...] would eventually brick themselves unless you applied some firmware?

Yeah, that was a thing with older enterprise SAS SSDs a few years ago. HP, Dell, Lenovo, Cisco, etc, all affected as they all rebadge the available hardware manufacturer's drives. And several manufacturers have had bricking-firmware problems over the years.

Re:Upgrading a Toshiba NAS HDD Firmware on Linux by numpad0 on Monday September 02, 2024 @10:31AM (#41425655)
Stay current with news is good, stay unproven with firmware isn't.
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by jonhohle on Monday September 02, 2024 @11:25AM (#41426237)
Crucial M4s certainly had one around 2010/2011. They would “fail” after 5,000 hours.
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by eliaspro on Monday September 02, 2024 @06:12PM (#41429197)
It killed a whole fleet of processing unit in our lab back then which had been all put in service at the same time, ran 24/7 and then failed all within the same night.
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by NikkiA on Tuesday September 03, 2024 @01:45AM (#41431670)
A year or two after that I had an M4 that after a certain period of uptime would just stop responding to commands, thus acting like a dead drive until power cycled (a reboot was not enough). There was of course a fw update available and it did fix it, although I don't recall it being listed in the changelog.
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by bustling-noose on Monday September 02, 2024 @08:01AM (#41424722)
according to the release notes 0602 fixed some SMART failures. Seems pretty important to me
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by _joel on Monday September 02, 2024 @06:42AM (#41424263)
Depends if they've read the changelog and saw something useful or not, right?
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by lupusreal on Monday September 02, 2024 @08:33AM (#41424895)
Don't fix it if it ain't broke.
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by TacticalCoder on Monday September 02, 2024 @08:38AM (#41424934)
> Updating HDD firmware is something you do to resolve a very specific problem, not ... just because it's available.

Indeed. As an anecdotical data point: number of personal and servers HDD firmware upgraded over four decades (!)... Zero.

Zero isn't not a lot.

We should ask people who know what it's like to have a lot of HDDs: does BackBlaze upgrade their HDDs' firmwares?

Re:Upgrading a Toshiba NAS HDD Firmware on Linux by dark-star on Monday September 02, 2024 @04:51PM (#41428711)
Here's another datapoint:

number of personal and servers HDD firmware upgraded over three decades: a couple thousand.

Our storage systems at work regularly ship firmware updates with bugfixes. I have never seen a FW update introduce new bugs. But of course there has been the occasional HDD that didn't spin up during the (required) power cycle

Re:Upgrading a Toshiba NAS HDD Firmware on Linux by perching_aix on Monday September 02, 2024 @08:51AM (#41424974)
It's just software like any other. It's vulnerable and it has defects. So it's very reasonable to at least check if there's an update available.

> People are blind to the fact that updates can and do introduce new bugs.

I do agree that this fact is often underappreciated - the single biggest cause of breakages are usually the rollout of changes themselves. But again, it's just like with any other software.

Re:Upgrading a Toshiba NAS HDD Firmware on Linux by kasper93 on Monday September 02, 2024 @09:09AM (#41425097)
> Updating HDD firmware is something you do to resolve a very specific problem, not ... just because it's available.

It is important to check if there is an update and what has been fixed. Like with any software, it may introduce new bugs, but blindly suggesting to "not touch, if it's not broken" is harmful too. Some time ago Samsung rolled out SSDs that were self-destructing after very short period of time and fixed this in firmware. If your SSD breaks or start having problems it is already too late to update, you have to be proactive. And hardware vendors doesn't release firmware updates for nothing, in most cases there is very good reason for that.

Re:Upgrading a Toshiba NAS HDD Firmware on Linux by Dalewyn on Monday September 02, 2024 @10:05AM (#41425476)
>Some time ago Samsung rolled out SSDs that were self-destructing after very short period of time and fixed this in firmware.

If it broke, fix it.

If it ain't broke, don't fix it.

The latter became a rule of life because many people decided to fix what ain't broke and got burned for their troubles.

Re:Upgrading a Toshiba NAS HDD Firmware on Linux by lhamil64 on Monday September 02, 2024 @11:28AM (#41426263)
But how would you know it's broken unless you proactively check for updates?
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by numpad0 on Monday September 02, 2024 @02:44PM (#41427830)
Just checking for updates is fine, actually installing every single firmware versions is bad.

This is because embedded firmware are less moving parts more tightly packed, which makes their failure modes inevitably catastrophic; they're incapable of progressively degrading like Electron apps, but the whole system always spontaneously crash into the wall and die from just one typo, and you don't want that.

Re:Upgrading a Toshiba NAS HDD Firmware on Linux by M95D on Monday September 02, 2024 @01:26PM (#41427209)
Stop encouraging the manufacturers to ship bad firmware that "will fix later".
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by syntheticnature on Monday September 02, 2024 @02:33PM (#41427746)
I think the manufacturers needed no encouragement, and at this point it would take multi-national intervention to get the genie back in the bottle. The poster you replied to simply recognized the situation for what it is: Samsung is still a going concern after releasing SSDs with firmware that would brick them after a relatively short service lifespan.
Re:Upgrading a Toshiba NAS HDD Firmware on Linux by userbinator on Monday September 02, 2024 @03:58PM (#41428318)
That wasn't an actual "fix", it was just a workaround --- the flash they were using was far too leaky and lost its charge very quickly, so they decided to have the firmware constantly rewrite in the background. Even the updated firmware won't help for machines that are powered off for months at a time.

https://forum.acelab.eu.com/viewtopic.php?t=8735

Re:Upgrading a Toshiba NAS HDD Firmware on Linux by Palomides on Monday September 02, 2024 @12:33PM (#41426787)
changelogs are generally useless and manufacturers hate having to release firmware updates, so the mere existence of an update is pretty strong evidence there's something seriously wrong

it's not like web or app dev where you can ship trivial upgrades every day for free

Re:Upgrading a Toshiba NAS HDD Firmware on Linux by dark-star on Monday September 02, 2024 @05:01PM (#41428787)
Case in point: We update the HDDs and SSDs on our storage systems regularly. Every month there's new disk drive firmware coming out for various (OEM) drives.

I have never seen a disk fw update introduce new bugs. Firmware development works quite a bit differently than regular software development. There's usually no new "feature" that can possibly be added to a disk drive. It's not like you're suddenly getting a new desktop environment on it.

But yeah, I have seen a handful of HDDs die during the mandatory power cycle because they wouldn't spin back up. And sometimes, disks fail at a higher rate after the firmware update, but that is not due to introduced bugs but rather because the error detection/reporting functionality has improved and is now reporting issues that it didn't before. For enterprise storage systems that is actually a good thing, because a predicted failure is always better than a sudden failure.

Case in point: A few years ago, the HGST Cobra drives were leaking fluid from the motor onto the platters. There was nothing that could be done, so the firmware was updated to move the heads more (to prevent fluid buildup) and the error detection was changed to detect the slowed head movement and report that via sense codes. Of course that caused more early failures, but that's better than having a handful disks fail all at once, and the disks would have failed anyway at some point.