News for hackers. Stuff that matters.
Re:Heavy SSD Writes from Firefox by lighttower on Friday September 23, 2016 @11:17AM (#12565226)
Chrome, on my system, is even more abusive. Watch the size of the .config/google-chrome directory and you'll see that it grows to multi-GB in the profile file.

There is a Linux utility that takes care of all browsers' abuse of your ssd called profile sync daemon, PSD. It's available in the debian repo or [1] for Ubuntu or [2] for source. It uses `overlay` filesystem to direct all writes to ram and only syncs back to disc the deltas every n minutes using rsync. Been using this for years. You can also manually alleviate some of this by setting up a tmpfs and symlink .cache to it.

[1] https://launchpad.net/~graysky/+archive/ubuntu/utils [2] https://github.com/graysky2/profile-sync-daemon

EDIT: Add link, grammar

EDIT2: Add link to source

Re:Heavy SSD Writes from Firefox by lucb1e on Friday September 23, 2016 @12:11PM (#12565657)
Launchpad is such a shitty website, aimed at Ubuntu and only Ubuntu, links to source code or more information are nowhere to be found...

Searching on Github, this seems to be it. Turns out, there's releases for Arch, Debian, etc. and it's even in the repositories. No need to add a ppa. https://github.com/graysky2/profile-sync-daemon

For Debian and co:

    $ apt-cache show profile-sync-daemon
    $ sudo apt-get install profile-sync-daemon
Re:Heavy SSD Writes from Firefox by matt4077 on Friday September 23, 2016 @02:27PM (#12566847)
I checked the Activity Monitor on OS X to see if it's the same and if I should do something. It's quite surprising:

    bird	         11,38 GB   119,8 MB		
    kernel_task	          8,82 GB    23,0 MB
    launchd	          8,46 GB    77,7 MB	
    revisiond	          8,35 GB   105,8 MB	
    Tweetbot	          3,01 GB    50,2 MB	
    Safari	        879,6 MB     10,0 MB	
    mds_stores          726,2 MB    570,1 MB		
    systemstatsd        695,0 MB    775,0 MB
    nsurlstorag         685,3 MB      9,5 MB	
    Google Chrom        681,4 MB    303,5 MB		
    iTerm2	        653,7 MB      1,6 GB	
    cfprefsd	        509,1 MB     45,8 MB	
    Papers 3.4.7        465,9 MB    169,8 MB
    cloudd	        441,3 MB      9,7 MB
    ruby	        224,6 MB      1,7 MB	
    coreduetd	        217,5 MB		
    Reeder	        207,2 MB     28,5 MB 
    apsd	         65,7 MB     11	MB    
    Safari	         55,3 MB     12,6 MB	
I've long gotten used to "bird" doing it's thing (something with the cloud I guess). But how can a twitter client write 3GB (while I'm not even actually using it?)
Re:Heavy SSD Writes from Firefox by bartvk on Saturday September 24, 2016 @02:49AM (#12569754)
You can check out what bird does. This Python script showed me that the Whatsapp desktop app was consuming a couple of gig: https://github.com/bwesterb/blame-bird/blob/master/blame-bir...
Re:Heavy SSD Writes from Firefox by mrsteveman1 on Friday September 23, 2016 @03:02PM (#12567114)
Caching tweets with all media, I would guess. Probably updates in the background all the time.
Re:Heavy SSD Writes from Firefox by fulafel on Friday September 23, 2016 @01:17PM (#12566265)
The link in question was to an Ubuntu PPA on Launchpad, but Launcpad is by no means Ubuntu-only.

See eg https://launchpad.net/openstack, https://launchpad.net/inkscape, https://launchpad.net/shutter etc

Re:Heavy SSD Writes from Firefox by javier2 on Saturday September 24, 2016 @06:33AM (#12570378)
Launchpad is absolutely horrific to navigate and contribute to.
Re:Heavy SSD Writes from Firefox by toxican on Friday September 23, 2016 @12:47PM (#12565989)
As someone who took the plunge into Debian from Ubuntu yesterday, thank you!
Re:Heavy SSD Writes from Firefox by m_rcin on Friday September 23, 2016 @12:41PM (#12565931)
Arch Wiki describes how to do it manually: https://wiki.archlinux.org/index.php/Firefox_on_RAM
Re:Heavy SSD Writes from Firefox by anubisresources on Friday September 23, 2016 @12:48PM (#12565994)
Thank you! I've been looking for this for ages
Re:Heavy SSD Writes from Firefox by brucedawson on Friday September 23, 2016 @09:47PM (#12569068)
> Chrome, on my system, is even more abusive.

The OP is complaining about excessive write traffic, which wastes power, steals disk-time from other applications, and may wear out SSDs prematurely.

A large profile file does not imply excessive write traffic, so it's not clear from your report that Chrome is hitting the same problem at all. Definitely worth watching, but big-file != excessive-writes.

Re:Heavy SSD Writes from Firefox by nchelluri on Friday September 23, 2016 @10:43PM (#12569190)
I think your point is valid, but even so the person you were replying to turned out to be correct.

There's an edit at the end of the article:

> Currently in the middle of a Chrome Version 52.0.2743.116 m test. We have been able to see a pace of over 24GB/ day of writes on this machine...

24GB is higher than the 10-12 he was complaining about from Firefox, even.

Re:Heavy SSD Writes from Firefox by socksy on Friday September 23, 2016 @03:59PM (#12567522)
Been using on my system ever since I started running everything from my USB drive... to prevent excessive freezing twice a minute. On NixOS, add the following two lines to your configuration.nix:

    services.psd.enable = true;
    services.psd.users = ["your-user-name"];
Re:Heavy SSD Writes from Firefox by vizzah on Friday September 23, 2016 @03:11PM (#12567172)
beware! profile-sync-daemon erased my browser profiles upon first run under Ubuntu Xenial due to apparent bug in eCryptfs support (and how /home was mounted). Report was submitted to the developer.
Re:Heavy SSD Writes from Firefox by p4bl0 on Friday September 23, 2016 @12:17PM (#12565717)
Just for info, profile-sync-daemon is available for Jessie only if you have the backport repos.
Re:Heavy SSD Writes from Firefox by pawadu on Friday September 23, 2016 @11:35AM (#12565382)
What is the typical memory usage of this for normal web browsing?
Re:Heavy SSD Writes from Firefox by lighttower on Friday September 23, 2016 @11:47AM (#12565476)
`mount -l` shows a filesystem of type `overlay` for each browser (in my case, chromium, chrome, firefox). Range from 50MB to 200MB. I have it to sync to disc every 15min - so I reckon it empties out the ram whenever it syncs. I've never had a problem (8GB of ram on the machine I initially installed it on) so I haven't watched to see what the max size is. The developer [1] is great - issues are resolved very quickly.

[1] https://github.com/graysky2/profile-sync-daemon

Re:Heavy SSD Writes from Firefox by codemac on Friday September 23, 2016 @01:18PM (#12566276)
Mine uses the following with firefox:

overlaid 2.0G 128M 1.9G 7% /run/user/1000/codemac-firefox-3ckuo4v7.default

Re:Heavy SSD Writes from Firefox by sandGorgon on Friday September 23, 2016 @12:47PM (#12565985)
this is the first I heard of this - why is this not a config flag ?
Re:Heavy SSD Writes from Firefox by gcb0 on Friday September 23, 2016 @05:20PM (#12568042)
because not losing data is infinitely better than saving your ssd life a couple years. we've had ssds for a long time now. still waiting for one to die on me. meanwhile I'm on my 27th dead spinning disk.
Re:Heavy SSD Writes from Firefox by Reason077 on Friday September 23, 2016 @05:36PM (#12568155)
Some people are unjustifiably worried about SSD endurance.

The reality is that even with fairly heavy use, most of them will far outlive the computers they're in. And their owners for that matter!

I purchased my current SSD in March 2015 - it's a Samsung 850 EVO 512GB. It's done 7332 Power_On_Hours, and 36486573259 Total_LBAs_Written, which works out as just under 17 TB written, or about 31.6 GB/day.

That kind of sounds like a lot. But let's put it in context. Even the previous generation of TLC NAND SSDs were recorded in endurance tests doing around 1 PETABYTE of writes before failing. The 850 EVO, with it's 3D V-NAND, should be capable of at least double that.

For arguments sake, lets assume it will last for 1 Petabyte. I've written 17TB in 1.5 years, or 11.3TB per year. At that rate, this drive is still going to last at least another 90 YEARS.

So is this really something to obsess over?

Re:Heavy SSD Writes from Firefox by Retra on Friday September 23, 2016 @07:20PM (#12568614)
>The reality is that even with fairly heavy use, most of them will far outlive the computers they're in.

I'm not sure what the lifetime of a computer is. In my family, we are still occasionally running machines from circa 1995. They work fine. When should I stop using my MacBook? The only reason I'd stop is if it died in a way that can't be cheaply fixed.

Re:Heavy SSD Writes from Firefox by dTal on Saturday September 24, 2016 @12:47PM (#12571591)
I'm curious what useful work such a computer can perform, given that a typical 1995 computer had a 33 MHz processor, 8 megs of RAM and 1 gigabyte of hard disk space.
Re:Heavy SSD Writes from Firefox by Retra on Saturday September 24, 2016 @03:06PM (#12572107)
The same work they did in 1995. Word processing, games, internet. Computers weren't useless back then.
Re:Heavy SSD Writes from Firefox by EpicEng on Friday September 23, 2016 @09:41PM (#12569051)
>I'm not sure what the lifetime of a computer is. In my family, we are still occasionally running machines from circa 1995.

Pretty sure it's not 21 years, c'mon...

Re:Heavy SSD Writes from Firefox by retro64 on Monday September 26, 2016 @11:06AM (#12582425)
1995 is a stretch for anything modern but I have kids playing Minecraft on Pentium 4’s (with decent graphic cards though - that is the critical part). These boxes would have been new around 2003 or so.
Re:Heavy SSD Writes from Firefox by sandGorgon on Saturday September 24, 2016 @12:45AM (#12569493)
this is an incorrect assumption http://techreport.com/review/27909/the-ssd-endurance-experim...

Most of the SSD failed before a petabyte... and started encountering lots of errors before that (forget about the Samsung Pro series which is the only one that survived till its multi-petabyte end).

Re:Heavy SSD Writes from Firefox by Reason077 on Saturday September 24, 2016 @04:37AM (#12570080)
Those were all ~250GB devices, however. Endurance increases with higher capcity.

If a 250GB 840 EVO can get to 800TB without errors, then I'd expect a 500GB 850 EVO to reach 1 PB, easily.

A 500GB 850 EVO has newer-generation, more durable NAND endurance and twice the capacity to work with.

Re:Heavy SSD Writes from Firefox by acqq on Friday September 23, 2016 @07:23PM (#12568625)
The warranty on your EVO is just 150 TBW. The older generations typically had better endurance, the newer generations are more dense (cheaper to produce) but less durable.

If you keep with 17 TB per one and half year your SSD should be OK for your for at least next 11 years.

Re:Heavy SSD Writes from Firefox by Reason077 on Friday September 23, 2016 @08:05PM (#12568780)
The warranty on your EVO is just 150 TBW.

Yes, but endurance tests almost invariably show that the warranty ratings are extremely conservative. The 1TB 850 PRO (which uses exactly the same NAND chips as the 850 EVO, just a different controller) has been endurance tested to more than 7 Petabytes. See: http://packet.company/blog/

The older generations typically had better endurance, the newer generations are more dense (cheaper to produce) but less durable.

That was true when comparing MLC flash to TLC. But the 850 EVO and PRO use 3D V-NAND, which has significantly more endurance than previous-generation TLC. This seems to be confirmed by endurance tests.

your SSD should be OK for your for at least next 11 years

The warranty runs out after 5 years, anyway. But that does not mean it will suddenly stop working on that date!

Re:Heavy SSD Writes from Firefox by alexhawke on Friday September 23, 2016 @09:43PM (#12569060)
No, you're wrong. The 850 PRO uses MLC while the EVO uses TLC. But given that they're both constructed on Samsung's 3D-VNAND technology, their endurance is still much higher than the competitors out there.
Re:Heavy SSD Writes from Firefox by Reason077 on Saturday September 24, 2016 @04:28AM (#12570052)
Yes, you're right. But I don't expect an EVO to last for a full 7 PB of writes like the PRO did. But even if it lasts a fraction of that (say, approx 1 PB, which previous generation 840 EVO was able to achieve) then it's still 90 years of life at current usage levels! A (1 TB) 850 PRO would last over 600 years!

In all likelihood the drive will never get anywhere near any of these values - it'll be replaced with newer, better tech at some point. It may then get used for a backups or secondary storage, but in those scenarios the daily writes will drop enormously.

Re:Heavy SSD Writes from Firefox by acqq on Saturday September 24, 2016 @03:13AM (#12569841)
> their endurance is still much higher

It's not "much" higher, actually, 3D MLC as implemented by Samsung is just "up to twice" better than the planar MLC given the same die area and capacity:

http://www.samsung.com/semiconductor/minisite/ssd/v-nand/tec...

"Samsung V-NAND provides up to twice the endurance of planar NAND."

But if the size of the cell drops, the number of P/E cycles drops. Samsung's endurance declarations are real and to be believed, they initially used bigger cells than some of their other chips (or competitors), but Samsung engineers know what they do, that "some tests" achieved "much more" can be either an accident or due to the errors in the test methodology (and I very much suspect the later, because it also allows the accidents to be taken as the "success"). At the time these SSDs appeared, Samsung declared twice less TBW than they do now, so now the declared endurance is surely not too pessimistic, but based on the real knowledge of what's inside.

Re:Heavy SSD Writes from Firefox by ams6110 on Friday September 23, 2016 @05:36PM (#12568156)
I am one of those people who sets browser to dump all cookies, cache and and history upon closing, so I'm not really sure what data would be lost.
Re:Heavy SSD Writes from Firefox by andybak on Friday September 23, 2016 @06:58PM (#12568532)
Do you ever miss the upside of all that persistence? I find my history useful, I find continually re-logging in a pain (obviously excepting the kind of sites where I don't want my login to persist i.e. financial etc).

However I also only close my browser once every couple of weeks as well. System updates etc.

Re:Heavy SSD Writes from Firefox by gcb0 on Saturday September 24, 2016 @02:37AM (#12569723)
that is the beauty of it. You can disable. Be glad you have firefox and not only IE or chrome (where you can't disable all the things)

But for 99% of the users, they want it.

Heck, anecdote for anecdote: the very first thing i do on new machines is to open up firefox, go to settings, set the browser to startup with my previous session's tabs.

Re:Heavy SSD Writes from Firefox by javier2 on Saturday September 24, 2016 @06:34AM (#12570379)
This is my experience too.
Re:Heavy SSD Writes from Firefox by tedmielczarek on Friday September 23, 2016 @01:51PM (#12566570)
Probably because most people don't want their profile to be broken if their OS crashes or their computer powers off unexpectedly?
Re:Heavy SSD Writes from Firefox by cortesoft on Friday September 23, 2016 @01:54PM (#12566600)
If you miss a sync, your profile isn't going to be 'broken', just out of date.
Re:Heavy SSD Writes from Firefox by spullara on Friday September 23, 2016 @02:35PM (#12566910)
if it happens during the sync you could be arbitrirly broken.
Re:Heavy SSD Writes from Firefox by avar on Friday September 23, 2016 @03:50PM (#12567453)
Only if the sync is the equivalent of:

    rsync [...] /dev/shm/.browser ~/.browser
Instead of:

    rsync [...] /dev/shm/.browser ~/.browser.new && ln -sf ~/.browser.new ~/browser
Or some such equivalent, i.e. doing the full sync and then atomically juggling a symlink around.
Re:Heavy SSD Writes from Firefox by wolf550e on Friday September 23, 2016 @04:33PM (#12567727)
that creates many more writes to the ssd.
Re:Heavy SSD Writes from Firefox by waterhouse on Friday September 23, 2016 @05:04PM (#12567930)
Does it? Consider this scheme: maintain two physical directories, one of which is current (and is named by a symlink) and the other of which is old. When doing a sync, you rsync the newest data into the old directory, then update the symlink. It'll probably result in somewhere between 1x and 2x the writes as the non-atomic single-directory scheme, depending on how well adjacent diffs combine. It will also result in twice the space used (barring some very clever filesystem).
Re:Heavy SSD Writes from Firefox by rovr138 on Sunday September 25, 2016 @08:49PM (#12578385)
He should have used --link-dest= on his rsync.

    PREVIOUS_BACKUP=$(cat ./prev_backup)
    NEW_BACKUP=$(date +%F_%T)

    rsync --link-dest=$PREVIOUS_BACKUP /path/source /path/dest/$NEW_BACKUP

    echo $NEW_BACKUP > ./prev_backup
This will copy /path/source to /path/dest/$NEW_BACKUP (which is a time stamped folder). But it will take into account the previous backup. If a file hasn't changed, it will create a hard link. If it changed, it will then copy the whole file.

And that's it.

Since it's the name is a time stamp. When you need to restore, just read ./prev_backup or just list the directory content, sort it and read the last entry.

Re:Heavy SSD Writes from Firefox by derefr on Friday September 23, 2016 @04:06PM (#12567558)
This isn't really the sync daemon's fault; it's Linux's (or rather, ext4's, and the Linux VFS ABI's) for not supporting multi-inode filesystem transactions. NTFS has them; APFS will have them. Linux should add them too.
Re:Heavy SSD Writes from Firefox by sitkack on Friday September 23, 2016 @05:08PM (#12567953)
The sync daemon could sync with hardlink and then rename which is atomic.
Re:Heavy SSD Writes from Firefox by Filligree on Friday September 23, 2016 @05:37PM (#12568160)
That would require rewriting the entire directory.
Re:Heavy SSD Writes from Firefox by sitkack on Friday September 23, 2016 @05:59PM (#12568279)
hardlinks for the stuff not changed. Browsers of course, should move to append only file formats.
Re:Heavy SSD Writes from Firefox by JadeNB on Friday September 23, 2016 @03:11PM (#12567171)
> There is a Linux utility that takes care of all browsers' abuse of your ssd called profile sync daemon, PSD.

I guess doing something like this on Windows is too much to hope for, but does anyone know how to do this sort of thing on the Mac?

Re:Heavy SSD Writes from Firefox by nicklaf on Friday September 23, 2016 @04:11PM (#12567590)
I would be very interested to see how Chomebooks fare here.
Re:Heavy SSD Writes from Firefox by d0vs on Saturday September 24, 2016 @04:18PM (#12572463)
Anyone actually noticing a speedup from using this?
Re:Heavy SSD Writes from Firefox by tychuz on Friday September 23, 2016 @06:12PM (#12568323)
Anything similar for macOS?
Re:Heavy SSD Writes from Firefox by smegel on Saturday September 24, 2016 @06:58AM (#12570434)
How do we switch it off on Chrome (Windows)?
Re:Heavy SSD Writes from Firefox by Freestyler_3 on Saturday September 24, 2016 @03:49PM (#12572320)
I have been wondering and asked the same for chrome based browser... did not find anything so far.