News for hackers. Stuff that matters.
Re:Syncthing: Syncing All the Things by e3bc54b2 on Friday July 23, 2021 @09:49AM (#27930609)
Syncthing is one of the "fire and forget" kind of programs. It takes some time to setup when doing for first time, but then you pretty much forget its there, and when you need it, you realise it has been chugging along doing its job.

Two things I like about it in particular, are

1. It can and favors syncing over local network. This has a huge cost savings in developing world. Even when you do have massive bandwidth, local network sync still has more throughput and lower latency. My music is shared between devices and if I add a track, it takes less than a second to appear on my other devices.

2. You can set conditions, such that delets can be ignored. Eg. I have a WhatsApp message+media backup going back almost 7 years now, ~65GiB. But I don't need all that on my phone. So I just disable syncing delets on my storage. Now my phone can get away with ~3 GiB of whatsapp data (only because I don't delete very often) and I still have complete backup in case I need it.

3. Absolute rock solid stable. At $DAYJOB we use syncthing to sync multi dozen GiBs of new data every day, total file count in 10s of thousands in about ~300 directories. And in past 5 years we have had about 3 instances where we had something that needed attention, out of which 2 were not syncthing's fault.

I've been using Syncthing for years and have absolutely zero complaints against it. Everyone with 2 or more devices should give it a try, just to see there are better options out there.

Re:Syncthing: Syncing All the Things by api on Friday July 23, 2021 @12:13PM (#27932346)
Tangent but: I sometimes feel like programs that just work like this don't get traction because they don't provoke as much discussion, search volume, etc. A complex mess that needs to be constantly babysat creates a cottage industry around it and gets discussed a lot, but something that works is like the quiet kid who sits in the corner and gets A's and that everyone ignores

Kubernetes is today's poster child for this. The Hashicorp stack can do just about all the same things, but it just works so there's no cottage industry of consultants and no market for as-a-service implementations. Why promote something like that?

Re:Syncthing: Syncing All the Things by e3bc54b2 on Friday July 23, 2021 @10:37PM (#27937954)
> programs that just work like this don't get traction because they don't provoke as much discussion.

Agree very much. That's why I never let go of a chance to enavgelise Syncthing and its OSS ilk in other fields.

Re:Syncthing: Syncing All the Things by 45ure on Friday July 23, 2021 @12:59PM (#27932938)
>2. You can set conditions, such that delets can be ignored. Eg. I have a WhatsApp message+media backup going back almost 7 years now, ~65GiB. But I don't need all that on my phone. So I just disable syncing delets on my storage. Now my phone can get away with ~3 GiB of whatsapp data (only because I don't delete very often) and I still have complete backup in case I need it.

I would like to know, how are you using Syncthing to create local backups for WhatsApp database/msgstore (crypt14) and media files. You also mention 65GiB as your complete backup ─ have you tested it via a restore? If so, how?

Re:Syncthing: Syncing All the Things by cesarb on Friday July 23, 2021 @05:17PM (#27935754)
> I would like to know, how are you using Syncthing to create local backups for WhatsApp database/msgstore (crypt14) and media files.

Not the parent poster, but I also do this; I just configured Syncthing to share the /storage/emulated/0/WhatsApp (aka /sdcard/WhatsApp) directory. WhatsApp stores its daily local backup on the Backups subdirectory of that directory, and the media files are all in the Media subdirectory of that directory.

> have you tested it via a restore? If so, how?

I have actually used it to move all the WhatsApp and Signal data from an old phone to a new phone. Just have Syncthing synchronize these directories (with the same path) on the new phone before installing WhatsApp and Signal, then install and launch WhatsApp and Signal. When first launched, if that directory already contains a backup, both WhatsApp and Signal ask if you want to restore from that backup. Signal then asks you to type a long backup encryption key you should have written down somewhere, while WhatsApp asks its servers for the backup encryption key.

Re:Syncthing: Syncing All the Things by e3bc54b2 on Friday July 23, 2021 @10:41PM (#27937972)
cesarb pretty much described the whole thing.

But to add, the message store the where the texts are stored. I usually keep it all. WhatsApp doesn't care about the media more than you do, so you can delete anything not needed, and now you have full text message history + important media on your new phone.

I have tested it while migrating across 3 devices and half dozen phone factory restores, so I'm fairly confident it still works. There are very ugly restrictions coming in future Android version with scoped storage and whatnot and I'm worried Google is finally going to cut me out. But until then, I keep mine.

Re:Syncthing: Syncing All the Things by swiley on Friday July 23, 2021 @01:57PM (#27933697)
Android used to let you administrate your devices and it was easy to do stuff like this.
Re:Syncthing: Syncing All the Things by seized on Friday July 23, 2021 @10:25PM (#27937897)
Is still is easy to do that. I have Syncthing backing up data from my phone easily and reliably.
Re:Syncthing: Syncing All the Things by NortySpock on Friday July 23, 2021 @10:48AM (#27931288)
> 2. You can set conditions, such that delet[e]s can be ignored.

I was evaluating Syncthing for 1-way, append-only phone photo backup, and my Google searches warned me away from this, talking about "unsupported" and "database corruption."

I assume you have not experienced this, but I guess I should have just read the docs and trusted them, instead of forum posts.

https://docs.syncthing.net/advanced/folder-ignoredelete.html

Re:Syncthing: Syncing All the Things by e3bc54b2 on Friday July 23, 2021 @10:54AM (#27931364)
I have been using that feature for, I don't even remember how long, but definitely since it was marked beta or something. I am not a heavyweight like some other in this thread, but my current local state is 70GB and ~30k files, global state is ~50GB (I need to prune WhatsApp on phone again) and ~10k files. I am no biggie, but its not a slouch to keep all this working in 7 devices, with overlapping directories shared between them, across 2 timezones, over local and public network. Zero complaints is exactly my experience. Syncthing Just Works™
Re:Syncthing: Syncing All the Things by egeozcan on Friday July 23, 2021 @11:02AM (#27931452)
Great, because even 1 complaint there would be too many, wouldn't it?
Re:Syncthing: Syncing All the Things by e3bc54b2 on Friday July 23, 2021 @11:07AM (#27931507)
I agree. And that's my gripe with Dropbox, Drive, OneDrive (the big 3 that I was able to try, no apple devices). Every one of them has shit the bed on me, on more than one occasion. That's after burning my bandwidth and taking away my privacy. Syncthing just wins out every way (as long as problem scope is file sync, not cloud storage or sharing).
Re:Syncthing: Syncing All the Things by egeozcan on Friday July 23, 2021 @11:01AM (#27931441)
That scared me too. I just move what I want to delete from the original sync folder over to another folder, causing the phone data to be deleted, I even wrote a script that "partitions" that old data, also syncing to some random server after being fully encrypted (that server ofc doesn't have decryption keys).
Re:Syncthing: Syncing All the Things by XorNot on Friday July 23, 2021 @11:46AM (#27932026)
In practice you could just turn on Trash File Versioning with infinite retention too.
Re:Syncthing: Syncing All the Things by sbam on Tuesday July 27, 2021 @05:43AM (#27970491)
>I was evaluating Syncthing for 1-way, append-only phone photo backup, and my Google searches warned me away from this, talking about "unsupported" and "database corruption."

Same here. What do you use instead? I haven't been able to find a good open-source multiplatform project for backups/1-way sync.

Re:Syncthing: Syncing All the Things by NortySpock on Tuesday July 27, 2021 @10:40AM (#27972995)
It's not open-source, but I ended up using PhotoSync for myself and my wife to a locked-down samba share on my LAN .

They had a pretty reasonable (~$6) in-app purchase to unlock samba access, and the fact that it's a German GmbH means I'm slightly more confident they're not trying to copy all our photos to a dark data-vault.

Happy user so far, though it took some cross-checking against the samba share feel confident it was backing up reliably (I'm just particular about trust-but-verify :)

https://www.photosync-app.com/home.html

Re:Syncthing: Syncing All the Things by blooalien on Thursday July 29, 2021 @12:18AM (#27992736)
I have several folders in SyncThing set for one-way only sync (from one device or machine to another only, never syncing back). It's easy to set various nifty folder options in the SyncThing web UI and on the Android app. (Dunno about Apple devices, because I own none.)
Re:Syncthing: Syncing All the Things by stewbrew on Friday July 23, 2021 @11:02AM (#27931448)
i had some of these problems with earlier versions. removing the database (after making sure everything is on sync) usually solved the issue. but that's years ago.
Re:Syncthing: Syncing All the Things by orblivion on Friday July 23, 2021 @12:30PM (#27932587)
I can come up with a couple issues related to Syncthing (whether or not it's about Syncthing per se).

1) I like to minimize the number of APT repositories I use. Debian's and Ubuntu's are well behind on Syncthing versions. Every so often (much more rarely than before), they have breaking changes. As such, my phone can no longer sync with my computers. Perhaps the answer is that Debian should just not be packaging it, because they have such a conservative policy. I'm considering downgrading my phone's version, or building the latest version from source. Worst case I may add their APT repo but I'm not eager.

2) At least with the version I've been using, opting for local-only sync isn't so straightforward. It's per-instance, not per-folder. This means that I have to have my home server running it as local-only, even for small things that I wouldn't mind getting synced over the wire. It means that if I want to have a local-only directory shared between my phone and laptop, it has to go through my home server first. Unless there's a better way I don't know about.

Otherwise Syncthing has been pretty great. I like your no-delete thing, I did not consider that. I could definitely use it for photos.

Re:Syncthing: Syncing All the Things by yunohn on Friday July 23, 2021 @12:50PM (#27932808)
> Worst case I may add their APT repo but I'm not eager.

What’s wrong with adding a first party apt repo for syncthing (or other software)?

Re:Syncthing: Syncing All the Things by joshu on Friday July 23, 2021 @12:57PM (#27932906)
supply chain attacks
Re:Syncthing: Syncing All the Things by e3bc54b2 on Friday July 23, 2021 @10:33PM (#27937932)
You can always setup Nix/Guix on your Debian. Granted, its still one extra repository, per se, but it will likely be the only extra repository you'll have to add.
Re:Syncthing: Syncing All the Things by orblivion on Sunday July 25, 2021 @04:58PM (#27952852)
I've been considering Nix anyway. Good point.
Re:Syncthing: Syncing All the Things by jedahan on Friday July 23, 2021 @01:09PM (#27933074)
Just checked the debian issue tracker. Not sure if there is supporting work that could move it along, but you are not alone in wanting the latest release https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983500
Re:Syncthing: Syncing All the Things by orblivion on Friday July 23, 2021 @08:51PM (#27937461)
I think it would go against Debian's philosophy to move fast enough to keep up with Syncthing.
Re:Syncthing: Syncing All the Things by divbzero on Friday July 23, 2021 @01:38PM (#27933440)
What are the “3 instances where we had something that needed attention, out of which 2 were not syncthing's fault”? I’m curious what rare exceptions there are to watch for.
Re:Syncthing: Syncing All the Things by e3bc54b2 on Friday July 23, 2021 @10:47PM (#27938005)
Unfortunately I don't clearly remember because last one was a while ago, but it was Sync conflicts due to external factors. The one where it was Syncthing's fault was years ago and is very likely fixed now.
Re:Syncthing: Syncing All the Things by srinathkrishna on Friday July 23, 2021 @11:40AM (#27931945)
Excellent idea to sync Whatsapp. Do you use any means to read your Whatsapp chat history on other devices - say your laptop?
Re:Syncthing: Syncing All the Things by e3bc54b2 on Friday July 23, 2021 @12:09PM (#27932292)
There are a few applications that can read WhatsApp database. I used one couple years ago to run some analytics on my chat, but don't remember now. I'm fairly certain it is still around and works, though.
Re:Syncthing: Syncing All the Things by Royi on Friday July 23, 2021 @01:08PM (#27933053)
Regarding WhatsApp, How do you merge different database files into one coherent big database?
Re:Syncthing: Syncing All the Things by e3bc54b2 on Friday July 23, 2021 @10:35PM (#27937944)
I.. don't. My days of tinkering directly with WA db are over. These days I just want an easy and reliable backup. After restoring, WhatsApp recognizes the files and takes care of the rest.