Over the years I have accumulated a sizable music library (mostly flacs, adding up to a bit less than 1TB) that I now want to reorganize (ie. gradually process with Musicbrainz Picard).
Since the music lives in my NAS, flacs are relatively big and my network speed is 1GB, I insalled on my computer a hdd I had laying around and replicated the whole library there; the idea being to work on local files and the sync them to the NAS.
I setup Syncthing for replication and... everything works, in theory.
In practice, Syncthing loves to rescan the whole library (given how long it takes, it must be reading all the data and computing checksums rather than just scanning the filesystem metadata - why on earth?) and that means my under-powered NAS (Celeron N3150) does nothing but rescanning the same files over and over.
Syncthing by default rescans directories every hour (again, why on earth?), but it still seem to rescan a whole lot even after I have set rescanIntervalS to 90 days (maybe it rescans once regardless when restarted?).
Anyway, I am looking into alternatives.
Are there any you would recommend? (FOSS please)
Notes:
- I know I could just schedule a periodic rsync from my PC to the NAS, but I would prefer a bidirectional solution if possible (rsync is gonna be the last resort)
- I read about unison, but I also read that it's not great with big jobs and that it too scans a lot
- The disks on my NAS go to sleep after 10 minutes idle time and if possible I would prefer not waking them up all the time (which would most probably happen if I scheduled a periodic rsync job - the NAS has RAM to spare, but there's no guarantee it'll keep in cache all the data rsync needs)
I suspect it's somewhat inevitable, since in order to sync you need to know what's the difference between files here and there (unless using smth like zfs send which should be able to send only the changes, I guess?). I'd probably tag everything at once and then sync
If a file has not been modified, why does it need to be scanned?
That's if you don't keep track of whether it was modified. It comes more or less for free if you're the filesystem, but may be more complicated for external programs. Although, ?maybe inotifywait can track for changes to the whole directory, but I'm not sure here
Isn't there a last modified time stamp on files?
Huh, didn't think of that 😅
I'm not an expert in that, But I also came to comment about something like zfs. I'm currently using btrfs and it works great for backup. The idea is that the filesystem itself contains the information what changed as far as I understand. And therefore the send operation are very lightway. As they are so lightway it would probbably be possible to also just do manuel sync(e.g. when you finished working for the day. However I can currently not think of abdirectional way where you have changes at both places and then merge them.