99.999999% of my readers can probably ignore this one, but if you are of the small minority who use Rsync and have Mac and Linux computers in your home you’ll want to read this.
I have Plex running on a Raspberry Pi for my music. I have a large mp3 folder. Too large to run Syncthing on it unfortunately, but an occasional rsync is perfectly fine.
I thought it worked fine until I quickly realised it was syncing the same files over and over again. It turns out the Mac and Linux machines I’m using have different ideas about character sets their filenames are stored in. A file with an accented character on the Mac is completely different to one that looks the same on the Linux box.
The solution took a while for me to find but it’s very simple. Rsync has an option named --iconv to convert between character sets!
The solution was embarrassingly simple: Much due to a comment I read when researching the problem, I thought you were supposed to specify the character set in the order of transformation; but it seems as that is not the correct syntax. Rather, one should always use
--iconv=utf-8-mac,utf-8when initialising the rsync from the mac, and always use--iconv=utf-8,utf-8-macwhen initialising the rsync from the linux machine, no matter if I want to sync files from the mac or linux machine.Then it works like magic!
EDIT: Indeed, sometimes, checking the manual page closely is a good thing to do. Here it is, black on white:
--iconv=CONVERT_SPEC
Rsync can convert filenames between character sets using this
option. Using a CONVERT_SPEC of "." tells rsync to look up the
default character-set via the locale setting. Alternately, you
can fully specify what conversion to do by giving a local and a
remote charset separated by a comma in the order
--iconv=LOCAL,REMOTE, e.g. --iconv=utf8,iso88591. This order
ensures that the option will stay the same whether you're push-
ing or pulling files.
This does only work from mac to *nux as the libiconv developer refuces to accept the change which would give all platforms utf-8-mac …
The way he either responds or does not responds to these requests shows that he probably has a personal vendetta towards Apple.
Casper – What do you use for syncing with Windows machines? Many years ago I did rsync between Windows and Linux machines but haven’t done so since. The closest I got to was using Syncthing to sync between a Windows machine (source) and Mac (destination). It mostly worked except it stalled with about 5GB of data refusing to sync.
In the end I used Filezilla to sftp the directories I wanted but it refused to create empty directories. To be extra safe I also copied the files to an NTFS formatted external SSD. I’ll go through them eventually and see what’s missing.
Thank you, your advice is incredibly helpful. I have lived with the problem of rsynch failing to work on music files containing accented characters for some years. I must indeed be one of the .000001% of the readers you mention will be interested in this post!
Just to add, for anyone else’s benefit, that you need to have a recent version of rsynch installed for this to work. The one that comes with any Mac has a pre-installed old version that doesn’t support the –iconv option. Updating Rsynch to a current version solves that problem..