Code page, locale, iconv and rsync

Post by **cah** » Fri Jun 26, 2015 6:06 pm

Due to last HD crash at home, I lost some Da-Yi's babyhood pictures and other important files. I thought over and the reason I didn't do regular backup more frequently is due to large amount of pictures, videos file that are big and less touched. In order to run backup, the existing way of backup is to back up everything no matter what. This is very inefficient. I just want to backup updated/modified/new files.

That led me to "rsync".

I had successfully came up with a backup script for Windows to back up to external HD. However, it didn't work well with UNIX servers (Linux or Solaris). Some of the filenames have Chinese characters in them and UNIX servers will see different encodings that is useless and annoying.

After spending some time and opening a ticket with Oracle, I tested the "--iconv" parameter with rsync and it came back with positive result. Not without some challenges of course.

Code: Select all

rsync -aprtv --iconv=utf-8,BIG-5 --protocol=29 /cygdrive/c/users/hsiaoc1/desktop /cygdrive/c/users/hsiaoc1/CAH hsiaoc1@dept02.ols.minimed.com:hsiaoc1-d5

Note 1: --iconv=<local character set>,<remote character set>
Note 2: Remote rsync must support this option --iconv. rsync 2.x does not support. I had to upgrade to rsync 3.0.6.
Note 3: 'iconv -l' lists all known character set encodings.
Note 4: Code page is used on Microsoft Windows. With Taiwan's locale, the code page is 950.
Note 5: iconv for windows () is NOT needed from the experiment I ran on hsiaoc1-d6 workstation.
Note 6: I have to add "--protocol=29" to avoid the following error. rsync 3.1.1 has protocol version 31, 3.0.6 has protocol 30. Not sure why I have to use 29 but it helps (from windows to rsync 3.0.6 RHEL 5.2 server). However, from windows to rsync 3.0.8 protocol version 30, it works fine without the need of "--protocol=29".

Code: Select all

rsync: [sender] write error: Connection reset by peer (104)
rsync error: error allocating core memory buffers (code 22) at io.c(829) [sender=3.1.1]

Note 7: When trying from TC's windows machine to rsync 3.0.8 server, most of the files got transferred as expected but some show permission denied (13) errors. I checked online and found some people suggested to have "--chmod=ug=rwX,o=rxX". Maybe -E, --executability can help too