rsync with multi-direction delete.
Hello, I've started working with rsync and I'm close to getting things the way I want it, but I need a bit of help from an expert to finish the project. I'm rsyncing from a directory on a server to a usb mass-storage device attached to a different computer. I'm running an rsync daemon on the server and I'm able to successfully synchronize the contents of a directory on the server to a directory on the usb device. What I want: when I delete a file or directory on either place it will remove it from the other. What it does now: If I delete a directory or file from the server it removes it from the USB device (ya!) but if I remove a file/directory from the USB device and perform an rsync it replaces the deleted file/directory. Here is the rsync command I'm using: rsync --delete --delete-excluded --recursive -uv quiringm@192.168.1.102:/mnt/hdd/music/podcasts/ /Volumes/MEK/Podcasts/ Thanks in advance for your help, -Montana
Montana Quiring wrote:
I'm rsyncing from a directory on a server to a usb mass-storage device attached to a different computer. I'm running an rsync daemon on the server and I'm able to successfully synchronize the contents of a directory on the server to a directory on the usb device.
I do not think rsync is the proper tool for that. It does not keep any status files of the last rsync so it can not determine that a USB file was deleted. It assumes that the source is the master. I do not have a suggestion. -- Bill
Montana Quiring wrote:
What I want: when I delete a file or directory on either place it will remove it from the other. What it does now: If I delete a directory or file from the server it removes it from the USB device (ya!) but if I remove a file/directory from the USB device and perform an rsync it replaces the deleted file/directory.
Here is the rsync command I'm using: rsync --delete --delete-excluded --recursive -uv quiringm@192.168.1.102:/mnt/hdd/music/podcasts/ /Volumes/MEK/Podcasts/
As Bill already suggested, rsync might not be ideal for this situation, as it assumes a single-master scenario; you're looking for a multi-master architecture. There is one easy way to use rsync - but you have to know in advance which location has the deleted files, and name that as the source. So when sync'ing from server to USB, use the command you mention above, but when syncing from USB to server, reverse the last two arguments. You see immediately that you can only make changes to one location OR the other in between syncs, as any changes made in one location will always be undone by the other. In fact, true two-way synchronization in the manner you're talking about is still a fairly hard problem in the generic case. It's fairly easy if you have some way of keeping a log of all activity to both trees, you can compare timestamps to see whose actions need to be replicated where, but without any a priori knowledge, how is any tool supposed to know the difference between these two scenarios: A. You've added a directory (and some files) to system #1, that don't appear on system #2 yet. B. You've deleted a directory (and some files) on system #2, but they remain on system #1. To a sync tool comparing systems #1 and #2, those two scenarios are indistinguishable. And, so far as I know, there's no DWIM extensions to rsync yet :-) One package that *might* be able to solve your problem with some extra work on your part is XFile (http://www.idiom.com/~zilla/xfiles.html). Good luck with this... -Adam
On Mon, 2007-09-17 at 15:17 -0500, Adam Thompson wrote:
One package that *might* be able to solve your problem with some extra work on your part is XFile (http://www.idiom.com/~zilla/xfiles.html).
Unison may also do what you want (I've never used it though): http://www.cis.upenn.edu/~bcpierce/unison/index.html Peter
On 17 Sep, Peter O'Gorman wrote:
On Mon, 2007-09-17 at 15:17 -0500, Adam Thompson wrote:
One package that *might* be able to solve your problem with some extra work on your part is XFile (http://www.idiom.com/~zilla/xfiles.html).
Unison may also do what you want (I've never used it though): http://www.cis.upenn.edu/~bcpierce/unison/index.html
I've used unison in a production environment for over a year to do a 2-way sync and it has worked great. Zero problems. A real hassle to compile though, as it uses some wacky language, meaning you have to compile the compiler! (On FC5 at least.)
Here's another interesting looking multifunction synchronization solution: http://www.conduit-project.org/ Intro video: http://video.google.com/videoplay?docid=-262844619875208739&hl=en I haven't tried it myself but it claims bi-direction sync to and from many different sources such as PC to PC. Hope it helps. Kelly On 9/27/07, Trevor Cordes <trevor@tecnopolis.ca> wrote:
On 17 Sep, Peter O'Gorman wrote:
On Mon, 2007-09-17 at 15:17 -0500, Adam Thompson wrote:
One package that *might* be able to solve your problem with some extra work on your part is XFile (http://www.idiom.com/~zilla/xfiles.html).
Unison may also do what you want (I've never used it though): http://www.cis.upenn.edu/~bcpierce/unison/index.html
I've used unison in a production environment for over a year to do a 2-way sync and it has worked great. Zero problems. A real hassle to compile though, as it uses some wacky language, meaning you have to compile the compiler! (On FC5 at least.)
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
Multi-master with rsync is well... there be dragons. At least this is a very easy way to drive yourself insane. I speak from experience here; for another PoV ask SW the pains this can cause, especially once the data set gets huge and the file system comes really busy. There is an 'methodology' hack which can work quite well but it isn't a true multi-master... think of it as a segregated master and it won't work in this situation. I would really give up trying to make rsync do multi-master... seriously that way leads to madness and lost data (without decent locking there will be race conditions and difference/delta timing/ transaction problems). I would suggest re-thinking your work flow to move towards a single master solution and just doing very periodic backups of the master (think revision control systems). You could also try rsnapshot (check google) but that's shoe horning and still requires single master. On 17-Sep-07, at 3:17 PM, Adam Thompson wrote:
Montana Quiring wrote:
What I want: when I delete a file or directory on either place it will remove it from the other. What it does now: If I delete a directory or file from the server it removes it from the USB device (ya!) but if I remove a file/directory from the USB device and perform an rsync it replaces the deleted file/directory.
Here is the rsync command I'm using: rsync --delete --delete-excluded --recursive -uv quiringm@192.168.1.102:/mnt/hdd/music/podcasts/ /Volumes/MEK/ Podcasts/
As Bill already suggested, rsync might not be ideal for this situation, as it assumes a single-master scenario; you're looking for a multi-master architecture.
There is one easy way to use rsync - but you have to know in advance which location has the deleted files, and name that as the source. So when sync'ing from server to USB, use the command you mention above, but when syncing from USB to server, reverse the last two arguments. You see immediately that you can only make changes to one location OR the other in between syncs, as any changes made in one location will always be undone by the other.
In fact, true two-way synchronization in the manner you're talking about is still a fairly hard problem in the generic case. It's fairly easy if you have some way of keeping a log of all activity to both trees, you can compare timestamps to see whose actions need to be replicated where, but without any a priori knowledge, how is any tool supposed to know the difference between these two scenarios:
A. You've added a directory (and some files) to system #1, that don't appear on system #2 yet.
B. You've deleted a directory (and some files) on system #2, but they remain on system #1.
To a sync tool comparing systems #1 and #2, those two scenarios are indistinguishable. And, so far as I know, there's no DWIM extensions to rsync yet :-)
One package that *might* be able to solve your problem with some extra work on your part is XFile (http://www.idiom.com/~zilla/ xfiles.html).
Good luck with this...
-Adam
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
-- Sean
OK. Thanks everyone for your input. I don't feel quite so bad for not being able to figure this out in a timely fashion. :) A couple things to note are: -When I delete something on either master I would always want it gone from both. -I would only be modifying files on the master on the server -i would never be adding files to the master on the USB device, I would only be deleting files. I guess I can handle having one master. I'll just always delete the files from the server when I want to get rid of them. Not a big deal. I'll look into Union too. -Montana On 9/17/07, Sean Cody <sean@tinfoilhat.ca> wrote:
Multi-master with rsync is well... there be dragons. At least this is a very easy way to drive yourself insane.
I speak from experience here; for another PoV ask SW the pains this can cause, especially once the data set gets huge and the file system comes really busy. There is an 'methodology' hack which can work quite well but it isn't a true multi-master... think of it as a segregated master and it won't work in this situation.
I would really give up trying to make rsync do multi-master... seriously that way leads to madness and lost data (without decent locking there will be race conditions and difference/delta timing/ transaction problems). I would suggest re-thinking your work flow to move towards a single master solution and just doing very periodic backups of the master (think revision control systems). You could also try rsnapshot (check google) but that's shoe horning and still requires single master.
On 17-Sep-07, at 3:17 PM, Adam Thompson wrote:
Montana Quiring wrote:
What I want: when I delete a file or directory on either place it will remove it from the other. What it does now: If I delete a directory or file from the server it removes it from the USB device (ya!) but if I remove a file/directory from the USB device and perform an rsync it replaces the deleted file/directory.
Here is the rsync command I'm using: rsync --delete --delete-excluded --recursive -uv quiringm@192.168.1.102:/mnt/hdd/music/podcasts/ /Volumes/MEK/ Podcasts/
As Bill already suggested, rsync might not be ideal for this situation, as it assumes a single-master scenario; you're looking for a multi-master architecture.
There is one easy way to use rsync - but you have to know in advance which location has the deleted files, and name that as the source. So when sync'ing from server to USB, use the command you mention above, but when syncing from USB to server, reverse the last two arguments. You see immediately that you can only make changes to one location OR the other in between syncs, as any changes made in one location will always be undone by the other.
In fact, true two-way synchronization in the manner you're talking about is still a fairly hard problem in the generic case. It's fairly easy if you have some way of keeping a log of all activity to both trees, you can compare timestamps to see whose actions need to be replicated where, but without any a priori knowledge, how is any tool supposed to know the difference between these two scenarios:
A. You've added a directory (and some files) to system #1, that don't appear on system #2 yet.
B. You've deleted a directory (and some files) on system #2, but they remain on system #1.
To a sync tool comparing systems #1 and #2, those two scenarios are indistinguishable. And, so far as I know, there's no DWIM extensions to rsync yet :-)
One package that *might* be able to solve your problem with some extra work on your part is XFile (http://www.idiom.com/~zilla/ xfiles.html).
Good luck with this...
-Adam
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
-- Sean
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
The situation you are in is exactly the way I've been running mine for a few years now. The solution is... There isn't one. I just don't ever delete anything. Seriously. If I really need to delete something I log into the server and the client machine(s) and manually delete the file at the same time. This is reasonable when you have one master and maybe at most a couple clients but any more than that and it's a real problem. I've had it in mind that the solution is to add some logic to the script so that it checks to see if there are any files with a ".deleted" extension and then ssh into the server and delete those files first before doing the rsync. But it's really quite rare that I delete documents and so I've never been motivated to actually implement this. So, just thinking out loud; something along the lines of: for FN in `ls *.deleted` do ssh <remotemachine> rm `basename $FN .deleted` done Then do the rsync. Note; THIS IS NOT REAL CODE but it might inspire you to get something working. John On Mon, 2007-09-17 at 14:16 -0500, Montana Quiring wrote:
Hello,
I've started working with rsync and I'm close to getting things the way I want it, but I need a bit of help from an expert to finish the project.
I'm rsyncing from a directory on a server to a usb mass-storage device attached to a different computer. I'm running an rsync daemon on the server and I'm able to successfully synchronize the contents of a directory on the server to a directory on the usb device.
What I want: when I delete a file or directory on either place it will remove it from the other. What it does now: If I delete a directory or file from the server it removes it from the USB device (ya!) but if I remove a file/directory from the USB device and perform an rsync it replaces the deleted file/directory.
Here is the rsync command I'm using: rsync --delete --delete-excluded --recursive -uv quiringm@192.168.1.102:/mnt/hdd/music/podcasts/ /Volumes/MEK/Podcasts/
Thanks in advance for your help, -Montana _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
participants (8)
-
Adam Thompson -
Bill Reid -
John Lange -
Kelly Leveille -
Montana Quiring -
Peter O'Gorman -
Sean Cody -
Trevor Cordes