Hi All, There exist a threaded and (even MPI-parallel) 'cp' for large/parallel filesystems. http://people.nas.nasa.gov/~kolano/projects/mutil.html It is a bit of pain to build as it patches a particular version of GNU coreutils. -- Grigory Shamov Westgrid/ComputeCanada Site Lead University of Manitoba E2-588 EITC Building, (204) 474-9625
On 2015-02-11 Grigory Shamov wrote:
Hi All,
There exist a threaded and (even MPI-parallel) 'cp' for large/parallel filesystems.
What did it get your cp time down to from 90 days?
Hi, About 6 days it takes, using 2 nodes with 6 cp processes on each of them, using MPI-parallel version of cp and copying from one shared FS (Lustre) to another (NFS). Threaded version of cp (i.e. on one node, several threads) was also tried but the speedup was less than when copying form the two clients. -- Grigory Shamov Westgrid/ComputeCanada Site Lead University of Manitoba E2-588 EITC Building, (204) 474-9625 On 15-02-21 12:47 AM, "Trevor Cordes" <trevor@tecnopolis.ca> wrote:
On 2015-02-11 Grigory Shamov wrote:
Hi All,
There exist a threaded and (even MPI-parallel) 'cp' for large/parallel filesystems.
What did it get your cp time down to from 90 days? _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
participants (2)
-
Grigory Shamov -
Trevor Cordes