Hi,

Lustre is more scalable and performs better than NFS.

But it is mostly used in HPC/BigData context where you have lots of concurrent I/O, which is also bulky. Lustre is scalable because you'd have separate data and metadata storage and can scale to several data servers and storage targets. That means to get most of Lustre you'd need to get several servers and a storage array ; usually the ones used in HPC are expensive. DDN I think would sell you 1/2 PB in 1/2 of rack space for something like 1/2M$ (Not that I have a direct exact quote from them). http://www.ddn.com/products/storage-platform-sfa12kx/ The DDN comes also with HA stuff preconfigured. (Lustre is free you could theoretically build a whitebox solution too; the hardest part is to have HA if you want it. )

Lustre does not like lots of small files at all because it hits metadata servers, although there were some improvements in the recent versions.

For Lustre you'll need Linux kernel modules for the clients (so if your clients are like, OpenBSD then you'd have to re-export LustreFS for them via NFS or something alike, losing most of the benefits of Lustre). Lustre will work over TCP but best speeds/most often happens in HPC that it is used over Infiniband.

GlusterFS is userspace so it is slower but doesn't need any kernel modules and thus should work (any?)where you got FUSE.

For the homes, will something more like a replicated FS (Andrew, Coda, Inferno) be possibly also? (I have zero experience with any of these).

-- Grigory Shamov Westgrid/ComputeCanada Site Lead University of Manitoba E2-588 EITC Building, (204) 474-9625

From: Adam Thompson <athompson@avant.camailto:athompson@avant.ca> Reply-To: Continuation of Round Table discussion <roundtable@muug.mb.camailto:roundtable@muug.mb.ca> Date: Monday, 20 April, 2015 4:24 PM To: "roundtable@muug.mb.camailto:roundtable@muug.mb.ca" <roundtable@muug.mb.camailto:roundtable@muug.mb.ca> Subject: [RndTbl] Distributed/replicated home directories?

I'm looking for the sweet spot between:

- NFS-mounted home directories, which allow easy central management and are portable across every workstation, but are (typically, and in my case) IOPS-limited,

and

- local home directories, which are tied to the workstation but offer great performance

At this point I'm seriously considering things like GlusterFS, Lustre, CephFS, etc., because the NFS performance sucks so badly for doing things like a) running Chrome, and b) running VirtualBox.

Looking for ideas...

[Avant logo] Adam Thompson Senior Systems Administrator voice: 204.789.9596 x24 | email: athompson@avant.camailto:athompson@avant.ca | web: avant.cahttp://avant.ca/