Just some observations, tangential ramblings from a recent experience:
For the first time in a long time I ran out of space on my home file
server. It was a 8 x 2TB drive linux software (md) RAID6, 12TB usable,
with a tweaked XFS on top. 99.5% full and running out of things I can
painlessly delete.
Yesterday, I added a 2TB 7200rpm WD SE ("enterprise") drive that is
actually quite affordable. A very solid, heavy drive. Built like a SCSI.
Crossed my fingers and did a md RAID6 grow/reshape, which is quite easy to
do, but took 24 hours 10 mins to complete. And I did it while the fs live
and mounted and nfs exported! (For my sanity I did turn off some
heavy-data daemons.) Hooray, no errors!
Read up on XFS grows and did, while mounted, an xfs_grow -d and
(surprisingly?) it *instantly* returned and df showed the array magically
was 2TB bigger. Hooray #2.
Had to adjust my mdadm.conf (I manually specify some parameters) and
luckily remembered to also do a dracut --force (Fedora system) for next
boot.
Had to adjust fstab to tweak sunit/swidth (s==stripe) to have the remount
use a new, sane, value for performance. I'm a bit confused after reading
the sparse docs and the nearly non-existent googles of people doing
precisely this: md grow + xfs grow. Of course, it's not imperative,
unless you care about performance.
It appears sunit is not changeable unless you increase it by a factor of a
power of 2. For instance, doubling your drives would allow you to double
your sunit. That kind of sucks. Some articles I read said you should set
sunit = RaidChunkSize * 2, which mine is set to. But reading the XFS mans
in more detail it strikes me as incorrect, unless you have only 2
usable-data drievs (ie 4 drive RAID6 or 3 drive RAID5). I'm not sure who
to believe.
Regardless, if my reading of the XFS docs is correct, it looks like my
sunit was never configured properly, and is set to perfectly match 8 data
drives, not my old 6 and certainly not my new 7. I may, in short order,
get a 10th drive for my array, and run perf tests before and after to see
if sunit should indeed be a multiple of usable-drives or not.
swidth is easier. Just make it sunits * usabledrives, and it appears it
can be any valid multiple, changed at will. swidth was set correctly for
my old setup and is now set correctly for my new setup.
Bottom line is, if my new reading of sunit is correct, and it needs to be
tailored to usable-disk-count and can only be changed by factors of 2,
then md+XFS is de facto limited to only ever doubling your usable-disk
count, and NOT allowing of adding 1 disk here, 1 disk there, as I first
thought. *IF* you care about performance, that is. Nothing stops you
from doing any of this, but if your FS striping does not match your RAID
striping you could see insane performance decreases.
Neat:
Filesystem 1K-blocks Used Available Use% Mounted on
before: /dev/md4 11716644864 11499740512 216904352 99% /data
after : /dev/md4 13669767872 11499740704 2170027168 85% /data
Final note: After hearing ZFS (on Linux at least) cannot grow by 1 disk
when using RAID5 or 6, and after nearly 10 years using XFS on md on huge
arrays, I say give some hero cookies to md/XFS. It's withstood some
pretty strange events on my server, and has never blown up. If I'm wrong
about the sunit being a problem, then md/XFS is a great option for those
who want to gradually add space as they need it.