[RndTbl] shrink to smaller disk, offsite

[Trevor Cordes]

On 2016-03-31 Adam Thompson wrote:
> 
> On most modern-ish systems with modern-ish versions of rsync, you may 
> want to consider "-vaxHAXES".  Some of those options won't exist on 
> older systems, adjust as necessary.  If you have sparse files and 
> *don't* use -S you're in for a looooooooong copy session.
[...]
> The much bigger, harder problem lies in convincing GRUB to install
> its bootloader correctly on the second volume (and point to the
> second volume for stage1, stage1.5, stage2, /boot, kernel, etc.).  I
> have *once* managed to do this successfully without using a rescue
> CD, and I sure wish I could figure out how I did it!
> The one pointer I can provide is that physically removing the first
> disk before trying to reinstall GRUB from rescue media does make the
> job a heck of a lot easier!

Thanks to all who responded!  Gave me some other ideas to play with.
Combining the brains together gave me.... success!  And no drive to
onsite required!

Here's what I did for posterity:

HOW TO MOVE A RAID-1 SYSTEM FROM A BIGGER DISK SETUP TO A SMALLER DISK
SETUP WITHOUT BEING ONSITE (where you want to replace one of 2 big
disks in a RAID-1 with a smaller disk for whatever reason; which will
require a shrinking/resize of root):

0. New ssd is physically installed a while ago (/dev/sdb).  fdisk
partition to match the original big disk layout (for me: boot root
swap) but of course root (/dev/sdb3) is made smaller.  For boot and
swap, just added the new partitions as normal to the existing arrays:
mdadm --add /dev/mdX /dev/sdbY

1. create new raid device on the ssd's big partition (default
metadata ver is ok)
madam --create -l1 -n2 /dev/md55 /dev/sdb3 missing
(md55 is just any spare md number, it's irrelevant)
Create an ext4 fs on it that new array.

2. initial rsync; "priming" it, which is really handy for making the
final pre-switch rsync much faster:
rsync -axHAXES /.  /mnt/misc/.

3. Root UUID in /etc/fstab.  Use blkid to get the UUID of the old md
array and also the new one.

4. Edit /etc/fstab.  Change the / root fs entry from the old UUID to
the new UUID.  (If you use labels, set it to the new label.  Experience
has taught me UUIDs are the way to go, so I use UUIDs.)

5. Edit /boot/grub2/grub.cfg (location might be different on other
distros).  Search & replace every old UUID to the new UUID.  I wasn't
sure about a few of the matches in the OS menu entry label, but in the
end changing everything seemed to work fine, so why not.

6. Get the md UUIDs:
mdadm -D /dev/mdOLD |grep UUID | tr -d ':'
mdadm -D /dev/mdNEW |grep UUID | tr -d ':'
Edit grub.cfg again and search & replace all the old md-UUIDs to the
new one.

DO NOT REBUILD YOUR grub.cfg with grub2-mkconfig (or other tools) yet!!

7. dracut -f --fstab  (Other distros might use other tools to make the
initramfs.)
Since dracut & systemd do mystical voodoo to handle all the fs's rather
than just plunking /etc/fstab in the initramfs, the --fstab is required
to tell dracut to use your current fstab (with the new UUID) to decide
what voodoo to do on boot, rather than /proc/self/mountinfo.  I have a
hunch dracut/initrd just take their cue from grub, but I'm not certain.

8. turn off nearly every daemon, quiet down system

9. Do final rsync, should be fast:
rsync -axHAXES /.  /mnt/misc/.

10. reboot, cross your fingers, and sweat some bullets

11. Cheer and rejoice that the box came up 2 minutes later!  First try!

12. Make sure things are actually sane.  Let the system do its normal
grub rebuild and dracut build:
grub2-mkconfig -o /boot/grub2/grub.cfg
dracut -f

reboot again, it's all good!  yay!

13. mdadm --stop and disable/hose the old root md array.  Eventually
re-add it to the new root array.  I wouldn't do it immediately, as it
makes a nice backup in case the entire above procedure in fact mangled
everything and I need to revert.

> Use LABEL= in your boot command lines and fstabs, not UUIDs or raw 
> devices nodes for scenarios like this.  At least you can sanely
> control LABELs, and they're easy to change as needed.

I went through a device-path phase, then a label phase, and now am
firmly in the camp that only UUIDs should be used.  Labels bit me when
I'd take a HDD from one of my client systems and load it on another (or
mine) for diagnostics, only to discover they both had the same label,
like "trevorsroot".  Seriously foobar'd booting!!  UUID's are fairly
easy to deal with once you understand everywhere they are referenced.

My biggest complaint of the whole process is that grub.cfg is too
opaque and the docs are horrible.  Like what on
earth is "$menuentry_id_option 'gnulinux-simple-...'" with a copy of
the UUID after gnulinux-simple?  Huh?  Where is this stuff documented?
search root with a boot UUID?  Huh?  Oh well, global search & replace
hammer worked anyway.  Also, I un-cpio'd the initramfs and grep'd the
whole thing for the UUID's referenced in fstab and they aren't anywhere
in there!  Huh?  How is it figuring out what to mount?  Docs?
Fuggitaboutit.  Cross your fingers and pray.  And keep the car keys
handy!