Move Mail RAID

Move the Mail RAID drive from Endeavour back to Gourd

OK, this can be done relatively risk free, but the order of the steps are important.

1. Prep the receiving system, in this case Gourd.
   1. Make sure Gourd does not have a degraded RAID, if it does, the new disk will become a hot-spare and immediately get overwritten.
   2. Stop KVM: "systemctl stop libvirtd"
   3. Make sure /kvm /mail /home exist, and are empty directories.
   4. Make sure mdadm works (i.e. is installed.)
   5. Make sure all systems mount from "npghome" and some sysadmin idiot didn't hardwire 10.0.0.100 or another ip. Check /etc/hosts and /etc/fstab
   6. Setup a bridge for the 10.0.0.240 (npghome) IP, called farmbr:1. Don't start it.
   7. Make sure there is an nfs server installed
2. Take down Einstein and Roentgen. Since they both depend on /mail and /home. (Actually Roentgen is optional, Einstein is not.)
3. Stop the npghome (10.0.0.240) on Endeavour
4. Unmount /home, /mail from Endeavour. You may need to kill all users who are accessing the drives (not literally! Their process!)
5. Move the physical drive from Endeavour to Gourd. Yup, on a running system. It's OK. You need to take the drive out of the caddy and mount it in a Gourd caddy.
6. Log into the RAID controller for Gourd.
   1. Delete the newly create "hot spare" (immediately, before it gets used!!!!)
   2. Create a pass-through drive for the newly inserted RAID disk.
7. Manipulate the RAID on Gourd:
   1. Linux on Gourd will recognize the drive and make a /dev/md125 etc out of it. SO:
      1. Stop these raids: mdadm --manage /dev/md125 --stop
      2. Check what the new drive is named: Get the SCSI lun from the RAID http page, map it to the correct drive with: ls -ld /sys/block/sd*/device
      3. If you find the new disk is /dev/sdd, then reassemble:
         1. mdadm --assemble --verbose /dev/md0 /dev/sdd1 --run
         2. mdadm --assemble --verbose /dev/md1 /dev/sdd2 --run
         3. mdadm --assemble --verbose /dev/md2 /dev/sdd3 --run
8. Add the new drives in /etc/fstab and mount them. Check this worked.
9. Move the npghome (10.0.0.240) to the new system: Gourd (i.e. bring up farmbr:1)
10. Export the drives on NFS, i.e. add to /etc/exports and export them. Check that other systems can mount them.
11. Start Einstein.
    1. Immediately log in and
       1. check the system date.
       2. check the /var/log/maillog for anything funny
    2. Check that you are now receiving mail, and the old mail still exists.
    3. Check from other systems that /net/home is available.
12. Now, your moved RAID is still degraded, and the old RAID still runs on Endeavour as well.