Upgrading Endeavour

Upgrade main system

Started with a sideways migration to Centos 5. This worked except for the infiniband packages, which were skipped:

• Bump Endeavour over to CENTOS 5: see http://wiki.centos.org/HowTos/MigrationGuide
  • Execute: rpm -ivh http://mirror.centos.org/centos/5/os/i386/CentOS/centos-release-5-11.el5.centos.i386.rpm http://mirror.centos.org/centos/5/os/i386/CentOS/centos-release-notes-5.11-0.i386.rpm http://mirror.centos.org/centos/5/os/x86_64/CentOS/centos-release-5-11.el5.centos.x86_64.rpm http://mirror.centos.org/centos/5/os/x86_64/CentOS/centos-release-notes-5.11-0.x86_64.rpm http://mirror.centos.org/centos/5/os/x86_64/CentOS/redhat-logos-4.9.99-11.el5.centos.noarch.rpm
  • Execute: yum update --skip-broken
  • Remove the broken packages by hand. Unfortunately, this took down the system :-)
    • rpm -e --allmatches ibsim ibutils ibutils-libs infiniband-diags libibcommon libibcommon-devel libibcommon-static libibmad libibmad-devel libibmad-static libibumad libibumad-devel libibumad-static opensm opensm-devel opensm-libs opensm-static perftest srptools mvapich_gcc mvapich2_gcc

New system @ Centos 6.6:

• Reconfigured RAID. All the 2TB drives are now in slots 1-9 and configured for a RAID6, 14TB raid.
  • Slots 10,11,12 will be hot-spare, and 2x passthrough.
    • The passthrough are for: slot11 - Can contain Home directory drive when Gourd is being upgraded. slot12 -- OldSys a 1TB drive with the old Centos 5.5 system.
  • There are 3 volumes on the RAID: "system" ~ 100GB, "system2" ~100GB, "data1"
  • Remaining 12 slots will be filled with high density new drives for another RAID6
• Restarted web server.
• Setup fail2ban

Installing new RPMs on Nodes

• The Centos DVDs are installed at /net/data/node10/RHEL/Centos-6.6
• This dir is added to the c6-media in /etc/yum.repos.d/CentOS-Media.repo
• Install packages with: yum --disablerepo \* --enablerepo c6-media install package_name

More recently, Endeavour is now a proxy server and yum on the nodes is setup to make use of the proxy. Thus "yum update" simply works.

Upgrading Endeavour nodes

Upgrade to Centos 6 started March 17, 2015 with node2:

• Reboot Node2 from a USB key with Centos6 distribution installed. Chose "minimal install"
  • Note: Should have added scp, i.e openssh-client stuff. Added this "by hand" by using from endeavour: cat openssh-client-... | ssh node2 "cat - > openssh-client.rpm" and then installing that rpm.
• SSH into the system
• Copy the Centos ISO to node2 with scp. Mount on /mnt/centos
• Install packages with: yum --disablerepo \* --enablerepo c6-media install
• List of old package installed are in ~root/new_packages.txt with the distribution and package version stripped already. From this list, the packages were filtered into "installed" and "available" with yum. From the resulting list of "available" only the x86_64 and noarch packages were installed.
• A number of config tweaks needed.
• Nodes @ Centos 6.6
  • nodes: 2,3,7, 11, 13

Done

• Initial installation of system
• Update the ethernet configuration (eth0) to 10.0.0.2 and onboot=yes.
• Install all previously installed packages
• setup ssh for passwordless entry. See: http://itg.chem.indiana.edu/inc/wiki/software/openssh/189.html or http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Host-based_Authentication
  • Note: Endeavour identifies as master.farm.physics.unh.edu, and so this needs to be in the ssh_known_hosts file and in the shosts.equiv file and .shosts file.
• Configure authentication: see SSSD
• Synchronise clock with endeavour: rdate -s endeavour && hwclock --systohc
• Configure automount: Frankly, it is a mystery why it works, since the setup seems incomplete, but hey, I won't complain.
• Configure Infiniband
  • See: http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-4.html includes a number of tests. All passed.
  • See: https://pkg-ofed.alioth.debian.org/howto/infiniband-howto-5.html -- tested OK.
  • See: https://access.redhat.com/solutions/301643
  • Note: The ib0 does not seem to come up automatically? Needs to be checked.
• Disk cloning: Run /sbin/Node_Clone.sh 2 7 from node2 to clone from node2 to node7, when both disks are in node2. Currently node2 and node3 are setup for cloning.
• Setup Splunk
• PBS scheduler
  • Still need to setup more queues.

Node Cloning Recipe

Caveat Empor -- do not follow these steps blindly. These are notes from the procedure in 2015.

Prepare drive:

1. remove the old partitions

 parted /dev/sdb rm 1
 parted /dev/sdb rm 2 

1. create new partitions:

 parted /dev/sdb mkpart primary 1049kb 525MB
 parted /dev/sdb mkpart primary 525MB 100%

1. Boot Drive:

 mkfs.ext4 /dev/sdb1
 e2label /dev/sdb1 boot
 test -r /mnt/boot || mkdir -p /mnt/tmp_boot
 mount /dev/sdb1 /mnt/tmp_boot
 rsync  -avxHAX --numeric-ids /boot/  /mnt/tmp_boot

1. Check to make sure it is all there.
2. Rename node stuff

 cd /mnt/tmp_boot/grub
 sed 's/node2/node3/g' grub.conf  > tmp.conf && mv grub.conf grub.conf.orig && mv tmp.conf grub.conf
 sed 's/node2/node3/g' menu.lst  > tmp.lst && mv menu.lst menu.lst.orig && mv tmp.lst menu.lst
 cd /
 parted /dev/sdb toggle 1 boot
 umount /dev/sdb1

1. Create Logical volume sets:

  pvcreate /dev/sdb2
  vgcreate vg_node3 /dev/sdb2
 lvcreate -L 8G  -n lv_swap  vg_node3
 lvcreate -L 30G  -n lv_root  vg_node3
 lvcreate -l 100%FREE  -n lv_home  vg_node3
 mkswap -L swap /dev/vg_node3/lv_swap 
 mkfs.ext4 -L root /dev/vg_node3/lv_root
 mkfs.ext4 -L home /dev/vg_node3/lv_home 

1. Copy data:

 cd /
 test -r /mnt/tmp_root || mkdir -p /mnt/tmp_root
 mount /dev/vg_node3/lv_root /mnt/tmp_root
 mkdir  /mnt/tmp_root/proc /mnt/tmp_root/dev /mnt/tmp_root/tmp
 chmod 1777 /mnt/tmp_root/tmp
 chmod  555 /mnt/tmp_root/proc
 mkdir /mnt/tmp_root/cgroup  /mnt/tmp_root/net  /mnt/tmp_root/srv  /mnt/tmp_root/sys /mnt/tmp_root/boot
 rsync  -avxHAX --numeric-ids bin data etc lib lib64 media misc opt root sbin selinux usr var /mnt/tmp_root

1. Prep BOOT for grubbing

 mount /dev/sdb1 /mnt/tmp_root/boot
 mount -o bind /dev /mnt/tmp_root/dev
 mount -o bind /proc /mnt/tmp_root/proc
 chroot /mnt/tmp_root

1. grub-install --recheck /dev/sdb # === Should work but fails

 grub << EOF
 root (hd1,0)
 setup (hd1)
 quit
 EOF

1. Exit chroot

 exit

1. Fixup /etc

 cd /mnt/tmp_root/etc
 rm -f udev/rules.d/70-persistent-net.rule

1. fixup mtab fstab sysconfig/network sysconfig/network-scripts

 sed 's/node2/node3/g;/vg_node3/d’ mtab > tmp.tmp && mv mtab mtab.orig && mv tmp.tmp mtab
 sed 's/node2/node4/g;' fstab > tmp.tmp && mv fstab fstab.orig && mv tmp.tmp fstab
 cd sysconfig
 sed 's/node2/node4/g;' network > tmp.tmp && mv network network.orig && mv tmp.tmp network
 cd network-scripts
 sed 's/10.0.0.2/10.0.0.4/g;/HWADDR/d;/UUID/d' ifcfg-eth0 > tmp.tmp && mv ifcfg-eth0 xx_ifcfg-eth0.orig && mv tmp.tmp ifcfg-eth0
 sed 's/10.1.0.2/10.1.0.4/g;/HWADDR/d;/UUID/d' ifcfg-ib0 > tmp.tmp && mv ifcfg-ib0 xx_ifcfg-ib0.orig && mv tmp.tmp ifcfg-ib0
 cd /
 umount /mnt/tmp_root/boot
 umount /mnt/tmp_root
 echo “ALL DONE - Shutdown the mother node, take out the 2nd hard drive, put it in the destiny node and boot that node. Repeat for next node.”
       

To Do

• Configure MPI
  • Later. I'm not sure anyone is using this right now.
• Reconfigure Ganglia