Revision as of 02:22, 11 June 2015

Upgrade main system

Started with a sideways migration to Centos 5. This worked except for the infiniband packages, which were skipped:

New system @ Centos 6.6:

Reconfigured RAID. All the 2TB drives are now in slots 1-9 and configured for a RAID6, 14TB raid.
- Slots 10,11,12 will be hot-spare, and 2x passthrough.
  - The passthrough are for: slot11 - Can contain Home directory drive when Gourd is being upgraded. slot12 -- OldSys a 1TB drive with the old Centos 5.5 system.
- There are 3 volumes on the RAID: "system" ~ 100GB, "system2" ~100GB, "data1"
- Remaining 12 slots will be filled with high density new drives for another RAID6
Restarted web server.
Setup fail2ban

The Centos DVDs are installed at /net/data/node10/RHEL/Centos-6.6
This dir is added to the c6-media in /etc/yum.repos.d/CentOS-Media.repo
Install packages with: yum --disablerepo \* --enablerepo c6-media install package_name

Upgrade to Centos 6 started March 17, 2015 with node2:

Reboot Node2 from a USB key with Centos6 distribution installed. Chose "minimal install"
- Note: Should have added scp, i.e openssh-client stuff. Added this "by hand" by using from endeavour: cat openssh-client-... | ssh node2 "cat - > openssh-client.rpm" and then installing that rpm.
SSH into the system
Copy the Centos ISO to node2 with scp. Mount on /mnt/centos
Install packages with: yum --disablerepo \* --enablerepo c6-media install
List of old package installed are in ~root/new_packages.txt with the distribution and package version stripped already. From this list, the packages were filtered into "installed" and "available" with yum. From the resulting list of "available" only the x86_64 and noarch packages were installed.
A number of config tweaks needed.
Nodes @ Centos 6.6
- nodes: 2,3,7, 11, 13

Initial installation of system
Update the ethernet configuration (eth0) to 10.0.0.2 and onboot=yes.
Install all previously installed packages
setup ssh for passwordless entry. See: http://itg.chem.indiana.edu/inc/wiki/software/openssh/189.html or http://en.wikibooks.org/wiki/OpenSSH/Cookbook/Host-based_Authentication
- Note: Endeavour identifies as master.farm.physics.unh.edu, and so this needs to be in the ssh_known_hosts file and in the shosts.equiv file and .shosts file.
Configure authentication: see SSSD
Synchronise clock with endeavour: rdate -s endeavour && hwclock --systohc
Configure automount: Frankly, it is a mystery why it works, since the setup seems incomplete, but hey, I won't complain.
Configure Infiniband
- See: http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-4.html includes a number of tests. All passed.
- See: https://pkg-ofed.alioth.debian.org/howto/infiniband-howto-5.html -- tested OK.
- See: https://access.redhat.com/solutions/301643
- Note: The ib0 does not seem to come up automatically? Needs to be checked.
Disk cloning: Run /sbin/Node_Clone.sh 2 7 from node2 to clone from node2 to node7, when both disks are in node2. Currently node2 and node3 are setup for cloning.
Setup Splunk
PBS scheduler
- Still need to setup more queues.

@@ Line 16: / Line 16: @@
 * Restarted web server.
 * Setup fail2ban
+= Installing new RPMs on Nodes =
+* The Centos DVDs are installed at /net/data/node10/RHEL/Centos-6.6
+* This dir is added to the c6-media in /etc/yum.repos.d/CentOS-Media.repo
+* Install packages with: yum --disablerepo \* --enablerepo c6-media  install package_name
 = Upgrading Endeavour nodes =