Difference between revisions of "Gourd"

From Nuclear Physics Group Documentation Pages
Jump to navigationJump to search
(commented out unused RAID configuration tables.)
 
(58 intermediate revisions by 4 users not shown)
Line 1: Line 1:
−
This is the new Gourd Hardware. The old one is here [[old gourd]]
+
Gourd is a 2 quad-CPU server in a 2U rackmount chassis put together nicely by Microway. It arrived at UNH on 11/24/2009. The system has an Areca [[RAID]] card with ethernet port and an IPMI card with ethernet port. The motherboard is from Super Micro.
  
−
Note: This is the page for the NEW GOURD - 8 core server from Microway, which will serve EINSTEIN as a VM.
+
This is the page for the new Gourd Hardware. The old Gourd is described [[old gourd|here]].
  
−
The previous einstein hardware is described in the previous page for Einstein at [[old einstein]]
+
Gourd now hosts [[Einstein]] as a VM. The previous einstein hardware is described [[old einstein|here]]
  
−
= New Microway Server =
+
[[Image:gourds.jpg|thumb|200px|Gourds]]
  
−
The new einstein is a 2 quad-CPU server in a 2U rackmount chassis put together nicely by Microway. It arrived at UNH on 11/24/2009. The system has an [[Areca]] RAID card with ethernet port and an IPMI card with ethernet port. The motherboard is from Super Micro. Details need to be put here!
+
=Hardware Details=
  
−
* IP address UNH: 132.177.88.75 (currently VLAN)
+
*Microway 2U Storage Chassis with 8 SAS/SATA-II Hot-Swap Drive Bays with SAS Backplane
−
* IP address Farm: 10.0.0.252
+
*500W Redundant Hot-Swap Power Supply (2)
 +
*Navion-S Dual Opteron PCI-Express Motherboard (H8DME-2);
 +
*Two AMD "Shanghai" 2.4 GHz Quad Core Opteron 2378 (Socket F) Processors
 +
*(8) 4GB DDR2 800 MHz ECC/Registered Memory (32GB Total Memory @ 533MHz)
 +
*Dual Integrated Gigabit Ethernet Ports; = "Nvidia" ethernet.
 +
*Integrated ATI ES1000 Graphics;
 +
*Six Integrated SATA-II Ports;
 +
*Two x8 PCI Express slots;
 +
*Two 64-bit 133/100MHz PCI-X slots;
 +
*Two 64-bit 100MHz PCI-X slots;
 +
*Up to 6 USB 1.1/2.0 ports (2 on rear);
 +
*PS/2 Mouse & Keyboard connectors;
 +
*Areca ARC-1220 8-Port SATA II Raid (256MB) (reports as 1680) - Low Profile PCI Express x8 at 10.0.0.152
 +
*SuperMicro AOC-SIM1U IPMI card.
 +
*Sony Slim CD/DVD 24x Black (IDE)
 +
 
 +
= Setup =
 +
 
 +
Details of Gourd [[Upgrading to Centos 7]]
 +
 
 +
= Network Settings =
 +
 
 +
In Centos 7 there is an issue with the drivers for the Nvidia ethernet, which was previously provided through the forcedeth driver. The forcedeth driver which is disabled in the CentOS 7
 +
kernel. You can use the kmod-forcedeth driver from elrepo.org:
 +
 
 +
http://elrepo.org/linux/elrepo/el7/x86_64/RPMS/kmod-forcedeth-0.64-1.el7.elrepo.x86_64.rpm
 +
 
 +
The new udev renaming scheme calls eth0 or FARM network = enp0s8,  and eth1 or UNH network=enp0s9
 +
 
 +
* IP address UNH: 132.177.88.75 (enp0s9)
 +
* IP address Farm: 10.0.0.252 (enp0s8)
 
* IP address RAID: 10.0.0.152
 
* IP address RAID: 10.0.0.152
 
* IP address IPMI:  10.0.0.151
 
* IP address IPMI:  10.0.0.151
  
−
= Setting Up the Server =
+
=Software and Services=
 +
 
 +
This section contains details about the services and software on Gourd and information about their configurations.
 +
 
 +
== Software RAID ==
 +
 
 +
Gourd has a (somewhat simpler) ARECA RAID card. The / system and /scratch are on a RAID created by the card. This was NOT chosen for /home and /mail. Instead /home and /mail are on a software Raid, so that if these services ever need to be moved to other hardware, there is no issues with incompatibility due to Raid card.
 +
 
 +
The config can be found from /etc/mdadm.conf (least reliable!), but better from /proc/mdstat or mdadm --detail --scan, mdadm --detail /dev/md0
 +
 
 +
The drives are passthrough drives on the ARECA card. The association between passthrough drive and device name is established by looking at the SCSI channel number (web interface of the ARECA card) and the Linux /sys:
 +
 
 +
ls -l /sys/bus/scsi/drivers/sd/0\:0\:0\:0/
 +
lrwxrwxrwx 1 root root    0 May 19 21:38 block:sda -> ../../../../../../../block/sda/
 +
 
 +
for each of the 0:0:0:x scsi channels.
 +
 
 +
*  '''See More about the disks below'''
 +
 
 +
==NFS Shares==
 +
 
 +
Gourd serves two volumes over [[NFS]]
 +
 
 +
'''Home folders''': /home on Gourd contains home directories for all npg users. The NFS share is accessible to all hosts in the servers, npg_clients and dept_clients Netgroup lists, and to all 10.0.0.0/24 (server room backend) hosts.
 +
 
 +
'''Mail''': To reduce the size of the size of the Einstein VM the /mail directory on Gourd stores mail for all npg users. The nfs share is accessible only to Einstein and is mounted in /var/spool/mail on Einstein.
 +
 
 +
===/etc/exports===
 +
 
 +
# Share home folders (copied from old Einstein)
 +
 +
/home                                        \
 +
          @servers(rw,sync)  \
 +
          @npg_clients(rw,sync) \
 +
          @dept_clients(rw,sync) \
 +
          10.0.0.0/24(rw,no_root_squash,sync)
 +
 +
# Share /mail with Einstein
 +
 +
/mail \
 +
132.177.88.52(rw,sync) \
 +
10.0.0.248(rw,no_root_squash,sync)
 +
 
 +
 
 +
== IPTables ==
 +
 
 +
Note that Centos 7 (i.e. RHEL 7) comes standard with "firewalld". Not wanting to bother with "yet another config system for firewalls (tm)", this was disabled in favor of good old iptables, which is the underlaying system anyway. This policy may change int he future.
 +
(To disable firewalld: "systemctl stop firewalld ; systemctl mask firewalld'. To setup iptables: "yum install iptables-services; systemctl enable iptables", and then of course, configure the tables.)
 +
Gourd uses the standard NPG [[iptables]] firewall (actually, I copied the endeavour version, with blacklist.) . Gourd allows ssh, svn, icmp, portmap and nfs.
 +
 
 +
A quick, nice tutorial on systemd iptables is here: https://community.rackspace.com/products/f/25/t/4504
  
−
== Important things to remember before this system takes on the identity of Einstein ==
+
==Subversion==
  
−
# The ssh fingerprint of the old einstein needs to be imported.
+
Gourd our subversion code repositories stored in /home/svn/. The subversion service runs under xinetd. Its configuration is located in /etc/xinetd.d/svn
−
# Obviously, all important data needs to be moved: Home Directories, Mail, DNS records, ... (what else?)
 
−
# Fully test functionality before switching!
 
  
−
== Configurations Needed ==
+
===/etc/xinetd.d/svn===
  
−
# RAIDs need to be setup on Areca card.
+
service svn
−
# Mail system needs to be setup
+
{
−
# Webmail needs to be setup. Uses Apache?
+
port        = 3690
−
# DNS needs to be setup.
+
socket_type = stream
−
# Backup needs to be made to work.
+
protocol    = tcp
−
# rhn_config  - I tried this but our subscriptions seem messed up. (Send message to customer support 11/25/09)
+
wait        = no
 +
user        = svn
 +
server      = /usr/bin/svnserve
 +
server_args = -i -r /home/svn
 +
disable    = no
 +
}
  
−
== Initialization ==
+
==VMWare==
  
−
Server arrived 11/24/2009, was unpacked, placed in the rack and booted on 11/25/2009.
+
<s>Gourd is running [[VMWare]] Server version 2.0.2. It acts as our primary virtualization server. it is accessible at https://gourd.unh.edu:8333/ or from localhost:8222 if you're logged in or port forwarding from Gourd over SSH.</s>
  
−
Initial configuration steps are logged here:
+
As of 6/9/2012 Gourd no longer runs VMWare!!!! We use KVM now.
−
* Initial host name is gourd (gourd.unh.edu) with eth0 at 10.0.0.252 and eth0.2 (VLAN) at 132.177.88.75
 
−
* The ARECA card is set at 10.0.0.152. The password is changed to the standard ARECA card password, user is still ADMIN.
 
−
* The IPMI card was configured using the SuperMicro ipmicfg utility. The net address is set to 10.0.0.151. Access is verified by IPMIView from Taro. The grub.conf and inittab lines are changed so that SOL is possible at 19200 baud.
 
−
* The LDAP configuration is copied from Taro. This means it is currently in '''client ldap''' mode, and needs to be change to an '''ldap server''' before production. You can log in as yourself.
 
−
* The autofs configuration is copied from Taro. The /net/home and /net/data directories work.
 
−
* The sudoers is copied from Taro, but it does not appear to work - REASON: pam.d/system-auth
 
−
* Added "auth        sufficient    pam_ldap.so use_first_pass" to /etc/pam.d/system-auth - now sudo works correctly.
 
  
−
== Disks and Raid Configuration ==
+
== KVM ==
  
−
'''Current Disk usage Estimates for Einstein:'''
+
===Guest VMs on Gourd===
−
{| style="wikitable;" border="0"
+
 
−
| Mail (/var/spool):
+
*[[Einstein]] - [[LDAP]] authentication and [[E-mail]]
−
| approx. '''30GB'''
+
*[[Roentgen]] - Websites and [[MySQL]]
−
|-
+
*[[Jalapeno]] - [[DNS]] and [[Printing]]
−
| Home Folders (/home):
+
 
−
| approx. '''122GB'''
+
== UPS Configuration ==
−
|-
+
 
−
| Virtual Machines (/data/VMWare on Taro):&nbsp;
+
Gourd is connected to the APC SmartUPS 2200XL. It uses the apcupsd service to monitor the UPS. Use the following command (with sudo, or as root) to see a detailed list of information about the status of the UPS, including battery charge and current load.
−
| approx. '''70GB'''
+
 
−
|-
+
service apcupsd status
−
| LDAP Database (/var/lib/ldap):
+
 
−
| approx. '''91MB'''
+
===Shutdown Script===
−
|}
+
 
−
<br/>
+
apcupsd allows us to do some other useful things, for example it runs the script /etc/apcupsd/shutdown2 which monitors the current battery charge and shuts down certain non-critical systems at specific points to extend battery life. First, when the battery reaches 50% it shuts down [[taro]], [[pumpkin]], [[tomato]], and [[roentgen]]. Later, when battery is at 5% it shuts down the remaining virtual machines, [[einstein]] and [[jalapeno]]. Shutting down einstein and jalapeno at 5% battery isn't meant to save battery, instead this is designed so that these machines have a chance to shut down normally before the battery backup is completely exhausted. The contents of the script can be viewed [[shutdown2|here]].
 +
 
 +
====SSH Keys====
 +
 
 +
In order to issue remote shutdown commands to other machines gourd needs to issue a command over an ssh connection without a password. It uses an rsa key for this purpose (/root/.ssh/shutdown_id_rsa) and each machine is configured to allow gourd to use this key to issue a remote shutdown command. This key can't be used for shell logins or any other commands.
 +
 
 +
The official Site and Manual
 +
  [http://apcupsd.org/ Official Site]
 +
  [http://nuclear.unh.edu/wiki/pdfs/apcupsd-manual.pdf User Manual]
  
−
Disk sizes in the following tables are based roughly on these current usage estimates with plenty of extra space to grow. They can be adjusted as appropriate to better suit our needs, and to make these designs more cost effective.
+
= Disks and Raid Configuration =
  
−
<hr/>
+
The disks is something that ''should not change'', but alas sometimes it does. Here are the investigation steps to figure out the, probably over complicated, disk setup.
  
−
<!-- Hiding these configuration proposals since we probably won't use them.
+
=== Step 1: Hardware  ===
−
=== Proposed Configuration 1 ===
 
  
−
This configuration is designed to modularize storage and keep related data on separate mirrors. Could be useful if we have a failover system, such as the old Einstein hardware, because individual components ( Mail, home folders, etc ) could be relocated physically to another machine in the event of some failure rather than copying large amounts of data over the network. This design could be modified to use fewer drives by storing Virtual machines either on the /var array or the /home array, opening up the bays to store spare drives.
+
'''Areca RAID Card:''' The hardware raid setup is found by visiting 10.0.0.152 (port 80) with a chrome browser, using admin and Tq....ab password.  
  
 +
'''Disks and Raid configuration'''
 
{| style="wikitable;"  border="1"
 
{| style="wikitable;"  border="1"
−
! Drive Bay
+
! Drive Bay !! Disk Size !! Usage !! Model
−
! Raid Type
 
−
! Contents
 
−
! Volume Size
 
−
! Disk Size
 
 
|-
 
|-
−
| 1
+
| 1 || 750.2 GB || System/Scratch || WDC WD7500AAKS-00RBA0
−
| rowspan="2" | Raid 1
 
−
| rowspan="2" | Operating System ( / )
 
−
| rowspan="2" | 250 GB
 
−
| 250 GB
 
 
|-
 
|-
−
| 2
+
| 2 || 2 TB  || Pass Through || ST2000DM001-1ER164
−
| 250 GB
 
 
|-
 
|-
−
| 3
+
| 3 || 1 TB  || Hot Spare || ST31000340NS
−
| rowspan="2" | Raid 1
 
−
| rowspan="2" | Mail/LDAP ( /var )
 
−
| rowspan="2" | 250 GB
 
−
| 250 GB
 
 
|-
 
|-
−
| 4
+
| 4 || 750.2 GB || Pass Through || ST3750640AS
−
| 250 GB
 
 
|-
 
|-
−
| 5
+
| 5 || 2 TB || Pass Through || ST2000DM001-1ER164
−
| rowspan="2" | Raid 1
 
−
| rowspan="2" | Home Folders
 
−
| rowspan="2" | 500 GB
 
−
| 500 GB
 
 
|-
 
|-
−
| 6
+
| 6 || Empty || None || None
−
| 500 GB
 
 
|-
 
|-
−
| 7
+
| 7 || 500.1 GB || Pass Through || WDC WD5000AAKS-00TMA0
−
| rowspan="2" | Raid 1
 
−
| rowspan="2" | Virtual Machines <br> Data
 
−
| rowspan="2" | 250 GB
 
−
| 250 GB
 
 
|-
 
|-
−
| 8
+
| 8 || 750.2 GB || System/Scratch  || WDC WD7500AAKS-00RBA0
−
| 250 GB
 
 
|}
 
|}
 
<br/>
 
<br/>
−
<hr/>
 
−
 
−
=== Proposed Configuration 2 ===
 
−
 
−
This configuration provides a larger amount of contiguous storage space than the previous design with redundancy provided by either a Raid 6 or Raid 5 array. Raid 5 would provide more usable storage, but Raid 6 will withstand more than one disk failure. It may be preferable to err on the side of caution with our user's data and use a Raid 6 for home folders.
 
  
 +
'''Volume Set and Partition configuration'''
 
{| style="wikitable;"  border="1"
 
{| style="wikitable;"  border="1"
−
! Drive Bay
+
! Raid set!! Devices !! Volume set !! Volume size !! Partitions
−
! Raid Type
 
−
! Contents
 
−
! Volume Size
 
−
! Disk Size
 
−
|-
 
−
| 1
 
−
| rowspan="2" | Raid 1
 
−
| rowspan="2" | Operating System ( / )
 
−
| rowspan="2" | 250 GB
 
−
| 250 GB
 
 
|-
 
|-
−
| 2
+
| Raid Set #000 || Slot2 || 0/0/3 || 2000.4 GB || ????
−
| 250 GB
 
 
|-
 
|-
−
| 3
+
| System/Scratcy || Slot1 + Slot8) || 0/0/0 (System: Raid1) || 250.1 GB || ?????
−
| rowspan="2" | Raid 1
 
−
| rowspan="2" | Mail/LDAP ( /var )
 
−
| rowspan="2" | 250 GB
 
−
| 250 GB
 
 
|-
 
|-
−
| 4
+
|                           ||                        || 0/0/1 (www: Raid1)  || 100.0 GB || ?????
−
| 250 GB
 
 
|-
 
|-
−
| 5
+
|                           ||                       || 0/0/2 (scratch: Raid1) || 399.9 GB || ?????
−
| rowspan="4" | Raid 5 '''or''' <br/> Raid 6
 
−
| rowspan="5" | Home Folders <br/> Virtual Machines <br/> Other data
 
−
| rowspan="4" | 1000 GB (Raid6) <br/> 1500GB(Raid5)
 
−
| 500 GB
 
 
|-
 
|-
−
| 6
+
| Raid Set #002 || Slot 4 || 0/0/4 || 750.2 GB || ????
−
| 500 GB
 
 
|-
 
|-
−
| 7
+
| Raid Set #003|| Slot 5 || 0/0/5 || 2000.4 GB || ?????
−
| 500 GB
 
 
|-
 
|-
−
| 8
+
| Raid Set #005 || Slot 7 || 0/0/7 || 500.1 GB || ????
−
| 500 GB
 
 
|}
 
|}
−
<br/>
 
−
<hr/>
 
  
−
=== Proposed Configuration 3 ===
+
To see how the volume set maps on the host, you can look at "cat /proc/scsi/scsi" which will list the SCSI devices on the system. This confirms that Channel 0, Id: 0, Lun: 0, is indeed "System". Each "run" is mapped to a /dev/s** device. So, /0/0/0/ (System, RAID1) is mapped to /dev/sda, which is divided into 3 partitions: /dev/sda1, /dev/sda2, /dev/sda3, /dev/sda4.
 +
 
 +
To confuse things, further, in the case of /dev/sda, /dev/sda2 and /dev/sda4 are "lvm", so these are part of a logical volume. The advantage of logical volume is that they can be resized and changed on a live, running system. The disadvantage is that there is quite a bit of additional complexity.  More on this below.
 +
 
 +
 
 +
 
 +
=== Step 2: Logical Volumes and Hardware Mapping ===
 +
In modern Linux, the hardware is often mapped, so you need to investigate what piece of hardware ends up to what device. Also, logical volumes are often used to add flexibility.
 +
For details on device mapping see: (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/logical_volume_manager_administration/device_mapper)
 +
 
 +
The command: "dmsetup info" will show the mapped devices. Each device shows up in /dev/mapper.
 +
 
 +
For LVM, you have several layers. The physical (i.e. disks), the group, and the layer.
  
−
This configuration would create one large data store for all user and system data, which could be stored on separate appropriately sized partitions. This design is less modular, but uses fewer drives than previous designs while leaving bays open to store spares in the event of a drive failure.  
+
Display the physical disks in use with "pvdisplay" or "pvs". Currently Gourd has 2 physical disks used for LVM: /dev/sda2 (30GB) and /dev/sda4 (140GB). Each disk is used for the same "centos" volume group.
  
−
{| style="wikitable;" border="1"
+
Display the volume groups with "vgdisplay". There is one "centos", with a VGsize of 167.55 GiB.
−
! Drive Bay
 
−
! Raid Type
 
−
! Contents
 
−
! Volume Size
 
−
! Disk Size
 
−
|-
 
−
| 1
 
−
| rowspan="2" | Raid 1
 
−
| rowspan="2" | Operating System ( / )
 
−
| rowspan="2" | 250 GB
 
−
| 250 GB
 
−
|-
 
−
| 2
 
−
| 250GB
 
−
|-
 
−
| 3
 
−
| rowspan="4" | Raid 6
 
−
| rowspan="4" | Home Folders ( /home ) <br/> Mail/LDAP ( /var ) <br/> Virtual Machines
 
−
| rowspan="4" | 1000 GB
 
−
| 500 GB
 
−
|-
 
−
| 4
 
−
| 500 GB
 
−
|-
 
−
| 5
 
−
| 500 GB
 
−
|-
 
−
| 6
 
−
| 500 GB
 
−
|-
 
−
| 7
 
−
| None
 
−
| Empty / Spare Drive
 
−
| 0
 
−
| 250/500 
 
−
|-
 
−
| 8
 
−
| None
 
−
| Empty / Spare Drive
 
−
| 0
 
−
| 250/500
 
−
|}
 
−
<br/>
 
−
<hr/>
 
−
-->
 
−
=== Proposed Configuration ===
 
  
−
Considering that "large storage" is both dangerous and inflexible, and we really don't want to have a large volume for /home or /var/spool/mail, the following configuration may actually be ideal.
+
Display the logical volumes with "lvdisplay" or "lvs". There are currently 3:
−
We use only RAID1 for the mail storage spaces, so that there is always the option of breaking the RAID and using one of the disks in another server for near instant failover. This needs to be tested for Areca RAID1. We also need to consider that a RAID card has "physical drives", "Volume Set" and "Raid Set". The individual physical drives are grouped into a "Volume Set". This volume set is then partitioned into "Raid Sets", and the Raid Sets are exposed to the operating system as disks. These disks can then (if you insist, as you do for the system) partitioned by the operating system using parted or LVM into partitions, which will hold the filesystems.
 
  
−
We only need 4 of the drive bays to meet our code needs. The other 4 drive bays can hold an additional storage of less critical data, exta VMs, a hot spare, and an empty, which can be filled with a 1 TB drive and configured like a "Time Machine" to automatically backup the /home and VMs to, so that this system no longer depends on Lentil for its core backup. (Just an idea for the future.)
+
    LV  VG    Attr      LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
 +
    root centos -wi-ao---- 50.00g                                                   
 +
    swap centos -wi-ao---- 15.75g                                                   
 +
    tmp centos -wi-ao----  5.00g
  
−
<!-- I broke this table down into two separate tables to make it more readable. If you prefer the original table just uncomment this code - Adam Duston (12/14/09)
+
Each of these logical volumes shows up in /dev/mapper as centos-root centos-swap and centos-tmp.  
  
−
{| style="wikitable;"  border="1"
+
See more at: (https://srobb.net/lvm.htm)l  and for growing the filesystem: (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/logical_volume_manager_administration/lv_extend)
−
! Drive Bay
 
−
! Disk size
 
−
! Raid Set
 
−
! Volume Set
 
−
! Volume Raid Level
 
−
! Volume Size
 
−
! Partitions
 
−
|-
 
−
| 1
 
−
| 250 GB
 
−
| rowspan="2" | Set 1
 
−
| rowspan="2" | Volume 1
 
−
| rowspan="2" | Raid1
 
−
| rowspan="2" | 250 GB
 
−
| rowspan="2" | System: (/, /boot, /var, /tmp, /swap, /usr, /data )
 
−
|-
 
−
| 2
 
−
| > 250GB
 
−
|-
 
−
| rowspan="2" | 3
 
−
| rowspan="2" | 750 GB
 
−
| rowspan="3" | Set 2
 
−
| rowspan="1" | Volume 2
 
−
| rowspan="1" | Raid1
 
−
| rowspan="1" | 500 GB
 
−
| rowspan="1" | Home Dirs: /net/home
 
−
|-
 
−
| rowspan="1" | Volume 3
 
−
| rowspan="1" | Raid1
 
−
| rowspan="1" | 100 GB
 
−
| rowspan="1" | Var: Mail and LDAP
 
−
|-
 
−
| rowspan="1" | 4
 
−
| rowspan="1" | 750 GB
 
−
| rowspan="1" | Volume 4
 
−
| rowspan="1" | Raid1
 
−
| rowspan="1" | 150 GB
 
−
| rowspan="1" | Virtual Machines: Einstein/Roentgen/Corn
 
−
|-
 
−
| 5
 
−
| 750 GB
 
−
| rowspan="2" | Set 3
 
−
| rowspan="1" | Volume 4
 
−
| rowspan="1" | Raid1
 
−
| rowspan="1" | 250 GB
 
−
| rowspan="1" | Additional VM space
 
−
|-
 
−
| 6
 
−
| 750 GB
 
−
| rowspan="1" | Volume 4
 
−
| rowspan="1" | Raid1
 
−
| rowspan="1" | 500 GB
 
−
| rowspan="1" | Additional Data space
 
−
|-
 
−
| 7
 
−
| 750 GB
 
−
| Hot Swap
 
−
|-
 
−
| 8
 
−
| None
 
−
| Empty
 
−
|}
 
−
<br/>
 
−
<hr/>
 
−
-->
 
  
 +
= OLD DISK SETUP =
 +
This was the post-migration Gourd disk setup (as of 03/01/10).
  
 
'''Disks and Raid configuration'''
 
'''Disks and Raid configuration'''
Line 295: Line 231:
 
! Drive Bay !! Disk Size !! Raid Set !! Raid level
 
! Drive Bay !! Disk Size !! Raid Set !! Raid level
 
|-
 
|-
−
| 1 || 250 GB ||rowspan="2"| Set 1 ||rowspan="2"| Raid1
+
| 1 || 750 GB ||rowspan="2"| System/Scratch ||rowspan="2"| Raid1 + Raid0
 
|-
 
|-
−
| 2 || 250 GB
+
| 2 || 750 GB
 
|-
 
|-
−
| 3 || 750 GB ||rowspan="2"| Set 2 ||rowspan="2"| Raid1
+
| 3 || 750 GB || Software RAID || Pass Through
 
|-
 
|-
−
| 4 || 750 GB
+
| 4 || 750 GB || Software RAID || Pass Through
 
|-
 
|-
−
| 5 || 750 GB || rowspan="2"| Set 3 ||rowspan="2"| Raid1
+
| 5 || Empty || None || None
 
|-
 
|-
−
| 6 || 750 GB
+
| 6 || Empty || None || None
 
|-
 
|-
 
| 7 || 750 GB || Hot Swap || None
 
| 7 || 750 GB || Hot Swap || None
 
|-
 
|-
−
| 8 || None || empty || None
+
| 8 || 750 GB || Hot Swap || None
 
|}
 
|}
 
<br/>
 
<br/>
Line 317: Line 253:
 
! Raid set!! Volume set !! Volume size !! Partitions
 
! Raid set!! Volume set !! Volume size !! Partitions
 
|-
 
|-
−
| Set 1 || Volume 1 || 250 GB || System: (/, /boot, /var, /tmp, /swap, /usr, /data )
+
| Set 1 || System(Raid1) || 250 GB || System: (/, /boot, /var, /tmp, /swap, /usr, /data )
−
|-
 
−
|rowspan="3"| Set 2 || Volume 2 || 500 GB || Home Dirs: /home
 
 
|-
 
|-
−
|                     Volume 3 || 100 GB || Var: Mail and LDAP
+
| Set 2 || Scratch(Raid0) || 1000GB || Scratch space (/scratch)
 
|-
 
|-
−
|                     Volume 4 || 150 GB || Virtual Machines: Einstein/Roentgen/Corn
+
| /dev/md0 &nbsp; || sdc1 & sdb1 || 500 GB || Home Dirs: /home
 
|-
 
|-
−
|rowspan="2"| Set 3 || Volume 5 || 250 GB || Additional VM space
+
| /dev/md1 &nbsp; || sdc2 & sdd2 || 100 GB || Mail: /mail
 
|-
 
|-
−
|                     Volume 6 || 500 GB || Additional Data Space
+
| /dev/md2 &nbsp; || sdc3 & sdd3 || 150 GB || Virtual Machines: /vmware
 
|}
 
|}

Latest revision as of 18:11, 9 July 2020

Gourd is a 2 quad-CPU server in a 2U rackmount chassis put together nicely by Microway. It arrived at UNH on 11/24/2009. The system has an Areca RAID card with ethernet port and an IPMI card with ethernet port. The motherboard is from Super Micro.

This is the page for the new Gourd Hardware. The old Gourd is described here.

Gourd now hosts Einstein as a VM. The previous einstein hardware is described here

Gourds

Hardware Details

  • Microway 2U Storage Chassis with 8 SAS/SATA-II Hot-Swap Drive Bays with SAS Backplane
  • 500W Redundant Hot-Swap Power Supply (2)
  • Navion-S Dual Opteron PCI-Express Motherboard (H8DME-2);
  • Two AMD "Shanghai" 2.4 GHz Quad Core Opteron 2378 (Socket F) Processors
  • (8) 4GB DDR2 800 MHz ECC/Registered Memory (32GB Total Memory @ 533MHz)
  • Dual Integrated Gigabit Ethernet Ports; = "Nvidia" ethernet.
  • Integrated ATI ES1000 Graphics;
  • Six Integrated SATA-II Ports;
  • Two x8 PCI Express slots;
  • Two 64-bit 133/100MHz PCI-X slots;
  • Two 64-bit 100MHz PCI-X slots;
  • Up to 6 USB 1.1/2.0 ports (2 on rear);
  • PS/2 Mouse & Keyboard connectors;
  • Areca ARC-1220 8-Port SATA II Raid (256MB) (reports as 1680) - Low Profile PCI Express x8 at 10.0.0.152
  • SuperMicro AOC-SIM1U IPMI card.
  • Sony Slim CD/DVD 24x Black (IDE)

Setup

Details of Gourd Upgrading to Centos 7

Network Settings

In Centos 7 there is an issue with the drivers for the Nvidia ethernet, which was previously provided through the forcedeth driver. The forcedeth driver which is disabled in the CentOS 7 kernel. You can use the kmod-forcedeth driver from elrepo.org:

http://elrepo.org/linux/elrepo/el7/x86_64/RPMS/kmod-forcedeth-0.64-1.el7.elrepo.x86_64.rpm

The new udev renaming scheme calls eth0 or FARM network = enp0s8, and eth1 or UNH network=enp0s9

  • IP address UNH: 132.177.88.75 (enp0s9)
  • IP address Farm: 10.0.0.252 (enp0s8)
  • IP address RAID: 10.0.0.152
  • IP address IPMI: 10.0.0.151

Software and Services

This section contains details about the services and software on Gourd and information about their configurations.

Software RAID

Gourd has a (somewhat simpler) ARECA RAID card. The / system and /scratch are on a RAID created by the card. This was NOT chosen for /home and /mail. Instead /home and /mail are on a software Raid, so that if these services ever need to be moved to other hardware, there is no issues with incompatibility due to Raid card.

The config can be found from /etc/mdadm.conf (least reliable!), but better from /proc/mdstat or mdadm --detail --scan, mdadm --detail /dev/md0

The drives are passthrough drives on the ARECA card. The association between passthrough drive and device name is established by looking at the SCSI channel number (web interface of the ARECA card) and the Linux /sys: 

ls -l /sys/bus/scsi/drivers/sd/0\:0\:0\:0/
lrwxrwxrwx 1 root root    0 May 19 21:38 block:sda -> ../../../../../../../block/sda/

for each of the 0:0:0:x scsi channels.

  • See More about the disks below

NFS Shares

Gourd serves two volumes over NFS

Home folders: /home on Gourd contains home directories for all npg users. The NFS share is accessible to all hosts in the servers, npg_clients and dept_clients Netgroup lists, and to all 10.0.0.0/24 (server room backend) hosts.

Mail: To reduce the size of the size of the Einstein VM the /mail directory on Gourd stores mail for all npg users. The nfs share is accessible only to Einstein and is mounted in /var/spool/mail on Einstein.

/etc/exports

# Share home folders (copied from old Einstein)

/home                                         \
         @servers(rw,sync)   \
         @npg_clients(rw,sync) \
         @dept_clients(rw,sync) \
         10.0.0.0/24(rw,no_root_squash,sync)

# Share /mail with Einstein 

/mail						\
	132.177.88.52(rw,sync)	\
	10.0.0.248(rw,no_root_squash,sync)


IPTables

Note that Centos 7 (i.e. RHEL 7) comes standard with "firewalld". Not wanting to bother with "yet another config system for firewalls (tm)", this was disabled in favor of good old iptables, which is the underlaying system anyway. This policy may change int he future. (To disable firewalld: "systemctl stop firewalld ; systemctl mask firewalld'. To setup iptables: "yum install iptables-services; systemctl enable iptables", and then of course, configure the tables.) Gourd uses the standard NPG iptables firewall (actually, I copied the endeavour version, with blacklist.) . Gourd allows ssh, svn, icmp, portmap and nfs.

A quick, nice tutorial on systemd iptables is here: https://community.rackspace.com/products/f/25/t/4504

Subversion

Gourd our subversion code repositories stored in /home/svn/. The subversion service runs under xinetd. Its configuration is located in /etc/xinetd.d/svn

/etc/xinetd.d/svn

service svn
{ 
	port        = 3690
	socket_type = stream
	protocol    = tcp
	wait        = no
	user        = svn
	server      = /usr/bin/svnserve
	server_args = -i -r /home/svn
	disable     = no
}

VMWare

Gourd is running VMWare Server version 2.0.2. It acts as our primary virtualization server. it is accessible at https://gourd.unh.edu:8333/ or from localhost:8222 if you're logged in or port forwarding from Gourd over SSH.

As of 6/9/2012 Gourd no longer runs VMWare!!!! We use KVM now.

KVM

Guest VMs on Gourd

UPS Configuration

Gourd is connected to the APC SmartUPS 2200XL. It uses the apcupsd service to monitor the UPS. Use the following command (with sudo, or as root) to see a detailed list of information about the status of the UPS, including battery charge and current load. 

service apcupsd status

Shutdown Script

apcupsd allows us to do some other useful things, for example it runs the script /etc/apcupsd/shutdown2 which monitors the current battery charge and shuts down certain non-critical systems at specific points to extend battery life. First, when the battery reaches 50% it shuts down taro, pumpkin, tomato, and roentgen. Later, when battery is at 5% it shuts down the remaining virtual machines, einstein and jalapeno. Shutting down einstein and jalapeno at 5% battery isn't meant to save battery, instead this is designed so that these machines have a chance to shut down normally before the battery backup is completely exhausted. The contents of the script can be viewed here.

SSH Keys

In order to issue remote shutdown commands to other machines gourd needs to issue a command over an ssh connection without a password. It uses an rsa key for this purpose (/root/.ssh/shutdown_id_rsa) and each machine is configured to allow gourd to use this key to issue a remote shutdown command. This key can't be used for shell logins or any other commands.

The official Site and Manual 

 Official Site
 User Manual

Disks and Raid Configuration

The disks is something that should not change, but alas sometimes it does. Here are the investigation steps to figure out the, probably over complicated, disk setup.

Step 1: Hardware

Areca RAID Card: The hardware raid setup is found by visiting 10.0.0.152 (port 80) with a chrome browser, using admin and Tq....ab password.

Disks and Raid configuration

Drive Bay Disk Size Usage Model
1 750.2 GB System/Scratch WDC WD7500AAKS-00RBA0
2 2 TB Pass Through ST2000DM001-1ER164
3 1 TB Hot Spare ST31000340NS
4 750.2 GB Pass Through ST3750640AS
5 2 TB Pass Through ST2000DM001-1ER164
6 Empty None None
7 500.1 GB Pass Through WDC WD5000AAKS-00TMA0
8 750.2 GB System/Scratch WDC WD7500AAKS-00RBA0


Volume Set and Partition configuration

Raid set Devices Volume set Volume size Partitions
Raid Set #000 Slot2 0/0/3 2000.4 GB ????
System/Scratcy Slot1 + Slot8) 0/0/0 (System: Raid1) 250.1 GB ?????
0/0/1 (www: Raid1) 100.0 GB ?????
0/0/2 (scratch: Raid1) 399.9 GB ?????
Raid Set #002 Slot 4 0/0/4 750.2 GB ????
Raid Set #003 Slot 5 0/0/5 2000.4 GB ?????
Raid Set #005 Slot 7 0/0/7 500.1 GB ????

To see how the volume set maps on the host, you can look at "cat /proc/scsi/scsi" which will list the SCSI devices on the system. This confirms that Channel 0, Id: 0, Lun: 0, is indeed "System". Each "run" is mapped to a /dev/s** device. So, /0/0/0/ (System, RAID1) is mapped to /dev/sda, which is divided into 3 partitions: /dev/sda1, /dev/sda2, /dev/sda3, /dev/sda4.

To confuse things, further, in the case of /dev/sda, /dev/sda2 and /dev/sda4 are "lvm", so these are part of a logical volume. The advantage of logical volume is that they can be resized and changed on a live, running system. The disadvantage is that there is quite a bit of additional complexity. More on this below.


Step 2: Logical Volumes and Hardware Mapping

In modern Linux, the hardware is often mapped, so you need to investigate what piece of hardware ends up to what device. Also, logical volumes are often used to add flexibility. For details on device mapping see: (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/logical_volume_manager_administration/device_mapper)

The command: "dmsetup info" will show the mapped devices. Each device shows up in /dev/mapper.

For LVM, you have several layers. The physical (i.e. disks), the group, and the layer.

Display the physical disks in use with "pvdisplay" or "pvs". Currently Gourd has 2 physical disks used for LVM: /dev/sda2 (30GB) and /dev/sda4 (140GB). Each disk is used for the same "centos" volume group.

Display the volume groups with "vgdisplay". There is one "centos", with a VGsize of 167.55 GiB.

Display the logical volumes with "lvdisplay" or "lvs". There are currently 3: 

   LV   VG     Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
   root centos -wi-ao---- 50.00g                                                    
   swap centos -wi-ao---- 15.75g                                                    
   tmp  centos -wi-ao----  5.00g

Each of these logical volumes shows up in /dev/mapper as centos-root centos-swap and centos-tmp.

See more at: (https://srobb.net/lvm.htm)l and for growing the filesystem: (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/logical_volume_manager_administration/lv_extend)

OLD DISK SETUP

This was the post-migration Gourd disk setup (as of 03/01/10).

Disks and Raid configuration

Drive Bay Disk Size Raid Set Raid level
1 750 GB System/Scratch Raid1 + Raid0
2 750 GB
3 750 GB Software RAID Pass Through
4 750 GB Software RAID Pass Through
5 Empty None None
6 Empty None None
7 750 GB Hot Swap None
8 750 GB Hot Swap None


Volume Set and Partition configuration

Raid set Volume set Volume size Partitions
Set 1 System(Raid1) 250 GB System: (/, /boot, /var, /tmp, /swap, /usr, /data )
Set 2 Scratch(Raid0) 1000GB Scratch space (/scratch)
/dev/md0   sdc1 & sdb1 500 GB Home Dirs: /home
/dev/md1   sdc2 & sdd2 100 GB Mail: /mail
/dev/md2   sdc3 & sdd3 150 GB Virtual Machines: /vmware
Retrieved from "https://nuclear.unh.edu/wiki/index.php?title=Gourd&oldid=6547"