Pumpkin

Pumpkin is our new 8 CPU 24 disk monster machine. It is really, really nice.

Basic Setup

We run Xen on this so that it has two RHEL5 personalies: Pumpkin, 64-bit, and Corn, 32-bit. More information is at our Xen page.

Current Xen domU's:

Domain0

Pumpkin

Red Hat EL5 - 64-bit 8-CPUs 24 GB of memory.

DomU

Corn
Red Hat EL5 - 32-bit 2 CPUs, 4 GB of memory. Para-virtualized, boots from /dev/sdb

DomU

Fermi
Red Hat EL4 - 64-bit 2 CPUs, 2 to 4 GB of memory, Para-virtualized, boots from /dev/sdg.

DomU

Compton
Red Hat EL4 - 32-bit 2 CPUs, 2 to 4 GB of memory, Fully-virtualized (cannot mix and match x86_64 and i686 for RHEL4), boots from /dev/sdh.

DomU

Landau
Red Hat EL4 or EL5 - boots from /dev/sdj. -- System for experimenting.

RAID Setup

The RAID is currently split. This allows for much easier maintenance and, in the future, possible upgrades.

Disk 1 to 11

RAID Set 0, which holds the RAID Volumes: System (300GB, RAID6, SCSI:0.0.0), System1(300GB, RAID6, SCSI:0.0.1), Data1 (6833GB, RAID5, SCSI:0.0.2)

Disk 11 to 22

RAID Set 1, which holds the RAID Volume: Data2 (7499GB, RAID5, SCSI:0.0.3)

Disk 23 and 24

Passthrough (single disks) at SCSI:0.0.6 and SCSI:0.0.7. These can be used as spares, as backup, or to expand the other RAID sets later on. Currently they are seen as /dev/sde* and /dev/sdf*. /dev/sdf and /dev/sde are currently used for Virtual Systems.

The RAID card can be monitored at http://10.0.0.99/ login as "admin" with a password that is the same as the door combo.

• To use this card with Linux you need a driver: arcmsr. This must be part of the initrd for the kernel, else you cannot boot from the RAID. You can also install from the CDs, if you have a driver floppy. It will then add the arcmsr driver into the initrd for you. You will still always need to have this driver!
• The kernel module can be built from the sources located on /dev/sdf in /usr/src/kernels/Acera_RAID. Just run make.

There exists a temporary drive which holds a RHEL5 distro and the original RHEL4 distro from the manufacturer. It is currently disconnected from pumpkin.

Virtual Host: Corn

We run a 32-bit personality "corn" using the Xen virtualization on pumpkin's /dev/sdb. Corn is a para-virtualized RHEL5 system, with pumpkin as the master host, or "domain0". It is a fully separate system (that could be booted as the main system with a few modifications to config files. Hint: don't do that!). This means that any system stuff installed on Pumpkin needs to be installed on corn separately.

Subscription Issue: A virtual host needs to be setup special. See the RedHat documentation. Both host and guest need rhn-virtualization-common and rhn-virtualization-host installed. This is now fixed.

The virtual host needs to have both ethernets bridged. According to Xen wiki, this is done by modifying the /etc/xen/scripts/network-bridge script, which is now network-bridge-two which calls the original twice. For the host, create two interfaces: first the one to xenbr1 and then the one to xenbr0, so that the first one ends up being eth1 and the second one eth0. Yes, it seems backwards, but it now works. The key is to have the lines

alias eth0 xennet
alias eth1 xennet

in the /etc/modprobe.conf file. This is now working.

To Do

• There must be other things....

Done

• • Pumpkin's iptables seem messed up after this morning's (1/8/2008) GRUB trouble. With the old config (pepper's), iptables wouldn't let anything in at all, it seemed (specifically things like pingbacks, LDAP…). I've copied roentgen's /etc/sysconfig/iptables-npg to pumpkin for now, and everything seems to be working as usual. Previously it had a copy of pepper's, and pepper works, so I wonder what the real problem is. This was the wrong iptables!. I fixed it with a new set.
• Sane iptables using ldap.
• Setup ethernet.
• Setup RAID volumes.
• Setup partitions and create file systems.
• Move the system to System drive and remove the current temp drive.
• Setup mount points for the data drives.
• Setup LDAP for users to log in.
• Setup Exports, so other systems can see the drives. There were issues with firewall, so I modeled the firewall after taro's. Seems to be working, I can successfully ls /net/data/pumpkin1 and ls /net/data/pumpkin2 on einstein.
• Setup autofs so that it can see other drives. What other drives? It's working for einstein:/home Other drives such as data drives
• Setup smartd so we will know when a disk is going bad. This can be done inside the RAID card using a system to send SNMP and EMAIL. but it needs to be done. E-mail seems to be set up, let's see if we get any through npg-admins
• Restrict access (/etc/security/access.conf)
• Setup sudo on both pumpkin and corn.
• Add the new systems to the lentil backup script. They're on there; lentil just needs the right SSH keys to rsync them.
• Setup SNMP for cacti monitoring.
• Setup sensors.

Hardware Configuration

• Microway Quote # MWYQ9518-03
• Sales contact: Eliot Eshelman
• Microway 5U 4-Way Opteron Server with up to 24 Drives

• 5U Storage Chassis with 24 SAS/SATA-II Hot-Swap Drive Bays with SATA Multilane Backplane (I think it is a Chenbro case)
• 1350 Watt Hot Swap Redundant Power Supply
• Microway Navion-T (TM) Quad Opteron Motherboard (Tyan S4985):
  • Four sockets for Socket F 8000 series processors
  • Nvidia nForce Pro 2200 + 2050
  • Four banks of memory (16 DIMM slots)
  • Supports up to 64GB of DDR2-667 memory
  • Two x16 PCI Express, Two x4 PCI Express,
  • One PCI 32 bit expansion slots
  • Integrated dual Marvell 88E1111 GbE ports
  • Integrated Intel 82541Pl GbE Port
  • SIS/Xabre Integrated Graphics 16MB
  • Integrated SATA-2 Controller (8 ports)
• 4x AMD Dual Core Socket F Opteron 8222 3.0 GHz, 1 MB Cache / core, 95 watts
• 8x 2GB DDR2 667 MHz ECC/Registered Memory
• 16x 750 GB Seagate Barracuda ES Nearline SATA/300 ST3750640NS 16MB Cache, 3Gb/s, NCQ, 7200rpm, 1.2 million hours MTBF
• Areca ARC-1280 24 port SATA II Raid - PCI Express x8
• Areca ARC-6120 Battery Backup
• 6x Mini-SAS to ML backplane Cable .5M - 4 SATA Drives
• Pioneer DVR-112 Dual Layer DVD/CD writer Internal (Black) 18x write DVD-R/+R, 10x write Dual Layer DVD-R/+R

Removing Xen

1. Install the latest version of kernel that is not xen, you might have to do a yum search kernel to find the specific name of a non-xen kernel and install it.

2. Now remove the kernel-xen with this command and make sure to enter the second one.

 yum groupremove Virtualization
 yum remove kernel-xenU

3. Then restart with the normal non-xen kernel and proceed to do upgrades with yum update command.