From Nuclear Physics Group Documentation Pages
Revision as of 17:31, 16 January 2008 by Maurik (talk | contribs)
Jump to navigationJump to search


Pumpkin is our new 8 CPU 24 disk monster machine. It is really, really nice.

Basic Setup

We run Xen on this so that it has two RHEL5 personalies: Pumpkin, 64-bit, and Corn, 32-bit. More information is at our Xen page.

Current Xen domU's:


Red Hat EL5 - 64-bit 8-CPUs 24 GB of memory.

Red Hat EL5 - 32-bit 2 CPUs, 4 GB of memory. Para-virtualized, boots from /dev/sdb
Red Hat EL4 - 64-bit 2 CPUs, 1 to 2 GB of memory, Fully virtualized (this may change), boots from /dev/sdf (for now).
Red Hat EL4 - 32-bit 2 CPUs, 1 to 2 GB of memory, Para-virtualized, doesn't boot yet cause I messed up.

The RAID is currently split. This allows for much easier maintenance and, in the future, possible upgrades.

Disk 1 to 11
RAID Set 0, which holds the RAID Volumes: System (300GB, RAID6, SCSI:0.0.0), System1(300GB, RAID6, SCSI:0.0.1), Data1 (6833GB, RAID5, SCSI:0.0.2)
Disk 11 to 22
RAID Set 1, which holds the RAID Volume: Data2 (7499GB, RAID5, SCSI:0.0.3)
Disk 23 and 24
Passthrough (single disks) at SCSI:0.0.6 and SCSI:0.0.7. These can be used as spares, as backup, or to expand the other RAID sets later on. Currently they are seen as /dev/sde* and /dev/sdf*. /dev/sdf1 and /dev/sdf2 hold the old RHEL4 install the system came with.

The RAID card can be monitored at login as "admin" with a password that is the same as the door combo.

  • To use this card with Linux you need a driver: arcmsr. This must be part of the initrd for the kernel, else you cannot boot from the RAID. You can also install from the CDs, if you have a driver floppy. It will then add the arcmsr driver into the initrd for you. You will still always need to have this driver!
  • The kernel module can be built from the sources located on /dev/sdf in /usr/src/kernels/Acera_RAID. Just run make.

There exists a temporary drive which holds a RHEL5 distro and the original RHEL4 distro from the manufacturer. It is currently disconnected from pumpkin. This drive was mirrored to /dev/sdf*, and /dev/sde{1,2} and also has an old RHEL5 distro. We can try to use these as temporary drives for cloning other systems.

Virtual Host: Corn

We run a 32-bit personality "corn" using the Xen virtualization on pumpkin's /dev/sdb. Corn is a para-virtualized RHEL5 system, with pumpkin as the master host, or "domain0". It is a fully separate system (that could be booted as the main system with a few modifications to config files. Hint: don't do that!). This means that any system stuff installed on Pumpkin needs to be installed on corn separately.

Subscription Issue: A virtual host needs to be setup special. See the RedHat documentation. Both host and guest need rhn-virtualization-common and rhn-virtualization-host installed. This is now fixed.

The virtual host needs to have both ethernets bridged. According to Xen wiki, this is done by modifying the /etc/xen/scripts/network-bridge script, which is now network-bridge-two which calls the original twice. For the host, create two interfaces: first the one to xenbr1 and then the one to xenbr0, so that the first one ends up being eth1 and the second one eth0. Yes, it seems backwards, but it now works. The key is to have the lines

alias eth0 xennet
alias eth1 xennet

in the /etc/modprobe.conf file. This is now working.

To Do

  • Setup SNMP for cacti monitoring.
  • There must be other things....
  • Setup sensors so that we can monitor the system. Will have to wait for a kernel that supports it


  • Sane iptables using ldap.
  • Setup ethernet.
  • Setup RAID volumes.
  • Setup partitions and create file systems.
  • Move the system to System drive and remove the current temp drive.
  • Setup mount points for the data drives.
  • Setup LDAP for users to log in.
  • Setup Exports, so other systems can see the drives. There were issues with firewall, so I modeled the firewall after taro's. Seems to be working, I can successfully ls /net/data/pumpkin1 and ls /net/data/pumpkin2 on einstein.
  • Setup autofs so that it can see other drives. What other drives? It's working for einstein:/home Other drives such as data drives
  • Setup smartd so we will know when a disk is going bad. This can be done inside the RAID card using a system to send SNMP and EMAIL. but it needs to be done. E-mail seems to be set up, let's see if we get any through npg-admins
  • Restrict access (/etc/security/access.conf)
  • Setup sudo on both pumpkin and corn.
  • Add the new systems to the lentil backup script. They're on there; lentil just needs the right SSH keys to rsync them.