Difference between revisions of "Pumpkin"

From Nuclear Physics Group Documentation Pages
Jump to navigationJump to search
 
(82 intermediate revisions by 6 users not shown)
Line 1: Line 1:
−
= Pumpkin =
+
Pumpkin is a 24 disk large storage system. It runs CENTOS 7.
 +
=Hardware =
 +
* 5U Storage Chassis with 24 SAS/SATA-II Hot-Swap Drive Bays with SATA Multilane Backplane (I think it is a Chenbro case)
 +
* 1350 Watt Hot Swap Redundant Power Supply
 +
* [[Areca]] ARC-1280 24 port SATA II Raid - PCI Express x8 -- Address: 10.0.0.199
 +
* Areca ARC-6120 Battery Backup
 +
* 6x Mini-SAS to ML backplane Cable .5M - 4 SATA Drives
 +
* Pioneer DVR-112 Dual Layer DVD/CD writer Internal (Black) 18x write DVD-R/+R, 10x write Dual Layer DVD-R/+R
  
−
Pumpkin is our new 8 CPU 24 disk monster machine. It is really, really nice.
+
== Old Hardware ==
−
Currently it is only tied to the "corn" ip address.
+
Pumpkin is our new 8 CPU 24 disk monster machine. It is really, really nice. Pumpkin runs Xen. It is 64-but CENTOS-7
 +
[[Image:pumpkins.jpg|thumb|200px|Pumpkins]]
 +
=== Old Hardware Details ===
 +
* Microway Quote # MWYQ9518-03  purchased 10/22/2007 for $18260.
 +
* Sales contact: Eliot Eshelman
 +
* Microway 5U 4-Way Opteron Server with up to 24 Drives
 +
* 5U Storage Chassis with 24 SAS/SATA-II Hot-Swap Drive Bays with SATA Multilane Backplane (I think it is a Chenbro case)
 +
* 1350 Watt Hot Swap Redundant Power Supply
 +
* Microway Navion-T (TM) Quad Opteron Motherboard ([http://www.tyan.com/product_board_detail.aspx?pid=271 Tyan S4985]):
 +
** Four sockets for Socket F 8000 series processors
 +
** Nvidia nForce Pro 2200 + 2050
 +
** Four banks of memory (16 DIMM slots)
 +
** Supports up to 64GB of DDR2-667 memory
 +
** Two x16 PCI Express, Two x4 PCI Express,
 +
** One PCI 32 bit expansion slots
 +
** Integrated dual Marvell 88E1111 GbE ports
 +
** Integrated Intel 82541Pl GbE Port
 +
** SIS/Xabre Integrated Graphics 16MB
 +
** Integrated SATA-2 Controller (8 ports)
 +
* 4x AMD Dual Core Socket F Opteron 8222 3.0 GHz, 1 MB Cache / core, 95 watts
 +
* 8x 2GB DDR2 667 MHz ECC/Registered  Memory
 +
*16x 750 GB Seagate Barracuda ES Nearline SATA/300  ST3750640NS 16MB Cache, 3Gb/s, NCQ, 7200rpm, 1.2 million hours MTBF
 +
* [[Areca]] ARC-1280 24 port SATA II Raid - PCI Express x8
 +
* Areca ARC-6120 Battery Backup
 +
* 6x Mini-SAS to ML backplane Cable .5M - 4 SATA Drives
 +
* Pioneer DVR-112 Dual Layer DVD/CD writer Internal (Black) 18x write DVD-R/+R, 10x write Dual Layer DVD-R/+R
 +
* [http://www.tyan.com/product_accessories_spec.aspx?pid=5 Tyan M3291] IPMI card (REMOVED)
  
−
== Basic Setup ==
+
= Network Configuration =
−
* We will run Xen on this so that it can have 2 personalies: Pumpkin, 64-bit, and Corn, 32-bit, RHEL5.
 
−
** In order to do this right, Pumpkin should be the host, since you can't virtualize 64-bit under 32-bit, but you can do the other way around. See the bottom of http://www.redhat.com/rhel/virtualization/. Currently, all boot options in GRUB are 32-bit. The only difference between the first and second boot options is that the first (default) loads an initrd ending with _raid.img, which panics.
 
−
** These may come in handy: http://www.linuxtopia.org/online_books/rhel5/rhel5_xen_virtualization/rhel5_virt-install-wizard.html and http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/en-US/RHEL510/Virtualization_Guide/task-virt-lab1.html.
 
−
* The RAID is currently split. This allows for much easier maintenance and in the future possible upgrades.
 
−
** Disk 1 to 11 is in RAID Set 0, which holds the RAID Volumes: System (300GB, RAID6, SCSI:0.0.0), System1(300GB, RAID6, SCSI:0.0.1), Data1 (6833GB, RAID5, SCSI:0.0.2)
 
−
** Disk 11 to 22 is RAID Set 1, which holds the RAID Volume: Data2 (7499GB, RAID5, SCSI:0.0.3)
 
−
** Disk 23 and 24 are passthrough (single disks) at SCSI:0.0.6 and SCSI:0.0.7. These can be used as spares, as backup, or to expand the other RAID sets later on.
 
−
** The RAID card can be monitored at http://10.0.0.99/ login as "admin" with a password that is the same as the door combo.
 
−
** To use this card with Linux you need a driver: arcmsr. This '''must be part of the initrd''' for the kernel, else you cannot boot from the RAID.
 
−
*** The kernel module can be build from the sources located in /usr/src/kernels/Acera_RAID. Just run make.
 
−
* Currently we have a temporary drive in the system on the onboard SATA which holds a RHEL5 distro and the original RHEL4 distro from the manufacturer.
 
  
−
== Matt's Notes 12/28 ==
+
The network card eth2 no longer works, and is unused. <== needs  verification:
−
It seems that right now, the only bootable install is on the temporary drive. From what I understand, you can use xen to create a guest os on a partition, and once it's all set up, you can even point grub to boot that as the "real" os. A possible plan of action seems like it could be to get raid working on the current install, use xen to put a rhel5_64 install (pumpkin) on one of the raid sets (probably system), boot that, and then xen-install rhel5_32 corn to the other raid set. At that point, we could pull the random drive we're using now and be close to done.
+
    4: wlp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
 +
    link/ether a4:c4:94:1f:6b:cf brd ff:ff:ff:ff:ff:ff
  
−
I tried rebuilding the arcmsr driver, but when I ran 'make clean' in the src dir, I got lots of errors. That's a sign something's amiss.  
+
*IP Address Farm: 10.0.0.243 (farm)
−
I can't access anything useful in the raid bios, since it's passworded and I haven't the slightest clue what the password is.  
+
*IP Address UNH: 132.177.88.228 (unh)
−
From the initialization screen I found the model is the ARC-1280. The pre-made driver can be found here ftp://ftp.areca.com.tw/RaidCards/AP_Drivers/Linux/DRIVER/RedHat/Redhat-EnterpriseLinux/RHEL5.0/. Also, on further research, it's not entirely clear if the xen trickery I was thinking about will work. Not a problem if it doesn't.
+
*IP Address RAID: 10.0.0.99 [http://10.0.0.99]
−
Also, I've downloaded the 64-bit install dvd onto feynman's data drive.
 
−
Trying to boot into the original corn install results in a kernel panic due to switchroot failing. This almost isn't worth fixing, we should start over.
 
  
−
Considering the website has a precompiled rhel5 driver ready to be put on a floppy, as well as instructions for a first-time installation, I think we should scrap this whole temporary drive idea and install pumpkin on one of the system sets, so it can be our dom0 without any weird setups. Considering at least half our problems are made more difficult by weird setups, this could be a step in the right direction.
+
=Software and Services=
  
−
=== My proposed course of action ===
+
==IPTables==
−
# Using the ARECA driver floppy, install RHEL5 64-bit on one of the "system" raid sets with a normal dvd-rom based install. Use a xen kernel. Name it pumpkin, use standard packages. Nothing fancy.
 
−
# Make sure pumpkin works.
 
−
# Use virt-manager to install corn on the other "system" raid set as RHEL 32-bit. We should use virt-manager because it's essentially a frontend to xen, so if something breaks we can do it cli-style, and if stuff isn't breaking there's less to be confused about. Also, it's the standard redhat tool for this. Based on my playing around with it today, this should be a breeze.
 
−
# Make sure corn works.
 
−
# ./hooray
 
  
−
== To Do ==
+
Pumpkin uses the standard [[iptables]] configuration.
  
−
* Move the system to System drive and remove the current temp drive.
+
==Splunk==
−
* Setup mount points for the data drives.
 
−
* Setup LDAP for users to log in. I started, but it's not working.
 
−
* Setup Exports, so other systems can see the drives.
 
−
* Setup autofs so that it can see other drives.
 
−
* Setup sensors so that we can monitor the system.
 
−
* Setup smartd so we will know when a disk is going bad. '''This can be done inside the RAID card''' using a system to send SNMP and EMAIL. but it needs to be done.
 
−
* Setup the other system with Xen on the System1 drive
 
−
* Setup SNMP for cacti monitoring.
 
−
* Add the new systems to the lentil backup script
 
−
* There must be other things....
 
  
−
== Done ==
+
Pumpkin is the master [[Splunk]] node and stores all of the splunk data in /data1/splunk. If you want to access the Splunk web interface it is at https://pumpkin.unh.edu:8000 (if you're connected via the Farm), or you can forward port 8000 over SSH.
−
* Setup ethernet.
+
 
−
* Setup RAID volumes.
+
==[[NFS]]==
−
* Setup partitions and create file systems.
+
 
 +
Pumpkin shares two data stores (/data and /scratch) over [[NFS]]. They can be accessed at /net/data/pumpkin and /net/data/scratch from any machine.
 +
 
 +
= RAID =
 +
The RAID is currently split.
 +
; Disk 1 to 17 @ 4TB
 +
* Disk 17 is hot spare
 +
* RAID Set #00
 +
* Volume: data lun(0/0/0) is RAID6 = 56 TB
 +
* Translates to /dev/sda1 mounted on /data
 +
; Disk 19 to 24 @ 750 GB
 +
* RAID Set #01
 +
* Volume: scratch lun(0/0/1) is RAID0 = 4.5 TB
 +
* Translates to /dev/sdb1 mounted on /scratch
 +
; Disk18 is N/A 
 +
 
 +
The RAID card can be monitored at http://10.0.0.99/ login as "admin" with a password that is described on the [[RAID]] page.
 +
 
 +
== Software RAID ==
 +
There are 2 internal drives in Pumpkin forming a software RAID.
 +
Note these are *not identical* partitions on the drives, so the RAID does not look symmetrical. Blame the installer.
 +
 
 +
Model: ATA WDC WD7500AAKS-0 (scsi)
 +
Disk /dev/sdc: 750GB
 +
Sector size (logical/physical): 512B/512B
 +
Partition Table: gpt
 +
Disk Flags:
 +
 +
Number  Start  End    Size    File system  Name                  Flags
 +
  1      1049kB  79.7MB  78.6MB  fat16        EFI System Partition  boot
 +
  2      79.7MB  300GB  300GB                                      raid
 +
  3      300GB  550GB  250GB                                      raid
 +
  4      550GB  650GB  100GB                                      raid
 +
  5      650GB  718GB  67.5GB                                    raid
 +
  6      718GB  750GB  32.0GB                                    lvm
 +
  7      750GB  750GB  251MB  ext4                              raid
 +
 +
Model: ATA WDC WD7500AAKS-0 (scsi)
 +
Disk /dev/sdd: 750GB
 +
Sector size (logical/physical): 512B/512B
 +
Partition Table: gpt
 +
Disk Flags:
 +
 +
Number  Start  End    Size    File system  Name  Flags
 +
  1      1049kB  300GB  300GB                      raid
 +
  2      300GB  550GB  250GB                      raid
 +
  3      550GB  650GB  100GB                      raid
 +
  4      650GB  718GB  67.5GB                    raid
 +
  5      718GB  750GB  32.0GB                    lvm
 +
  6      750GB  750GB  251MB  ext4              raid
 +
 +
 
 +
Personalities : [raid1]
 +
md123 : active raid1 sdc5[0] sdd4[1]
 +
      65853440 blocks super 1.2 [2/2] [UU]
 +
      bitmap: 0/1 pages [0KB], 65536KB chunk
 +
 +
md124 : active raid1 sdc4[0] sdd3[1]
 +
      97655808 blocks super 1.2 [2/2] [UU]
 +
      bitmap: 0/1 pages [0KB], 65536KB chunk
 +
 
 +
md125 : active raid1 sdc2[0] sdd1[1]
 +
      292968448 blocks super 1.2 [2/2] [UU]
 +
      bitmap: 2/3 pages [8KB], 65536KB chunk
 +
 +
md126 : active raid1 sdc3[0] sdd2[1]
 +
      244140032 blocks super 1.2 [2/2] [UU]
 +
      bitmap: 0/2 pages [0KB], 65536KB chunk
 +
 +
md127 : active raid1 sdc7[0] sdd6[1]
 +
      244672 blocks super 1.0 [2/2] [UU]
 +
      bitmap: 0/1 pages [0KB], 65536KB chunk
 +
 
 +
Filesystem      Size  Used Avail Use% Mounted on
 +
/dev/md123      62G  85M  59G  1% /kvm
 +
/dev/md124      92G  527M  87G  1% /var
 +
/dev/md125      275G  24G  238G  9% /
 +
/dev/md126      230G  4.7G  213G  3% /usr
 +
/dev/md127      228M  224M    0 100% /boot
 +
/dev/sdc1        75M  9.4M  66M  13% /boot/efi
 +
/dev/sda1        51T  30T  22T  58% /data
 +
/dev/sdb1      4.1T  89M  4.1T  1% /scratch
 +
 
 +
= To Do =
 +
 
 +
= Done =

Latest revision as of 20:34, 3 January 2018

Pumpkin is a 24 disk large storage system. It runs CENTOS 7.

Hardware

  • 5U Storage Chassis with 24 SAS/SATA-II Hot-Swap Drive Bays with SATA Multilane Backplane (I think it is a Chenbro case)
  • 1350 Watt Hot Swap Redundant Power Supply
  • Areca ARC-1280 24 port SATA II Raid - PCI Express x8 -- Address: 10.0.0.199
  • Areca ARC-6120 Battery Backup
  • 6x Mini-SAS to ML backplane Cable .5M - 4 SATA Drives
  • Pioneer DVR-112 Dual Layer DVD/CD writer Internal (Black) 18x write DVD-R/+R, 10x write Dual Layer DVD-R/+R

Old Hardware

Pumpkin is our new 8 CPU 24 disk monster machine. It is really, really nice. Pumpkin runs Xen. It is 64-but CENTOS-7

Pumpkins

Old Hardware Details

  • Microway Quote # MWYQ9518-03 purchased 10/22/2007 for $18260.
  • Sales contact: Eliot Eshelman
  • Microway 5U 4-Way Opteron Server with up to 24 Drives
  • 5U Storage Chassis with 24 SAS/SATA-II Hot-Swap Drive Bays with SATA Multilane Backplane (I think it is a Chenbro case)
  • 1350 Watt Hot Swap Redundant Power Supply
  • Microway Navion-T (TM) Quad Opteron Motherboard (Tyan S4985):
    • Four sockets for Socket F 8000 series processors
    • Nvidia nForce Pro 2200 + 2050
    • Four banks of memory (16 DIMM slots)
    • Supports up to 64GB of DDR2-667 memory
    • Two x16 PCI Express, Two x4 PCI Express,
    • One PCI 32 bit expansion slots
    • Integrated dual Marvell 88E1111 GbE ports
    • Integrated Intel 82541Pl GbE Port
    • SIS/Xabre Integrated Graphics 16MB
    • Integrated SATA-2 Controller (8 ports)
  • 4x AMD Dual Core Socket F Opteron 8222 3.0 GHz, 1 MB Cache / core, 95 watts
  • 8x 2GB DDR2 667 MHz ECC/Registered Memory
  • 16x 750 GB Seagate Barracuda ES Nearline SATA/300 ST3750640NS 16MB Cache, 3Gb/s, NCQ, 7200rpm, 1.2 million hours MTBF
  • Areca ARC-1280 24 port SATA II Raid - PCI Express x8
  • Areca ARC-6120 Battery Backup
  • 6x Mini-SAS to ML backplane Cable .5M - 4 SATA Drives
  • Pioneer DVR-112 Dual Layer DVD/CD writer Internal (Black) 18x write DVD-R/+R, 10x write Dual Layer DVD-R/+R
  • Tyan M3291 IPMI card (REMOVED)

Network Configuration

The network card eth2 no longer works, and is unused. <== needs verification: 

   4: wlp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
   link/ether a4:c4:94:1f:6b:cf brd ff:ff:ff:ff:ff:ff
  • IP Address Farm: 10.0.0.243 (farm)
  • IP Address UNH: 132.177.88.228 (unh)
  • IP Address RAID: 10.0.0.99 [1]

Software and Services

IPTables

Pumpkin uses the standard iptables configuration.

Splunk

Pumpkin is the master Splunk node and stores all of the splunk data in /data1/splunk. If you want to access the Splunk web interface it is at https://pumpkin.unh.edu:8000 (if you're connected via the Farm), or you can forward port 8000 over SSH.

NFS

Pumpkin shares two data stores (/data and /scratch) over NFS. They can be accessed at /net/data/pumpkin and /net/data/scratch from any machine.

RAID

The RAID is currently split.

Disk 1 to 17 @ 4TB
  • Disk 17 is hot spare
  • RAID Set #00
  • Volume: data lun(0/0/0) is RAID6 = 56 TB
  • Translates to /dev/sda1 mounted on /data
Disk 19 to 24 @ 750 GB
  • RAID Set #01
  • Volume: scratch lun(0/0/1) is RAID0 = 4.5 TB
  • Translates to /dev/sdb1 mounted on /scratch
Disk18 is N/A

The RAID card can be monitored at http://10.0.0.99/ login as "admin" with a password that is described on the RAID page.

Software RAID

There are 2 internal drives in Pumpkin forming a software RAID. Note these are *not identical* partitions on the drives, so the RAID does not look symmetrical. Blame the installer. 

Model: ATA WDC WD7500AAKS-0 (scsi)
Disk /dev/sdc: 750GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name                  Flags
 1      1049kB  79.7MB  78.6MB  fat16        EFI System Partition  boot
 2      79.7MB  300GB   300GB                                      raid
 3      300GB   550GB   250GB                                      raid
 4      550GB   650GB   100GB                                      raid
 5      650GB   718GB   67.5GB                                     raid
 6      718GB   750GB   32.0GB                                     lvm
 7      750GB   750GB   251MB   ext4                               raid

Model: ATA WDC WD7500AAKS-0 (scsi)
Disk /dev/sdd: 750GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End    Size    File system  Name  Flags
 1      1049kB  300GB  300GB                      raid
 2      300GB   550GB  250GB                      raid
 3      550GB   650GB  100GB                      raid
 4      650GB   718GB  67.5GB                     raid
 5      718GB   750GB  32.0GB                     lvm
 6      750GB   750GB  251MB   ext4               raid


Personalities : [raid1]
md123 : active raid1 sdc5[0] sdd4[1]
      65853440 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md124 : active raid1 sdc4[0] sdd3[1]
      97655808 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk
 
md125 : active raid1 sdc2[0] sdd1[1]
      292968448 blocks super 1.2 [2/2] [UU]
      bitmap: 2/3 pages [8KB], 65536KB chunk 

md126 : active raid1 sdc3[0] sdd2[1]
      244140032 blocks super 1.2 [2/2] [UU]
      bitmap: 0/2 pages [0KB], 65536KB chunk

md127 : active raid1 sdc7[0] sdd6[1]
      244672 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

Filesystem      Size  Used Avail Use% Mounted on
/dev/md123       62G   85M   59G   1% /kvm
/dev/md124       92G  527M   87G   1% /var
/dev/md125      275G   24G  238G   9% /
/dev/md126      230G  4.7G  213G   3% /usr
/dev/md127      228M  224M     0 100% /boot
/dev/sdc1        75M  9.4M   66M  13% /boot/efi
/dev/sda1        51T   30T   22T  58% /data
/dev/sdb1       4.1T   89M  4.1T   1% /scratch

To Do

Done

Retrieved from "https://nuclear.unh.edu/wiki/index.php?title=Pumpkin&oldid=6262"