Difference between revisions of "Pumpkin"

From Nuclear Physics Group Documentation Pages
Jump to navigationJump to search
 
(85 intermediate revisions by 6 users not shown)
Line 1: Line 1:
−
= Pumpkin =
+
Pumpkin is a 24 disk large storage system. It runs CENTOS 7.
 +
=Hardware =
 +
* 5U Storage Chassis with 24 SAS/SATA-II Hot-Swap Drive Bays with SATA Multilane Backplane (I think it is a Chenbro case)
 +
* 1350 Watt Hot Swap Redundant Power Supply
 +
* [[Areca]] ARC-1280 24 port SATA II Raid - PCI Express x8 -- Address: 10.0.0.199
 +
* Areca ARC-6120 Battery Backup
 +
* 6x Mini-SAS to ML backplane Cable .5M - 4 SATA Drives
 +
* Pioneer DVR-112 Dual Layer DVD/CD writer Internal (Black) 18x write DVD-R/+R, 10x write Dual Layer DVD-R/+R
  
−
Pumpkin is our new 8 CPU 24 disk monster machine. It is really, really nice.
+
== Old Hardware ==
−
Currently it is only tied to the "corn" ip address.
+
Pumpkin is our new 8 CPU 24 disk monster machine. It is really, really nice. Pumpkin runs Xen. It is 64-but CENTOS-7
 +
[[Image:pumpkins.jpg|thumb|200px|Pumpkins]]
 +
=== Old Hardware Details ===
 +
* Microway Quote # MWYQ9518-03  purchased 10/22/2007 for $18260.
 +
* Sales contact: Eliot Eshelman
 +
* Microway 5U 4-Way Opteron Server with up to 24 Drives
 +
* 5U Storage Chassis with 24 SAS/SATA-II Hot-Swap Drive Bays with SATA Multilane Backplane (I think it is a Chenbro case)
 +
* 1350 Watt Hot Swap Redundant Power Supply
 +
* Microway Navion-T (TM) Quad Opteron Motherboard ([http://www.tyan.com/product_board_detail.aspx?pid=271 Tyan S4985]):
 +
** Four sockets for Socket F 8000 series processors
 +
** Nvidia nForce Pro 2200 + 2050
 +
** Four banks of memory (16 DIMM slots)
 +
** Supports up to 64GB of DDR2-667 memory
 +
** Two x16 PCI Express, Two x4 PCI Express,
 +
** One PCI 32 bit expansion slots
 +
** Integrated dual Marvell 88E1111 GbE ports
 +
** Integrated Intel 82541Pl GbE Port
 +
** SIS/Xabre Integrated Graphics 16MB
 +
** Integrated SATA-2 Controller (8 ports)
 +
* 4x AMD Dual Core Socket F Opteron 8222 3.0 GHz, 1 MB Cache / core, 95 watts
 +
* 8x 2GB DDR2 667 MHz ECC/Registered  Memory
 +
*16x 750 GB Seagate Barracuda ES Nearline SATA/300  ST3750640NS 16MB Cache, 3Gb/s, NCQ, 7200rpm, 1.2 million hours MTBF
 +
* [[Areca]] ARC-1280 24 port SATA II Raid - PCI Express x8
 +
* Areca ARC-6120 Battery Backup
 +
* 6x Mini-SAS to ML backplane Cable .5M - 4 SATA Drives
 +
* Pioneer DVR-112 Dual Layer DVD/CD writer Internal (Black) 18x write DVD-R/+R, 10x write Dual Layer DVD-R/+R
 +
* [http://www.tyan.com/product_accessories_spec.aspx?pid=5 Tyan M3291] IPMI card (REMOVED)
  
−
== Basic Setup ==
+
= Network Configuration =
−
* We will run Xen on this so that it can have 2 personalies: Pumpkin, 64-bit, and Corn, 32-bit, RHEL5.
 
−
** In order to do this right, Pumpkin should be the host, since you can't virtualize 64-bit under 32-bit, but you can do the other way around. See the bottom of http://www.redhat.com/rhel/virtualization/. Currently, all boot options in GRUB are 32-bit. The only difference between the first and second boot options is that the first (default) loads an initrd ending with _raid.img, which panics.
 
−
** These may come in handy: http://www.linuxtopia.org/online_books/rhel5/rhel5_xen_virtualization/rhel5_virt-install-wizard.html and http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/en-US/RHEL510/Virtualization_Guide/task-virt-lab1.html.
 
−
* The RAID is currently split. This allows for much easier maintenance and in the future possible upgrades.
 
−
** Disk 1 to 11 is in RAID Set 0, which holds the RAID Volumes: System (300GB, RAID6, SCSI:0.0.0), System1(300GB, RAID6, SCSI:0.0.1), Data1 (6833GB, RAID5, SCSI:0.0.2)
 
−
** Disk 11 to 22 is RAID Set 1, which holds the RAID Volume: Data2 (7499GB, RAID5, SCSI:0.0.3)
 
−
** Disk 23 and 24 are passthrough (single disks) at SCSI:0.0.6 and SCSI:0.0.7. These can be used as spares, as backup, or to expand the other RAID sets later on.
 
−
** The RAID card can be monitored at http://10.0.0.99/ login as "admin" with a password that is the same as the door combo.
 
−
** To use this card with Linux you need a driver: arcmsr. This '''must be part of the initrd''' for the kernel, else you cannot boot from the RAID.
 
−
*** The kernel module can be build from the sources located in /usr/src/kernels/Acera_RAID. Just run make.
 
−
* Currently we have a temporary drive in the system on the onboard SATA which holds a RHEL5 distro and the original RHEL4 distro from the manufacturer.
 
  
−
It seems that right now, the only bootable install is on the temporary drive. From what I understand, you can use xen to create a guest os on a partition, and once it's all set up, you can even point grub to boot that as the "real" os. A possible plan of action seems like it could be to get raid working on the current install, use xen to put a rhel5_64 install (pumpkin) on one of the raid sets (probably system), boot that, and then xen-install rhel5_32 corn to the other raid set. At that point, we could pull the random drive we're using now and be close to done.
+
The network card eth2 no longer works, and is unused. <== needs  verification:
 +
    4: wlp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
 +
    link/ether a4:c4:94:1f:6b:cf brd ff:ff:ff:ff:ff:ff
  
−
I tried rebuilding the arcmsr driver, but when I ran 'make clean' in the src dir, I got lots of errors. That's a sign something's amiss. According to the areca raid manual, I should be able to download the precompiled driver from their website. I can't access anything useful in the raid bios, since it's passworded and I haven't the slightest clue what the password is. From the initialization screen I found the model is the ARC-1280. Also, on further research, it's not entirely clear if the xen trickery I was thinking about will work. No problem if it doesn't. Also, I've downloaded the 64-bit install dvd onto feynman's data drive, which still doesn't mount by default. It's physically  /dev/mapper/vg_improv-data. Trying to boot into the original corn install results in a kernel panic due to switchroot failing. What's going on here?
+
*IP Address Farm: 10.0.0.243 (farm)
 +
*IP Address UNH: 132.177.88.228 (unh)
 +
*IP Address RAID: 10.0.0.99 [http://10.0.0.99]
  
−
== To Do ==
+
=Software and Services=
  
−
* Move the system to System drive and remove the current temp drive.
+
==IPTables==
−
* Setup mount points for the data drives.
 
−
* Setup LDAP for users to log in. I started, but it's not working.
 
−
* Setup Exports, so other systems can see the drives.
 
−
* Setup autofs so that it can see other drives.
 
−
* Setup sensors so that we can monitor the system.
 
−
* Setup smartd so we will know when a disk is going bad. '''This can be done inside the RAID card''' using a system to send SNMP and EMAIL. but it needs to be done.
 
−
* Setup the other system with Xen on the System1 drive
 
−
* Setup SNMP for cacti monitoring.
 
−
* There must be other things....
 
  
−
== Done ==
+
Pumpkin uses the standard [[iptables]] configuration.
−
* Setup ethernet.
+
 
−
* Setup RAID volumes.
+
==Splunk==
−
* Setup partitions and create file systems.
+
 
 +
Pumpkin is the master [[Splunk]] node and stores all of the splunk data in /data1/splunk. If you want to access the Splunk web interface it is at https://pumpkin.unh.edu:8000 (if you're connected via the Farm), or you can forward port 8000 over SSH.
 +
 
 +
==[[NFS]]==
 +
 
 +
Pumpkin shares two data stores (/data and /scratch) over [[NFS]]. They can be accessed at /net/data/pumpkin and /net/data/scratch from any machine.
 +
 
 +
= RAID =
 +
The RAID is currently split.
 +
; Disk 1 to 17 @ 4TB
 +
* Disk 17 is hot spare
 +
* RAID Set #00
 +
* Volume: data lun(0/0/0) is RAID6 = 56 TB
 +
* Translates to /dev/sda1 mounted on /data
 +
; Disk 19 to 24 @ 750 GB
 +
* RAID Set #01
 +
* Volume: scratch lun(0/0/1) is RAID0 = 4.5 TB
 +
* Translates to /dev/sdb1 mounted on /scratch
 +
; Disk18 is N/A 
 +
 
 +
The RAID card can be monitored at http://10.0.0.99/ login as "admin" with a password that is described on the [[RAID]] page.
 +
 
 +
== Software RAID ==
 +
There are 2 internal drives in Pumpkin forming a software RAID.
 +
Note these are *not identical* partitions on the drives, so the RAID does not look symmetrical. Blame the installer.
 +
 
 +
Model: ATA WDC WD7500AAKS-0 (scsi)
 +
Disk /dev/sdc: 750GB
 +
Sector size (logical/physical): 512B/512B
 +
Partition Table: gpt
 +
Disk Flags:
 +
 +
Number  Start  End    Size    File system  Name                  Flags
 +
  1      1049kB  79.7MB  78.6MB  fat16        EFI System Partition  boot
 +
  2      79.7MB  300GB  300GB                                      raid
 +
  3      300GB  550GB  250GB                                      raid
 +
  4      550GB  650GB  100GB                                      raid
 +
  5      650GB  718GB  67.5GB                                    raid
 +
  6      718GB  750GB  32.0GB                                    lvm
 +
  7      750GB  750GB  251MB  ext4                              raid
 +
 +
Model: ATA WDC WD7500AAKS-0 (scsi)
 +
Disk /dev/sdd: 750GB
 +
Sector size (logical/physical): 512B/512B
 +
Partition Table: gpt
 +
Disk Flags:
 +
 +
Number  Start  End    Size    File system  Name  Flags
 +
  1      1049kB  300GB  300GB                      raid
 +
  2      300GB  550GB  250GB                      raid
 +
  3      550GB  650GB  100GB                      raid
 +
  4      650GB  718GB  67.5GB                    raid
 +
  5      718GB  750GB  32.0GB                    lvm
 +
  6      750GB  750GB  251MB  ext4              raid
 +
 +
 
 +
Personalities : [raid1]
 +
md123 : active raid1 sdc5[0] sdd4[1]
 +
      65853440 blocks super 1.2 [2/2] [UU]
 +
      bitmap: 0/1 pages [0KB], 65536KB chunk
 +
 +
md124 : active raid1 sdc4[0] sdd3[1]
 +
      97655808 blocks super 1.2 [2/2] [UU]
 +
      bitmap: 0/1 pages [0KB], 65536KB chunk
 +
 
 +
md125 : active raid1 sdc2[0] sdd1[1]
 +
      292968448 blocks super 1.2 [2/2] [UU]
 +
      bitmap: 2/3 pages [8KB], 65536KB chunk
 +
 +
md126 : active raid1 sdc3[0] sdd2[1]
 +
      244140032 blocks super 1.2 [2/2] [UU]
 +
      bitmap: 0/2 pages [0KB], 65536KB chunk
 +
 +
md127 : active raid1 sdc7[0] sdd6[1]
 +
      244672 blocks super 1.0 [2/2] [UU]
 +
      bitmap: 0/1 pages [0KB], 65536KB chunk
 +
 
 +
Filesystem      Size  Used Avail Use% Mounted on
 +
/dev/md123      62G  85M  59G  1% /kvm
 +
/dev/md124      92G  527M  87G  1% /var
 +
/dev/md125      275G  24G  238G  9% /
 +
/dev/md126      230G  4.7G  213G  3% /usr
 +
/dev/md127      228M  224M    0 100% /boot
 +
/dev/sdc1        75M  9.4M  66M  13% /boot/efi
 +
/dev/sda1        51T  30T  22T  58% /data
 +
/dev/sdb1      4.1T  89M  4.1T  1% /scratch
 +
 
 +
= To Do =
 +
 
 +
= Done =

Latest revision as of 20:34, 3 January 2018

Pumpkin is a 24 disk large storage system. It runs CENTOS 7.

Hardware

  • 5U Storage Chassis with 24 SAS/SATA-II Hot-Swap Drive Bays with SATA Multilane Backplane (I think it is a Chenbro case)
  • 1350 Watt Hot Swap Redundant Power Supply
  • Areca ARC-1280 24 port SATA II Raid - PCI Express x8 -- Address: 10.0.0.199
  • Areca ARC-6120 Battery Backup
  • 6x Mini-SAS to ML backplane Cable .5M - 4 SATA Drives
  • Pioneer DVR-112 Dual Layer DVD/CD writer Internal (Black) 18x write DVD-R/+R, 10x write Dual Layer DVD-R/+R

Old Hardware

Pumpkin is our new 8 CPU 24 disk monster machine. It is really, really nice. Pumpkin runs Xen. It is 64-but CENTOS-7

Pumpkins

Old Hardware Details

  • Microway Quote # MWYQ9518-03 purchased 10/22/2007 for $18260.
  • Sales contact: Eliot Eshelman
  • Microway 5U 4-Way Opteron Server with up to 24 Drives
  • 5U Storage Chassis with 24 SAS/SATA-II Hot-Swap Drive Bays with SATA Multilane Backplane (I think it is a Chenbro case)
  • 1350 Watt Hot Swap Redundant Power Supply
  • Microway Navion-T (TM) Quad Opteron Motherboard (Tyan S4985):
    • Four sockets for Socket F 8000 series processors
    • Nvidia nForce Pro 2200 + 2050
    • Four banks of memory (16 DIMM slots)
    • Supports up to 64GB of DDR2-667 memory
    • Two x16 PCI Express, Two x4 PCI Express,
    • One PCI 32 bit expansion slots
    • Integrated dual Marvell 88E1111 GbE ports
    • Integrated Intel 82541Pl GbE Port
    • SIS/Xabre Integrated Graphics 16MB
    • Integrated SATA-2 Controller (8 ports)
  • 4x AMD Dual Core Socket F Opteron 8222 3.0 GHz, 1 MB Cache / core, 95 watts
  • 8x 2GB DDR2 667 MHz ECC/Registered Memory
  • 16x 750 GB Seagate Barracuda ES Nearline SATA/300 ST3750640NS 16MB Cache, 3Gb/s, NCQ, 7200rpm, 1.2 million hours MTBF
  • Areca ARC-1280 24 port SATA II Raid - PCI Express x8
  • Areca ARC-6120 Battery Backup
  • 6x Mini-SAS to ML backplane Cable .5M - 4 SATA Drives
  • Pioneer DVR-112 Dual Layer DVD/CD writer Internal (Black) 18x write DVD-R/+R, 10x write Dual Layer DVD-R/+R
  • Tyan M3291 IPMI card (REMOVED)

Network Configuration

The network card eth2 no longer works, and is unused. <== needs verification: 

   4: wlp6s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
   link/ether a4:c4:94:1f:6b:cf brd ff:ff:ff:ff:ff:ff
  • IP Address Farm: 10.0.0.243 (farm)
  • IP Address UNH: 132.177.88.228 (unh)
  • IP Address RAID: 10.0.0.99 [1]

Software and Services

IPTables

Pumpkin uses the standard iptables configuration.

Splunk

Pumpkin is the master Splunk node and stores all of the splunk data in /data1/splunk. If you want to access the Splunk web interface it is at https://pumpkin.unh.edu:8000 (if you're connected via the Farm), or you can forward port 8000 over SSH.

NFS

Pumpkin shares two data stores (/data and /scratch) over NFS. They can be accessed at /net/data/pumpkin and /net/data/scratch from any machine.

RAID

The RAID is currently split.

Disk 1 to 17 @ 4TB
  • Disk 17 is hot spare
  • RAID Set #00
  • Volume: data lun(0/0/0) is RAID6 = 56 TB
  • Translates to /dev/sda1 mounted on /data
Disk 19 to 24 @ 750 GB
  • RAID Set #01
  • Volume: scratch lun(0/0/1) is RAID0 = 4.5 TB
  • Translates to /dev/sdb1 mounted on /scratch
Disk18 is N/A

The RAID card can be monitored at http://10.0.0.99/ login as "admin" with a password that is described on the RAID page.

Software RAID

There are 2 internal drives in Pumpkin forming a software RAID. Note these are *not identical* partitions on the drives, so the RAID does not look symmetrical. Blame the installer. 

Model: ATA WDC WD7500AAKS-0 (scsi)
Disk /dev/sdc: 750GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name                  Flags
 1      1049kB  79.7MB  78.6MB  fat16        EFI System Partition  boot
 2      79.7MB  300GB   300GB                                      raid
 3      300GB   550GB   250GB                                      raid
 4      550GB   650GB   100GB                                      raid
 5      650GB   718GB   67.5GB                                     raid
 6      718GB   750GB   32.0GB                                     lvm
 7      750GB   750GB   251MB   ext4                               raid

Model: ATA WDC WD7500AAKS-0 (scsi)
Disk /dev/sdd: 750GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End    Size    File system  Name  Flags
 1      1049kB  300GB  300GB                      raid
 2      300GB   550GB  250GB                      raid
 3      550GB   650GB  100GB                      raid
 4      650GB   718GB  67.5GB                     raid
 5      718GB   750GB  32.0GB                     lvm
 6      750GB   750GB  251MB   ext4               raid


Personalities : [raid1]
md123 : active raid1 sdc5[0] sdd4[1]
      65853440 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md124 : active raid1 sdc4[0] sdd3[1]
      97655808 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk
 
md125 : active raid1 sdc2[0] sdd1[1]
      292968448 blocks super 1.2 [2/2] [UU]
      bitmap: 2/3 pages [8KB], 65536KB chunk 

md126 : active raid1 sdc3[0] sdd2[1]
      244140032 blocks super 1.2 [2/2] [UU]
      bitmap: 0/2 pages [0KB], 65536KB chunk

md127 : active raid1 sdc7[0] sdd6[1]
      244672 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

Filesystem      Size  Used Avail Use% Mounted on
/dev/md123       62G   85M   59G   1% /kvm
/dev/md124       92G  527M   87G   1% /var
/dev/md125      275G   24G  238G   9% /
/dev/md126      230G  4.7G  213G   3% /usr
/dev/md127      228M  224M     0 100% /boot
/dev/sdc1        75M  9.4M   66M  13% /boot/efi
/dev/sda1        51T   30T   22T  58% /data
/dev/sdb1       4.1T   89M  4.1T   1% /scratch

To Do

Done

Retrieved from "https://nuclear.unh.edu/wiki/index.php?title=Pumpkin&oldid=6262"