Here are some commands for reference that my friend Shadd sent me. Some of them are IPMI, while the others are dmidecode commands to get system information. Thanks Shadd!
Are the system fans are all working? Or just not moving?
psh <noderange> ipmitool sdr |grep "| 0 RPM"
BMC firmware level check
psh -g <noderange> 'ipmitool -I open bmc info | grep -A3 "Firmware Revision" |sort' | xcoll
Clear the BMC Error Log cluster wide
psh <noderange> ipmitool sel clear
Check the Fan Controller Board firmware level
psh <noderange> 'ipmitool raw 0x34 0x16' |xcoll # works on Dell, maybe others?
Check why each node rebooted last
psh <noderange> 'ipmitool chassis restart_cause |sort' |xcoll
Check PSU states
psh <noderange> ipmitool sensor |grep POUT |xcoll
C6100 and IPMItool on individual nodes
Identify chassis with beacon light
ipmitool -H ${NODE} -U root -P root chassis identify
Serial Over Lan session
ipmitool -e ?. -I lanplus -H ${NODE} -U root -P root sol activate
Seting BMC/DRAC time
ipmitool -H ${NODE} -U root -P root sel time set "01/21/2011 16:20:44" &
System Inventory Data
ipmitool -H ${NODE} -U root -P root fru
Cassis/Power Sub info
ipmitool -H ${NODE} -U root -P root chassis status
Chassis power State
ipmitool -H ${NODE} -U root -P root chassis power status
Power-On Hours counter
ipmitool -H ${NODE} -U root -P root chassis poh
Cause of last reboot
ipmitool -H ${NODE} -U root -P root chassis restart_cause
Sel Log
ipmitool -H ${NODE} -U root -P root sel list
Sensor Data Records
ipmitool -H ${NODE} -U root -P root sdr
Sensor thresholds wide table format
ipmitool -H ${NODE} -U root -P root sensor
Health Check Commands
BIOS Level Check (usually xCAT's rinv will give this to you on some nodes as well)
psh <noderange> 'dmidecode | grep -A3 "BIOS Information" | grep Version' | xcoll
Check System Memory
psh <noderange> free |grep Mem |awk {'print $1 $3'} | xcoll
Check 6100 DIMM Slots
psh <noderange> dmidecode -t 17 |egrep '(Size: 4096)|(Bank Locator: DIMM01)|(Bank Locator: DIMM03)|(Bank Locator: DIMM05)|(Bank Locator: DIMM07)|(Bank Locator: DIMM09)|(Bank Locator: DIMM11)' | xcoll
Check R710 DIMM Slots
psh <noderange> dmidecode -t 17 |egrep '(Size: 8192)|(Locator: DIMM_A1)|(Locator: DIMM_A2)|(Locator: DIMM_A3)|(Locator: DIMM_A4)|(Locator: DIMM_A5)|(Locator: DIMM_A6)|(Locator: DIMM_A7)|(Locator: DIMM_A8)|(Locator: DIMM_A9)|(Locator: DIMM_B1)|(Locator: DIMM_B2)|(Locator: DIMM_B3)|(Locator: DIMM_B4)|(Locator: DIMM_B5)|(Locator: DIMM_B6)|(Locator: DIMM_B7)|(Locator: DIMM_B8)|(Locator: DIMM_B9)'
Check IB State
psh <noderange> ibstat | egrep '(State: Active)' | xcoll