One of our customers had some Dell R410 machines that were humming along just nicely. One day something happened and all of the sudden xCAT rpower stopped working. Was it that we updated the firmware? Was it new xCAT code? We couldn’t figure it out.
Our first epiphany came when we realized that ipmitool worked just fine with lan (IPMI 1.5) and lanplus (IPMI2.0). ipmitool works? Why didn’t xCAT? It turns out that there was a problem authenticating. In fact, if we used lanplus, we didn’t have to even enter the correct password and we could turn machines off and on!
# ipmitool -I lanplus -U user1 -P thispasswordisbogus -H node001-drac -C 0 mc info
Wow! That’s a major security violation. We alerted Dell. But that still didn’t get xCAT’s rpower working. As a temp solution, we modified /opt/xcat/lib/perl/xCAT/IPMI.pm to not try IPMI 2.0 and instead use only IPMI 1.5. This worked fine for some things, but the great necessities of rcons, reventlog, and rinv would have taken more time to get working… and after all this was a temporary patch right?
So today I woke up determined to resolve the issue once and for all. Working with Dell support (who were very nice and eager to help us) I figured out that there was a an IPMI Encryption key that was set to some random 40 character hexadecimal string. How it was set, I still don’t know. Viewing it in the iDRAC looked like this:
We instead cleared that key and set it to ’00’ (It was required to be an even number of hexadecimal characters). Doing this solved out IPMI issue. xCAT rpower then worked without a problem. Dell then gave us a way to run this via the command line:
racadm -r drac-comp036 -u user1 -p asdfasdf config -g cfgIpmiLan -o cfgIpmiEncryptionKey 0000000000000000000000000000000000000000000000
With that, order was restored.
This also shows one great thing about xCAT: The IPMI packets coming back were not authenticating correctly. There was a problem with the way the challenges were coming back. ipmitool seemed to be very forgiving about that and not care. xCAT didn’t like it at all and would not let it pass. We view this as an ipmitool bug than an xCAT bug. Wouldn’t you rather know about a potential security problem?