I’ve always found a good way to get started with a new language is to just go head into it trying to solve some problem you’re working on, then ask the Internet via a search engine how to do something you want to do.
Today I started for the first time with PowerShell. Here’s the problem I was trying to solve:
I have a lot of VMs that I want to bring up for a class that I’m giving. There could be between 10-50 people at any given class. What I need to do is clone a master VM from a template, change its MAC address to something I have reserved in my DHCP server, and then power it up.
All of this can be done via vCenter. But its a slow and painful point-and-click-yourself-to-death process. Automation is the way to get this done.
All of this can be done via xCAT. But I figured why not give it a try. Other people can live without xCAT. Maybe I can too.
I have a history of perl. Yes, I have lots of skeletons in the closet with that language. So I was going to do all of this with Perl. But I figured that since I was going to be doing power shell scripts anyway for the UCS emulator portion, why not just do it all with the same thing? So I gave it a shot.
First off, the Windows editors suck. I had to stick with VIM because there’s nothing better for me. Let’s not even argue about that. +1 for me for retaining my dignity.
Next, after installing the VMware PowerCLI tool it was pretty easy.
CreateVMs.ps1
Here’s the script to clone 9 VMS from a template called UCSPEMaster. I just change it depending on how many I need.
1..9 | foreach {
# pad each name with 0s so we have: ucspe08,ucspe09,ucspe10,...
$strNum = [Convert]::ToString([int]$_)
$suffix = $strNum.padLeft(2,"0")
New-VM -VMHost 192.168.1.4 -datastore datastore1 -Template UCSPEMaster \
-Folder 'UCS-Name UCSPE0$suffix -diskstorageFormat Thin \
-Location 'UCS Emulators' -runAsync
}
Notice that each machine will be called UCSPEXX, where XX is the range I specify. That way if
4 more people walk into the class after I’ve configured 14, then I can do 15..8.
I also put all of them on the same host (192.168.1.4), the same datastore (datastore1), the same Folder (UCS Emulators), and made them thin clones.
If you have a cluster running Storage DRS (vSphere 5.0+) then you don’t have to specify the datastore and DRS will put it where it sees fit.
ConfigVMs.ps1
The only thing I need to do now is change the MAC address to something that I have reserved. That way, I can tell the user to just log into the IP address that I’ve set up beforehand. I’ve configured 60 IP addresses so that I’m ready for a big class.
get-vm UCSPE* | foreach {
$number = $_.name.Replace("UCSPE","")
$hexNumber = [Convert]::ToString([int]$number, 16)
$lastPart = $hexNumber.PadLeft(2,"0")
$adapter = Get-NetworkAdapter $_
write-host "Changing MAC address for $_"
Set-NetworkAdapter -NetworkAdapter $adapter \
-MacAddress 00:50:56:00:01:$lastPart -Confirm:$false
}
Most of the script is made up of getting the last number part of the VM name. Since each VM is named UCSPE01-UCSPE60 then it grabs the 60, converts it to hex, then adds that as the last part of the MAC address of the adapter. This will work as long as I don’t have more than 255 VMs.
I could have put a Start-VM on the end of this. Generally, that’s just a powershell one-liner:
Get-VM UCSPE* | Start-VM
removeVMs.ps1
The last script will just remove these VMs. As I tweak around with them, or as people in the class tweak around with them, I just want to erase them and start fresh for the next class. This is fairly easy:
Get-VM UCSPE* | foreach {
Stop-VM $_ -confirm:$false
Remove-VM $_ -DeleteFromDisk -confirm:$false
}
The only thing I need to update this with is to not power it off if it isn’t on. It throws some nasty errors if it’s not powered on. But I’ll do that some other time.
I’ll get this lab working with PowerShell. It’s not a bad language. It’s another tool in the handbag. I still prefer command line scripting with Bash or Perl. But every now and then its fun to go over and see how the other side lives. Now, back to xCAT.
SSH: The ultimate firewall poker
I’m in a hotel tonight and my internet connection is slow. My friend sent me some cool YouTube videos that I wanted to check out and I was wondering if the hotel was limiting the bandwidth of YouTube? A few days ago a friend of mine and I were talking about how to bypass corporate firewalls where companies block facebook/gmail/twitter access. The way to check the YouTube problem and to bypass corporate firewalls can be done using the same solution: SSH.
SSH encrypts data in and out of networks. It is the ultimate firwall poker. If you have SSH capabilities on at least one server on the internet and you can get to it then you can let any traffic in and out of a network. Traffic is all encrypted so no sniffers know anything other than the fact that you have a connection from your machine (inside the corporate firewall / hotel wireless) on port 22 to some remote server that you have access to outside of the network you are on.
So here’s a quick test to see if YouTube was being throttled on this network, provided you have a Mac. I’m doing this on OSX Lion.
1. SSH to some remote internet server. I have one, so I open a port to it:
ssh -D 2011 vallard@myserver.com
This essentially turns your ssh connection into a SOCKS server.
2. Open chrome and navigate to:
chrome://settings/advanced
From here select ‘Change Proxy Settings’ and your Mac settings will come up. Change the Socks settings as shown below:
That’s it. Now you can visit http://whatismyip.com and see that the IP address changed. You can also go to google maps and it will think you are in the city where the server is getting its network access from.
So, what about the YouTube video? Seemed to be a bit better actually… but was hard to tell. Probably was the same…
The other cool thing about this is how if you were in an airport then you can skip annoying adds that come on your web browser then this is how you could do it.
I’ve written in the past about all the cool tunnels you could make with SSH. (See Trick 5 and Trick 6)
The point is: There are ways around corporate policies that block you from checking sites you like, and ways to stop service providers from limiting your bandwidth based on the sites you visit. Now obviously, you have to trust that the machine you SSH into allows you access to all that. But that server is your choice!
SSH through proxy
Problem of the day is I have a computer that is on some local intranet that can not SSH out into the real world. There is however a proxy server on my network that I can configure in my browser to get outside internet access…
But I want ssh. So… after a bit of internet searching and then finally some nagging to a friend who knows this stuff better than I do, we came up with the following:
1. Download connect.c
2. Compile connect.c on your Linux server:
gcc connect.c -o /usr/bin/ssh-proxy-connect
3. Edit /etc/ssh/ssh_config by appending this last line:
ProxyCommand /usr/bin/ssh-proxy-connect -5 -S <proxy-server-goes here>:1080 %h %p
4. SSH normally to where you need to go.
ssh joe@smith.com
That’s it! Once you get SSH through then anything can happen. Its the ultimate firewall poker. Back doors, etc. You just opened up Pandora’s box.
UCS Dashboard
UCS dashboard is a read only tool that came out a couple of months ago. It’s prime objective is to make it easy to manage multiple UCS domains. Today it is read only. Tomorrow… I’m sure it will be able to do more. :-)
It is a free download from Cisco. You can get it here.
The installation instructions are there as well. Here’s my instructions on how I did it that I hope will provide you hints.
Installation
The first instruction of UCS_Dashboard is to unzip the file. There are three files that you have to download. Now all of you may find that easy, but as a Linux user I’m looking around trying to figure out how to unzip those three files.
Turns out that once you download them, you need to cat them together and then unzip them. So for me it was:
cat UCS_Dashboard_v1_0.z01 UCS_Dashboard_v1_0.z02 UCS_Dashboard_v1_0.zip >UCS_Dashboard.zip
then
unzip UCS_Dashboard.zip
You’ll then get the following nice message:
Archive: UCS_Dashboard.zip warning [UCS_Dashboard.zip]: zipfile claims to be last disk of a multi-part archive; attempting to process anyway, assuming all parts have been concatenated together in order. Expect "errors" and warnings...true multi-part support doesn't exist yet (coming soon). warning [UCS_Dashboard.zip]: 650117120 extra bytes at beginning or within zipfile (attempting to process anyway) file #1: bad zipfile offset (local header sig): 650117124 (attempting to re-compensate) creating: UCS_Dashboard_v1_0/ inflating: UCS_Dashboard_v1_0/UCS_Dashboard_v1_0-disk1.vmdk.gz file #3: bad zipfile offset (local header sig): 241444728 (attempting to re-compensate) inflating: UCS_Dashboard_v1_0/UCS_Dashboard_v1_0.mf inflating: UCS_Dashboard_v1_0/UCS_Dashboard_v1_0.ovf
They’ve been saying coming soon since 2003 (at least!). Anyway, don’t worry about the errors. They’re fine.
The next thing to do is deploy this OVF file. This can be done from vCenter. I ran into a problem when I did this:
Unable to load DLL 'zlib1.dll': the specified module could not be found
I then googled this and found a free zlib1.dll file that I could download and install on the vCenter server. It looked like very cheezy software and I really didn’t want to install it. The alternative would have been to unzip it from the linux command line and then figure out a way to modify the ovf file. But, I just installed the zlib and things went fine.
The host then booted up and I saw a familiar CentOS splash screen. My file was up.
Log In
The Login User is root, the password is cisco123. But you won’t need that. Cisco likes to make it easy for you. I couldn’t help myself so I immediately went in and configured the IP address in /etc/sysconfig/network-scripts/ifcfg-eth0.
But the right way is to just login as ‘ucsadmin’ with password ‘ucsadmin’. It will immediately start a script and ask you for the admin password. Here you enter ‘admin’.
Then you can configure the IP address. Nice.
Configuration
You’ll have to create a read-only account on UCS for the Dashboard to log into. You have to do this in good old fashion UCSM java client or the command line. (Note: This threw me for a loop. Don’t try to check the “read-only” box. All users by default are read-only 
). (Note2: command line is left as exercise to the reader)
Navigate your web browser to the IP address of the UCS Dashboard. From here you’ll be tempted to immediately add your UCSM. Don’t! First create a domain and group from the configuration menu. Its not as obvious at first. Took me a while to find it, but then you’ll get it. From there you can add your UCSM back on the “Manage Inventory” side.
Now what?
Well, now you’ve got a way to view your service profile information and all the details of multiple UCS systems. Not bad right? Its read-only, but we’re just getting started!
Hope that helps!
xCAT r* commands with UCS
xCAT out of the box works on UCS. Or UCS out of the box works with xCAT? Whichever way you look at it, it works. All of the cool things you can do with xCAT like provision nodes, KVM, vSphere, stateless computing, etc, can all be done with UCS. In fact, you can even run most of the r* commands on UCS.
Cisco UCS allows this through IPMI. And configuring IPMI on UCS is easier than any other system I’ve ever used. While I still plan on furthering my xCAT UCS plugin to get more capabilities into xCAT, most xCAT functions can be used with UCS managing the servers with IPMI. For most people, this is good enough.
Using IPMI this is what seems to work with xCAT 2.6.6 and UCSM 2.0(1): (See the end of this for sample output)
- rpower on|off|stat|boot
- rbeacon on|off
- reventlog [clear]
- rvitals (this is quite thorough)
rinv seems to hang on me. This I think is due to the nature of service profiles, where UUIDs and MAC addresses are transient. I’ll investigate this further.
So how do you do it?
Configuring an IPMI machine with xCAT has been well documented. What I haven’t seen documented so much is configuring IPMI inside UCS. This is surprisingly easy. Here’s how its done:
1. Create a Service Profile Template that you will apply to your blades. This is documented very well in various places so I won’t go into it here. Creating a Service Profile Template is UCS 101. After you’ve created your service profile, assuming its an updating template you can proceed to the next step. (Don’t worry, any changes made for doing IPMI don’t require a reboot)
2. From the Servers tab, filter by Service Profile Templates, and navigate to your service profile template.
3. Click on the policies table and look at the IPMI Access Profile Policy
4. Create a new policy. In this policy you’ll give the name of the user and give it a password. Make sure they have admin privileges. For simplicity, I just made my user and password the same as my UCSM user and password.
5. Apply the setting and click save.
From here on out you can just run IPMI commands. The only issue now is to know which IP address corresponds to the IPMI interface of which blade.
This can be found in UCSM under the Admin tab, Communication Management, Management IP pool. If you click on the IP addresses tab on the left hand side, you’ll see all the IP addresses.
Ok my friend, you now have it. xCAT running rpower commands.
And now, here is a sample output running rvitals on a UCS B200 M1:
# rvitals lucky01 lucky01: BIOSPOST_TIMEOUT: N/A lucky01: BIOS_POST_CMPLT: 0 lucky01: CATERR_N: 0 lucky01: CPUS_PRCHT_N: 0 lucky01: DDR3_P1_A1_ECC: 0 error lucky01: DDR3_P1_A1_PRS: 0 lucky01: DDR3_P1_A1_TMP: 26 C (79 F) lucky01: DDR3_P1_A2_ECC: 0 error lucky01: DDR3_P1_A2_PRS: 0 lucky01: DDR3_P1_A2_TMP: 25 C (77 F) lucky01: DDR3_P1_B1_ECC: 0 error lucky01: DDR3_P1_B1_PRS: 0 lucky01: DDR3_P1_B1_TMP: 26 C (79 F) lucky01: DDR3_P1_B2_ECC: 0 error lucky01: DDR3_P1_B2_PRS: 0 lucky01: DDR3_P1_B2_TMP: 27 C (81 F) lucky01: DDR3_P1_C1_ECC: 0 error lucky01: DDR3_P1_C1_PRS: 0 lucky01: DDR3_P1_C1_TMP: 24 C (75 F) lucky01: DDR3_P1_C2_ECC: 0 error lucky01: DDR3_P1_C2_PRS: 0 lucky01: DDR3_P1_C2_TMP: 25 C (77 F) lucky01: DDR3_P2_D1_ECC: 0 error lucky01: DDR3_P2_D1_PRS: 0 lucky01: DDR3_P2_D1_TMP: 22 C (72 F) lucky01: DDR3_P2_D2_ECC: 0 error lucky01: DDR3_P2_D2_PRS: 0 lucky01: DDR3_P2_D2_TMP: 22 C (72 F) lucky01: DDR3_P2_E1_ECC: 0 error lucky01: DDR3_P2_E1_PRS: 0 lucky01: DDR3_P2_E1_TMP: 22 C (72 F) lucky01: DDR3_P2_E2_ECC: 0 error lucky01: DDR3_P2_E2_PRS: 0 lucky01: DDR3_P2_E2_TMP: 22 C (72 F) lucky01: DDR3_P2_F1_ECC: 0 error lucky01: DDR3_P2_F1_PRS: 0 lucky01: DDR3_P2_F1_TMP: 21 C (70 F) lucky01: DDR3_P2_F2_ECC: 0 error lucky01: DDR3_P2_F2_PRS: 0 lucky01: DDR3_P2_F2_TMP: 22 C (72 F) lucky01: ECC_STROM: 0 lucky01: FM_TEMP_SENS_IO: 21 C (70 F) lucky01: FM_TEMP_SEN_REAR: 22 C (72 F) lucky01: HDD0_PRS: 0 lucky01: HDD1_PRS: 0 lucky01: HDD_BP_PRS: 0 lucky01: IOH_THERMALERT_N: 0 lucky01: IOH_THERMTRIP_N: 0 lucky01: IRQ_P1_RDIM_EVNT: 0 lucky01: IRQ_P1_VRHOT: 0 lucky01: IRQ_P2_RDIM_EVNT: 0 lucky01: IRQ_P2_VRHOT: 0 lucky01: LED_BLADE_STATUS: 0 lucky01: LED_FPID: 0 lucky01: LED_MEZZ_FAULT: 0 lucky01: LED_MEZZ_TP_FLT: 0 lucky01: LED_SAS0_FAULT: 0 lucky01: LED_SAS1_FAULT: 0 lucky01: LED_SYS_ACT: 0 lucky01: MAIN_POWER: 0 lucky01: MEZZ_PRS: 0 lucky01: P0V75_DDR3_P1: 0.7644 Volts lucky01: P0V75_DDR3_P2: 0.7644 Volts lucky01: P12V_BP: 11.948 Volts lucky01: P12V_CUR_SENS: 10.78 Amps lucky01: P1V05_ICH: 1.0486 Volts lucky01: P1V1_IOH: 1.078 Volts lucky01: P1V1_VCCP_P1: 1.0192 Volts lucky01: P1V1_VCCP_P2: 0.931 Volts lucky01: P1V1_VTT_P1: 1.1368 Volts lucky01: P1V1_VTT_P2: 1.1564 Volts lucky01: P1V2_SAS: 1.2152 Volts lucky01: P1V5_DDR3_P1: 1.5288 Volts lucky01: P1V5_DDR3_P1_IMN: 5.13 Amps lucky01: P1V5_DDR3_P2: 1.5386 Volts lucky01: P1V5_DDR3_P2_IMN: 14.25 Amps lucky01: P1V5_ICH: 1.5092 Volts lucky01: P1V8_IOH: 1.813 Volts lucky01: P1V8_P1: 1.7836 Volts lucky01: P1V8_P2: 1.7836 Volts lucky01: P1_PRESENT: 0 lucky01: P1_TEMP_SENS: 39.5 C (103 F) lucky01: P1_THERMTRIP_N: 0 lucky01: P2_PRESENT: 0 lucky01: P2_TEMP_SENS: 37.5 C (100 F) lucky01: P2_THERMTRIP_N: 0 lucky01: P3V3_SCALED: 3.2548 Volts lucky01: P3V_BAT_SCALED: 3.102 Volts lucky01: P5V_SCALED: 4.9405 Volts lucky01: POWER_ON_FAIL: 0 lucky01: POWER_USAGE: 126 Watts (430 BTUs/hr) lucky01: SAS0_FAULT: N/A lucky01: SAS1_FAULT: N/A lucky01: SEL_FULLNESS: 0 lucky01: VR_P1_IMON: 1.75 Amps lucky01: VR_P2_IMON: 3.5 Amps
UCSM: Changing IP addresses
You just got a UCS system and now you want to install it. You configured it, but now you realize you need to change the IP addresses. You could rerun setup and it would take about 15 minutes to reboot and reconfigure. Not the most efficient use of your time. What do you do?
You just change the IP addresses silly! Normally you can just do this via the web interface. But for fun, you want to do it via the command line. Here is how its done:
scope fabric-interconnect a set out-of-band ip ip-address netmask netmask gw gateway-ip-address scope fabric-interconnect b set out-of-band ip ip-address netmask netmask gw gateway-ip-address scope system set virtual-ip vip-address commit-buffer
Too easy!
UCS Login Banner
One of the cool things in UCS 2.0 is the ability to display custom banners (or MOTDs) when people log into the system. How to do this?
- Admin Tab
- User Services
- Banner Tab
- Create Banner
Now when the user logs in they see a fun message from you. Pretty simple, but thought I’d share anyway.
UCS 2.0 Software Update Notes
Today I updated my Cisco UCS system from 1.4(3q) to UCS 2.0(1m). I’m happy to say it was an easy process. First lets talk about why you want to do it, then lets talk about how to do it :
Why upgrade to 2.0?
- Disjoint Layer 2 network support
- iSCSI boot support with service profiles
- RedHat KVM support for VM-FEX
- Configuratble pre-login banner (for GUI and CLI)
- Support for vSphere 5 including auto-deployment with stateless computing, VMDirectPath with vMotion
- HDD operability indication without host agents.
Sound like something you want? Yes? Ok, lets do it:
1. Get UCS 2.0 software
- UCS Infrastructure Software Bundle. Click to download.(ucs-k9-bundle-infra.2.0.1m.A.bin)
- UCS B-Series Server Software Bundle for UCS Manager. Click to download.(ucs-k9-bundle-b-series.2.0.1m.B.bin)
- UCS C-Series Server Software Bundle for UCS Manager. Click to download.(ucs-k9-bundle-c-series.2.0.1m.C.bin) (only get this if you have rack mounts)
Most likely you’ll only need the first 2 packages. One for the infr
2. Get the UCS software update guide
You get this here. There’s lots of versions to upgrade from. Yours is there, trust me, cause we’re thinking of you first. From the update guide you can update the firmware in a cluster configuration without any disruption of the network. The blades will need to be rebooted for firmware updates to take place, but you probably knew that.
From 1.4 to 2.0 the order is shown as follows:
- Disable Call home
- Adapters (Activate / ignore compatibility / set startup only)
- CIMC (Activate / ignore compatibility)
- IOMs (Activate / ignore compatibility / set startup only, don’t reboot)
- UCS Manager!
- FI subordinate (Activate / ignore compatibility)
- Verify step 4 is done, that IOMs connected to FI subordinate have been updated, traffic is passing through
- FI primary (Activate / ignore compatibility)
- Update Host firmware for servers
- Reenable call home
That’s how I plan to do it. Lets see how it all turns out.
3. Upload Firmware to UCS Manager
From the Equipment Tab select Equipment. Then on the right go to the Firmware Management Tab, then click Download Firmware.
In our case we did it for the two packages. The Infrastructure bundle and the Server bundle. I also used this time to get rid of some of the older firmware bundles. I kept the one that we had been using and deleted all the other ones. I like my UCS Firmware image list clean.
4. Update CIMCs
Choose Update Firmware from the Firmware Management Tab. Select CIMC from the top and then select 2.0 then set apply.
From here go to the Activate button, filter CIMCs and then update all of them to 2.0 checking the ignore compatibility button.
Stay on this screen for a while and you’ll see the CIMCs apply the update, then reboot and eventually show the running version equal to 2.0(1m) and show ready.
5. Update IOMs – Set as Backup, don’t activate
Go to the Update Firmware menu again, filter with IO Modules, then select 2.0.
Going to the Activate button, now set Activate, Ignore Compatibility Check, and Set Startup version only. This will make it so they only reboot when the Fabric Interconnect reboots. Thus saving you a few reboots.
6. Update UCSM!
Can’t wait to see this part! Go to Activate. Hit apply. In a minute or two you’ll be disconnected.
About 1 minute after that open your web browser to the screen and behold UCSM 2.0!
Yes, that just happened to you! Lets log in. And we’re back. Things look mostly the same, but you’ll see a new icon here and there that show the new features. For example in the VM tab you’ll see the cluster icon for KVM VM-FEX. Under the LAN tab you’ll see iSCSI Initiator Pools. Let’s move along and finish updating.
7. Update Fabric Interconnects
Same place we were before: Equipment table, Firmware management tab, and Activate firmware. Filter for Fabric Interconnect and this time, set the start up version of the subordinate fabric interconnect. Ignore compatibility and then hit ‘Apply’. You’ll get a message that you will be logged out, but if you are doing this to just the subordinate, then nothing to worry about here.
You can remain on that screen as the kernel and system firmware updates take place. No big deal. This one took about 12 minutes before it updated in the view. What’s going on in the background? Well, the FI is downloading the new firmware, then applying, then rebooting. You can watch the Finite State Machine (FSM) in the Fabric Interconnect table too as you go.
Once you are back up, then its time to update the primary fabric interconnect. Nothing strange here. Same thing. If you’re still on the same screen, just repeat those steps:
In a few minutes, you’ll be disconnected from UCS and Fabric Interconnect B will take over the UCS manager. You can still log back into the same address. Then you can see where you are in the state of things:
In a few minutes, that will be updated and all will be well with your UCS chassis. You may notice that FI-B becomes the primary and FI-A becomes the subordinate. If that bugs you, change A back to be the primary.
There’s only one step left to go!
8. Update the Blades
You may not want this to happen all at once. What I recommend is doing this at a Service Profile Template level. Set the soul of the server to be the firmware level you want. Then as new blades are added, they’ll always get this level.
At the server tab, you’ll see the service profile templates tab where you created your service profile templates. Click on the Policies tab and create a new firmware policy for nodes in this group.
I just named mine 2.0.1m and clicked all the applicable fields for my hardware. I made one for the host firmware and the management firmware. Then I set them to my policy. Upon saving I get a nice message:
Since my policy is set to ‘user acknowledge’. The blades won’t reboot immediately, but instead I’ll get a nice blinking ‘Pending Activities’ notification. You may want to make sure this policy is set before saving. This can be done by going to the general tab under the service profile and selecting ‘Change Maintenance Policy’. Since I live dangerously, I reboot my live VMware cluster but I do half of the servers at once. This way, the VMs migrate. You can watch it in vCenter. Very fun. Then when those nodes are back up and integrated into vCenter, we reboot the other half of machines.
That’s all folks!
That was painless and easy. If you did that on any other legacy blade platform, or legacy rack mount it would take you hours. Keep in mind with legacy systems for each chassis you need to update
- the network switches
- the onboard administrator
- Each blade in the chassis individually including: IPMI interface, NIC, BIOS, Raid Controller, and Disk Drives.
This took 1 hour and all I did was click buttons, take screen shots, and write things down. No waiting for machines to reboot and hitting F2 at the right moment. Everything only rebooted once and it was easily orchestrated. I don’t have to guess if things were done correctly because I can look at my single pane of glass on UCSM and see that everything is updated.
Doing things the UCS way scales big time. While we only did 4 blades here, it would have taken the same amount of time to do 12o servers. No additional infrastructure was required to do it.
Doing things the old way in a legacy blade environment is like writing assembly code. As an undergraduate, we had to write a stack calculator using the MIPS instruction set in CS61c. It took hours and about 15 instructions to do one thing. UCS is like a higher level programming language for your data center. You do things quicker, and more efficiently. Even today compilers can make high level programming languages more efficient than you probably can doing it in assembly. And when there are so many other cool things like VMs, clouds, self service catalogs, etc… why waste time writing assembly language?
1. Get your basic wiring down:
2. Get management IP addresses
Make sure you have at least 3 IP addresses for the Fabric Interconnect (FI) management network. You will also need an IP address for each blade’s CIMC (translation: From HP this is the iLO IP address, From IBM this is the IMM IP address). The CIMCs and the 3 IP addresses for the FIs should be in the same subnet/VLAN. If I had 8 blades then I would need 11 IP addresses.
It is easy to get confused on this because people look at the management port coming from the back of the Fabric Interconnect and think its just the management port of the Fabric Interconnect. However, internally there is an ASIC that performs level 2 switching so all the blade CIMCs go up through this port as well. That’s why you need to keep this on the same network and VLAN.
3. Cable the chassis to the Fabric Interconnects.
We usually put all the ports on the left side of the back of the chassis up to the top fabric interconnect and all the ports on the right side to the bottom fabric interconnect. When I first saw this I thought: Where is the redundancy? It seemed to me that each I/O module in the back of the chassis should be redundantly cross connected to each FI. Well, that’s not how it goes. Look at it like those cables going from the I/O modules (IOMs) to the FIs are just part of the backplane. Its just that UCS externalizes the backplane to be able to manage more chassis with one management infrastructure. (One of the big selling points of UCS! No mini-racks!)
4. Connect 10GbE uplinks
The 10 GbE connections from the FIs to the main switches can cross connect and you can create a VPC across them. This should get you started.
5. Configure FI-A
With my Mac, I connect a USB serial adapter to the serial cable to the serial port on the top FI. (which I call FI-A) From there I run:
screen /dev/tty.usbserial
and press enter. Then I’m in. I still have issues with this on my Mac where I need to change USB ports to make it work. But eventually, I’m always able to get something.
From here I get:
Enter the configuration method. (console/gui) ? console Enter the setup mode; setup newly or restore from backup. (setup/restore) ? setup You have chosen to setup a new Fabric interconnect. Continue? (y/n): y Enforce strong password? (y/n) [y]: y Enter the password for "admin": Confirm the password for "admin": Is this Fabric interconnect part of a cluster(select 'no' for standalone)? (yes/no) [n]: yes Enter the switch fabric (A/B) []: A ...
Then it asks for IP address information, etc. Its pretty easy. It also asks for a system name. I usually give it a name like UCS0001, so that you can buy up to 9,999 UCS systems before you have to change your naming convention. ;-) Each FI gets its own IP address and then there is a Virtual IP address that they both share (remember: this should be on the same VLAN, and this is why you need 3 IP addresses for the FIs).
The system will reinitialize and settings will take effect. Make sure you can ping this IP address before moving on. You may get an error like: ”Clustering services are not available on other fabric interconnect”. Don’t worry, just move along.
6. Configure FI-B
Same thing as the first, but when you first log in, you’ll give it an IP and netmask and then it will ask you to join the other FI in its cluster. You’ll enter the password and you’ll be up!
7. Log in!
Point your browser to the Virtual IP address and you’ll open a Java JNLP program and start administering your UCS system. Wasn’t that easy?
Now what?
This section deserves a blog of its own. Basically, you’ll start setting up policies and pools, VLANs, VSANs, etc. Then create a service profile templates from those resources. Once this is complete, you are ready to install your OS and BAM! You are up and running.
As anyone new to Cisco can tell you, joining this company is no game of candy land. But I tell you, it is an exciting ride! We lovingly call the first couple of weeks for a new hire ‘the firehose’. Well, during the last three months that I’ve been at Cisco, I’ve been drinking from the firehose and focusing very heavily on the technical aspects of the UCS product line. Learning its nuts and bolts, kicking its tires, and even setting up a new system in our internal lab. Its been very exciting. I’ve been learning a ton, and can pretty much talk the value of UCS to anyone.
But this week when I was asked to give a general overview of Cisco’s vision of the Data Center lets just say I did a less than stellar job. When you’re down in the blissful technical trenches, its easy to talk all day about trade-offs of remote storage alternatives. But talking about it like you’re standing on the moon looking down at the earth is a different conversation all together, and one that I should have been more prepared to discuss.
I’ve thought about it some more and I’ve tried to break it down into something that is truly differentiating and doesn’t reek of generalities. To do that, I’ve had to frame it in the context of what is happening in the industry today. There are three parts:
1. What the future data center is doing
2. Why were not there
3. How Cisco will get us there.
The Future Data Center
Its about applications
The way we use technology to solve problems, to be more productive, and how to do it more efficiently is what IT has been about and what it will be about in the future. We do this today using fixed purpose applications: Microsoft Office Products, Email, Custom Apps that leverage the organizations data repositories, etc. The vision of most technology firms is that of a future even more focused on applications. Just ask Google, Apple, VMware, and Cisco.
In the future, we will still compose documents, but what will we use to do this? It used to be papyrus and lamb’s blood. Then we moved to ink, then the typewriter, and today in business it largely consists of Microsoft Word and some to Google Docs. Whatever our data center of the future looks like, it will be purposefully built to support our applications.
Applications are moving to the cloud or SaaS based
At VMworld 2011, VMware CEO Paul Maritz talked about how most applications today are being written by people younger than 35 and that they don’t feel the needs to adapt to the mold of software that has been written previously. Most of this software will be delivered using new frameworks, methodologies, etc. In essence: applications are being moved from being installed on your desktop, to being delivered over the network, or ‘the cloud’. Examples: Salesforce.com, Google Docs, 37 Signals apps, etc. Data is no longer stored locally on the end device. Its stored in a data center.
Why? To name a few:
- distribution cost = 0
- installation time = 0
- time to update the application on the users device = 0
- fraction of support costs, because lets face it: a lot of our support problems stem from users not being able to install the applications.
In essence: Once the SaaS app is created, it cost little to support. For the user: He can access it on any device, nearly any where, at any time. This is part of Cisco’s borderless networks vision. BYOD: Bring your own device. Where Cisco is focused on this is really making people feel secure about accessing sensitive data from anywhere on any device.
So what is the data center of the future doing? It’s hosting apps that can be run anywhere. If the apps can be run anywhere, then guess what? The apps can be hosted anywhere. The data center of the future is communicating with other data centers of the future. Its been called the Inter-Cloud. And much like Cisco paved the way for the Internet, Cisco will pave the way for the Inter-Cloud.
(Note: Do not confuse Inter-Cloud with private cloud. Inter-Cloud is like the Internet, but instead of many computers connected, there are many data centers connected. Those data centers will need to figure out how services like DNS, DHCP, etc will work across them. We’re still a ways off)
Why we’re not at the Future of the Data Center today
If your datacenter is doing nothing but hosting SaaS based apps most organizations will not be able to do it as cost effectively as cloud providers. (Amazon EC2, Microsoft Azure, Google App Engine, etc). This will only be more true as service providers continue to be more efficient. Some believe that Public Clouds will win the game.
I don’t think its that cut and dry. Certainly over the next 10 years I doubt larger organizations will be willing to give up their entire data center. So it will be a hybrid. And much like you can put solar panels on your house and give back to the power grid, you’ll be able to host workload on your own data center and seamlessly migrate it to the Public Cloud, or Inter-Cloud.
So what do we need to overcome to get us there?
- Security and Reliability
- Apps aren’t completely SaaS based.
- Still not as cost effective
- Transition costs
- Future protocols
Let’s talk about each point:
Security and Reliability
How many times this year have you read a headline about how some company’s data was compromised and now millions of people’s personal information is floating around in some cyber terrorists briefcase? I don’t think I need to dwell any more on that. Companies just don’t feel safe enough. Many individuals don’t feel safe enough. My wife won’t even use Facebook.
As for reliability, remember how EC2 had a major crash and caused outages in major web sites like FourSquare, Reddit, etc? What if 911 was running on it? I remember a story when some peers of mine were upgrading a network for the city. The customer asked them: “If we upgrade this now, are you sure we’ll still remain up?” The engineers were very sure. They’d done it before and had no issues. The customer then asked: “Would you be willing to bet people’s lives on it? Because if this goes down, 911 will be affected, and people can die”. The engineers decided to wait for a maintenance window. No matter how many 9′s of availability you have, you’ll always have a 1 with 0′s in front of it of non-availability.
In the future, if these security concerns can be overcome, we won’t care where the workloads are running, just as long as they know they are secure. Just like you don’t know exactly where your water comes from as long as you know its safe to drink.
Apps aren’t completely SaaS based
Think about how long it took from apps to migrate from the Main Frame to the client/server model today used by many desktop applications. Now forget about that number, because if you believe in the singularity, then the rate of change of technology is only increasing. For Apps to move from the client/server model of today to a SaaS based model it will still take some time. And if you’re not completely SaaS then you need to keep supporting desktops that have older clients on them.
This brings up an interesting side note about Virtual Desktop Infrastructure (VDI). In essence, if what I have written going forward is true, then VDI is just a temporary solution for many places. You won’t need a corporate approved desktop if you’re just running SaaS based apps. You just need something that boots up and has secure access. Today that is done with 0 clients, or thin clients. But guess what? Those thin clients will just be running apps. No need for cumbersome operating systems.
Personally, I can’t wait until I no longer need my Windows XP Virtual Machine to run just 2 or 3 apps. It is an immense source of pain to me. I can’t believe it took VMware this long to make a client for vCenter that didn’t require Windows!
Still not cost effective
Amazon and others tote the cost savings of outsourcing your data center. This is true for small and even some medium size businesses. But there comes a critical mass where outsourcing the data center, is less cost effective then having your own in house.
Zynga started out using EC2 but as the games got bigger, it was too expensive and so Zynga invented its own Z cloud. Today Zynga still uses EC2 for small tests or games that it is just launching, but for the real popular games, its just too much cost and they run on their own internal cloud.
Transition Costs
It takes too much work right now to move your app to the cloud. Where would you move it to? And if you moved it to Amazon, then isn’t that vendor lockin ? Where else could you go? The market just isn’t mature enough. So no thanks, we’ll keep our apps in house.
Protocols and Standards
What is the definite way to migrate a VM between Data Centers of different suppliers? What are the standard security settings needed on both ends to complete the handshake? How do data centers get certified that they can work together? Much of this still needs to be ironed out and I think you’ll find Cisco right there helping. The latest announcement at VMworld 2011 with VXLAN is a great start. (More on that in a second)
So even though public clouds aren’t extremely attractive today for many organizations, the concepts of a cloud are extremely attractive. Charge back, self provisioning, accounting can transform a data center into an extremely lean, efficient, transparent, and agile operation. That is why private clouds are being built. As private clouds mature, they’ll be able to share application hosting seamlessly through different data centers. That’s what the Inter-Cloud is all about and is what will give rise to our future public cloud model. This is the future of the data center.
The Road Map to the Data Center of The Future
How do we get to the Inter-Cloud? First start reaping the benefits of the private cloud. As more people do this, more technology will be developed, standards will arise, and we’ll be ready for the data center of the future and the Inter-Cloud. Cisco is already doing this.
At VMworld 2011, Cisco announced VXLAN which can enable virtual machines to migrate between physical servers in different subnets. Cisco has already been clarifying the gray area between the network administrator and the virtual server administrator with the Nexus 1000V. The only non-VMware dVS.
That’s all great stuff. But what should your organization be doing to future proof your data center?
Follow this roadmap:
- Begin SaaS based solutions transition
- Consolidate Hardware Infrastructure
- Virtualize infrastructure
- Build Private Cloud
- Hybrid Cloud
- The Inter-Cloud
This is how Cisco will get you to the Data Center of the Future.
1. Begin SaaS based solutions transition
Most organizations have already been doing this. Our applications are a combination of ones that must be installed on the desktop to ones that run straight from the Internet.
Universities like the University of Louisville and Case Western Reserve Institute completely outsource their mail to Google. The admins love it! No more headache’s with Exchange. No need to store all that email. The students love it too because its just like the GMail accounts they’re used to using.
Just like Google has solved the mail problem for hundreds of organizations, there other innovative companies have done this with other solutions like Payroll, accounting, CRM, etc. These applications will only mature and become more compelling.
Cisco itself offers a great SaaS based solution called WebEx for meetings and other collaborative products. Starting to use solutions like this is how organizations begin to transform from old world applications to the applications of the future.
Phase 1 summary: Get acquainted with and consider SaaS technologies: Try WebEx. Most likely, you’re already doing this.
2. Consolidate and simplify Hardware Infrastructure: Get control!
Internally as we look at our present day data center, it is siloed, static, and inefficient. There are all kinds of cables and networks intermingled causing massive overhead for IT. This infrastructure needs to be simplified and consolidated. According to a report last year, organizations spend 38% on IT personal. Thus, making these people more efficient is one of the fastest way to reduce operational costs.
Cisco offers its Unified Fabric solutions to help in this. It has two big impacts: First, it reduces management and cabling complexity for now you don’t need to buy as many devices. Second, it allows for multitenency. Fabric’s can be separated logically. Consider Cisco’s pioneering of technologies like VSANs where you essentially have multiple SAN devices running on a single physical SAN switch. This is a layer above zoning and once again collapses the need for multiple devices. With Unified Fabric, you only need to wire it once, then you can run whatever protocol you want over the network, whether it be Fibre Channel or Ethernet.
Unified Computing System or UCS is the other and important aspect of simplifying the management of hardware infrastructure. A single management tool, UCS manager allows you to do things you can’t do with any other server management platform. Administering legacy Blade architectures (anything non-UCS at this point), or rack mount servers is like writing assembly code to develop a web page. You may be able to do it, but using higher level programming languages like HTML and Javascript to do it is much more efficient. With UCS you’ll find something that used to take you 15 steps, now takes a single click. You’ll feel that you were writing assembly code. As an example, you’ll find you can update firmware on 120 servers with a single click. Think anyone else can do that today? Nope.
Phase 2 summary: Get your hardware under control. Consolidate with Unified Fabric & UCS
3. Virtualize the Infrastructure
The hardware capabilities of one physical server far excede the requirements of most applications. Indeed, they exceed the requirements of an entire operating system. Even my laptop is able to run 2 operating systems without even feeling a hiccup.
The next phase is to virtualize that infrastructure. With UCS, you’re most of the way there. But Cisco can make it even easier for you. We offer two major converged architecture solutions that enable you to have virtual machine appliances. They are called Vblocks and FlexPods.
With Vblock, Cisco has teamed up with VMware and EMC to create a virtual machine appliance. We’ve even created a company around it called VCE. VCE has its own engineers and support organization who create solutions based on this virtual machine appliance. They offer best recipes for all types of solutions. With VCE, you can order a single SKU, it comes to you preconfigured, and within a week you will be up and running. Try doing this with any other single vendor, or collection of vendors and it can take you nearly 100 days.
A Vblock gets you the best of breed architecture with the worlds leaders in storage, networking, and virtualization.
A FlexPod is a reference architecture with NetApp, Cisco, and VMware. NetApp is the fastest growing storage company and its innovative approach to storing data with its RAID-DP technology, De-duplication, and snapshotting make it a very attractive storage vendor of choice for many organizations. NetApp even offers a money back guarantee that if you don’t reduce your storage by 50% then they’ll give you storage for free. To date, they’ve never paid that out.
A FlexPod can come preconfigured with everything ready to go. Or you can buy the components separate, then register it with Cisco, NetApp, and VMware and we’ll tie the support on the back end.
Once you have started to virtualize your primary applications, its time to get serious and start virtualizing the serious stuff: databases and other critical apps. With VMware Fault Tolerance, your applications will be more reliable then they were on physical machines.
Phase 3 summary: Virtualize the infrastructure with Vblock or FlexPod
4. Build a Private Cloud
With your organization virtualized, you’ll already start seeing great advantages to managing your infrastructure. But it gets better. At this point the administrators are still provisioning machines. Its time to free them to move on to bigger and better things. Its time for the private cloud.
In this step, you automate the business processes for IT. You’ll build a catalog of services for organizations that can easily be provisioned and allocated on demand. This includes networking, virtual machine images, vApps, etc. You build capabilities to charge time to different departments for what they use. You start to bring incredible transparency to your IT organization. This is a good thing. Now people can see how valuable IT is and you may find IT is allocated more budget dollars.
In the last two years, Cisco made two incredible acquisitions. new Scale and Tidal. Since then, Cisco has adopted these companies in, combined their technology and created Cisco Intelligent Automation for Cloud.
With Cisco Intelligent Automation for Cloud, you get the only tier one server vendor who’s built a cloud solution from the ground up, and not just a rebrand of existing technologies. Its tested on the market and will only improve over time.
Phase 4 summary: Build the self service catalog with datacenter automation with Cisco Intelligent Automation for Cloud.
5. Hybrid Cloud
The term ‘Cloudbursting’ can refer to the situation when an organization receives spikes in demands for its cloud and must ‘burst’ the workload to a public cloud. This is the next step on the journey to the data center of the future. Your data center begins to establish relationships with other data centers.
Standards are still being developed to handle this stage. There are some interesting solutions in the open source field like Open Stack, and Cloud Stack that may aid with this journey.
Phase 5: Begin flirting with cloud service providers. Become a power user of Cisco Intelligent Automation for Cloud
6. The Inter-Cloud
Much like Cisco helped pioneer the Internet, and the Internet was built on top of Cisco networks, the Inter-Cloud will be built upon Cisco’s network. As we get to this exciting destination those who have taken the leap to the data center of the future will literally leave behind those who never started the journey.
“He who rejects change is the architect of decay. The only human institution which rejects progress is the cemetery. ” ~Harold Wilson
Phase 6: Become a part of the Inter-Cloud
Summary
I hope this article has been useful for any organization looking to understand the current data center landscape and see what an innovative company like Cisco is doing to aid people looking for solutions. In the end, building clouds and data centers is all about how we can help people work together more efficiently. This is exciting because as people are able to work better, more innovation will take place at an accelerated pace. Imagine as a result of technology the types of problems we’ll be able to solve both scientifically, biologically, politically, and organizationally. Cisco has a cool slogan about how “Together, we are the Human Network”. I think its safe to extend that to say: “Together we are the Cloud.”
If there is anything I missed here or corrections, please let me know. I’d love to get your thoughts.
