At the Gartner data center conference in December 2010 they reported that 38% of IT costs are personnel. Given that almost 40% of your cost in IT is personnel, it would make sense to invest in tools and equipment that make it so they can be as efficient as possible. I’d like to illustrate how UCS decreases costs by making employees more efficient. Instead of high level over views like you may be accustomed to reading, let me give a hands on analysis and illustrate just one feature in UCS that can save several thousand dollars.
Consider this example:
You just ordered 300 non-UCS blades and they just arrived to your datacenter. You will now need to set the BIOS in these servers to enable Intel VT because you want virtualization right? How do you do this? Your entry level systems engineer that needs to do this is getting paid $64,000 a year.
A. The Dumb way) You will now need to boot each blade up, press F1 (and hope you don’t miss that screen!), go to through the menus, select the right place, save, and reboot. That probably took 5 minutes if you’re good. (Boot times on blades and rack servers are about 2 minutes given all the fancy hardware checks). That was easy, took little thought and it cost you $775 to get it done. ($64k /52 weeks / 40 hours/week =~ $31/hour). Except, now you boot up and find that he missed a few. Oh, and then you find out that “Surprise!” you didn’t enable VT for Directed IO. Bummer. So now that cost you another $775 to change it. So solution A only cost $1550. …Except, you forgot to factor in opportunity cost. Our entry level engineer could have been doing something else productive. So your real cost is about $3100 AND, you still don’t know if everything is right do you? If you want to know what the BIOS is set to, you can just try it blindly, or you can reboot, press F1 and take a look. This is a great strategy for 1 server. If you have more, then this is just plain dumb. Also, because of the difficulty of changing this, you may miss out on the benefits of other BIOS settings like Turby Boost. So the best you can do with this is pay $1550. Thats just a drop in the bucket for 300 servers isn’t it? But does it sound painful? Yes. And I can tell you from experience, it does nothing to improve moral.
B. The smarter way) Many vendors include command line tools with their systems. So you can get a smarter person who knows how to do this stuff via the network. This way, you don’t have to wait for the systems to boot up. In some cases you can configure it through the BMCs. There are a few drawbacks with this solution:
– You have to run some sort of script again to see what state your BIOS settings are in. I have often done this in the past with an awesome perl script, usually within the xCAT framework.
– It turns out vendors still don’t implement this right. Often it will require several reboots for this to be correct.
– It still takes a lot of time to set up and that setup is usually outside the framework of the application that manages the blades. So now you have to document this and make sure you have it right. Oh, and skills transfer to others in the organization? Probably not.
– You need to go on the vendors web site and hunt down this information: Where is this magical tool? At IBM its called ASU. At Dell its Open Manage. (or kcsflash if you’re lucky enough to be working on Linux). The other tools you can find will be left as an exercise to the reader.
– This is not the same interface that you manage the blades with. For example, on IBM blades you can configure the boot order in the AMM, but you can’t enable Hyperthreading on the processors. So you’ll just have to deal with it. Sorry.
My typical implementation of this is that I automate this with xCAT. It generally works pretty well and as new hardware items are added, they automatically get updated with the correct BIOS settings. The problem with this is that it hardly ever looks like an Enterprise solution. In fact it is only a band-aid to a glaring hole in most of the Server management solutions that I’ve seen. I still like it and if you’re ok doing it, go for it. While I think this method improves moral, I think its still just as expensive as solution A.
C. The UCS way (much easier). You open up UCS manager, head over to the Server tab on the left hand side. From there go down the Servers -> Policies -> root (or create a subgroup) -> BIOS policies. From here, you point and click and get the settings the way you want them. You add this to your service profile template, and then create service profiles from it for all 300 blades. (to be honest, if this is UCS, you have to do this 2 times since each UCSM can only manage at most 160 blades.) This takes a few minutes to do. No waiting. From there, you just power on the server and the BIOS settings are updated by default.
Here are the advantages of this:
– No third party tools, very easy, no pain!
– Single Pane of Glass (meaning you do it from the same management tool that you manage the rest of the servers).
– Enterprise level. I sometimes think that Enterprise Level = “A dumb person could do it with little training”. But yeah, that same guy who can press F1 on 300 servers certainly has enough skill to configure this.
– If you decide you need to update the BIOS settings, you do it once and its done everywhere.
These BIOS settings take about 5 minutes to set up. You can experiment to see if you get better performance with Hyperthreading, or Turbo boost. In essence: your system engineers can now do things that are more productive!
Ok, so what, you saved $1500. I think you could argue that it was maybe $2,000. Yes, there are more features in UCS that can add more. In fact, they just keep adding up. And that’s just the point: Often in managing data centers its the little things that keep adding up that keep adding to cost, complexity, and problems. By taking care of lots of little things, all the sudden the costs start to go down big time!
I expect other vendors to add this functionality over time. Its these little things that make a system engineer happy. And its just one reason I really like UCS.