Category Archives: VMware

Configure VMware from scratch without Windows

One of the things that bugs me about vCenter (still) is that it is still very tied to the Windows operating system.  You have to have Windows to set it up and trying to go about without Windows is still somewhat difficult.  In my lab I’m trying to get away from doing Windows.  I have xCAT installed to PXEboot UCS Blades to do what I want.  Its great, and its automated.  But when I installed 8 nodes to be ESXi hosts I quickly realized I needed vCenter to demonstrate this and use this as others would.

That requires vCenter.  VMware has had the vCenter appliance out for a few years now.  It runs on SLES and comes preconfigured.  The only problem is installing it when you have no vCenter client because today those clients are only made for the Windows Operating system.  How to get around this?

ovftool was the thing I found that did the job for me.  I found the link by reading the ever prolific Virtual Ghetto post on deploying ovftool on ESXi.  Since I had Linux, installing ovftool on the ESXi host wasn’t necessary for me.  Instead I just installed it on my Linux server (with some trouble since it deploys this stub and you have to make sure you don’t modify the file).

I ran the command:

ovftool -ds=NFS1 VMware-vCenter-Server-Appliance-5.0.5201-1476389_OVF10.ova vi://root:password@node01

After that, I watched my DHCP server and saw that it gave the vCenter appliance the IP address of  Hopefully you have DHCP or you might be hosed.

Then after finding the docs, I intuitively opened my web browser to (everyone knows that port number right?) I then logged in with user ‘root’ and password ‘vmware’ and started the auto setup.  After changing the IP address and restarting the appliance I was pretty golden.

Once configured, log into the appliance at and then be stoked that you have flash player already installed and that it works.  Oh you didn’t have flash player installed on your linux server?  That sucks, I didn’t either.  Guess that’s another hoop we have to jump through. But wait, then you find that Flash 11.2.0 is the last Flash that has been released for Linux.  Guess what?  VMware requires Flash version 11.5.  Nice.

At this point I just copied a Windows VM that I had laying around and started managing it from there.  The moral of the story is that you can’t do a Windows free VMware environment.  Sure, I could have done fancy scripting and managed it all remotely with some of their tools, but if I’m going to be doing all that, why should I pay for VMware?  I’d be better off just doing straight native KVM.  YMMV.

Nexus 1000v – A kinder gentler approach

One of the issues skeptical Server Administrators have with the 1000v is that they don’t like the management interface being subject to a virtual machine.  Even though the 1000v can be configured so that if the VSM gets disconnected/powered-off/blownup the system ports can still be forwarded.  But that is voodoo.  Most say:  Give me a simple access port so I can do my business.

I’m totally on board with this level of thinking.  After all, we don’t want any Jr. Woodchuck network engineer to be taking down our virtual management layer.  So let’s keep it simple.

In fact!  You may not want Jr. Woodchuck Networking engineer to be able to touch your production VLANs for your production VMs.  Well, here’s a solution for you:  You don’t want to do the networking, but you don’t want the networking guy to do the networking either.  So how can we make things right?  Why not just ease into it.  The diagram below, presents, the NIC level of how you can configure your ESXi hosts:

Here, is what is so great about this configuration.  The VMware administrator can use things “business as usual” with the first 6 NICs.

Management A/B teams up with vmknic0 with IP address  This is the management interface and used to talk to vCenter.  This is not controlled by the Nexus 1000v.  Business as usual here.

IP Storage A/B teams up with vmknic1 with IP address This is to communicate with storage devices (NFS, iSCSI).  Not controlled by Nexus 1000v.  Business as usual.

VM Traffic A/B team up.  This is a trunking interface and all kinds of VLANs pass through here.  This is controlled either by a virtual standard switch or using VMware’s distributed Virtual Switch.  Business as usual.  You as the VMware administrator don’t have to worry about anything a Jr. Woodchuck Nexus 1000v administrator might do.

Now, here’s where its all good.  With UCS you can create another vmknic2 with IP address  This is our link that is managed by the Nexus 1000v.  In UCS we would configure this as a trunk port with all kinds of VLANs enabled over it.  This can use the same VNIC Template that the standard VM-A and VM-B used.  Same VLANs, etc.

(Aside:  Some people would be more comfortable with 8 vNICs, Then you can do vMotion over its own native VMware interface.  In my lab this is

The difference is that this IP address belongs on our Control & Packet VLAN.  This is a back end network that the VSM will communicate with the VEM over.  Now, the only VM kernel interface that we need to have controlled by the Nexus 1000v is the IP address.  And this is isolated from the rest of the virtualization stack.  So if we want to move a machine over to the other virtual switch, we can do that with little problem.  A simple edit of the VMs configuration can change it back.

Now, the testing can coexist on a production environment because the VMs that are being tested are running over the 1000v.  Now you can install the VSG, DCNM, the ASA 1000v, and all that good vPath stuff, and test it out.

From the 1000v, I created a port profile called “uplink” that I assign to these two interfaces:

port-profile type ethernet uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 1,501-512
channel-group auto mode on mac-pinning
no shutdown
system vlan 505
state enabled

By making it a system VLAN, I make it so that this control/packet VLAN stays up. For the vmknic ( I also created a port profile for control:

port-profile type vethernet L3-control
capability l3control
vmware port-group
switchport mode access
switchport access vlan 505
no shutdown
system vlan 505
state enabled

This allows me to migrate the vmknic over from being managed by VMware to being managed by the Nexus 1000v. My VSM has an IP address on the same subnet as vCenter (even though its layer 3)

n1kv221# sh interface mgmt 0 brief

Port VRF Status IP Address Speed MTU
mgmt0 — up 1000 1500

Interestingly enough, when I do the sh module vem command, it shows up with the management interface:

Mod Server-IP Server-UUID Server-Name
— ————— ———————————— ——————–
3 00000000-0000-0000-cafe-00000000000e
4 00000000-0000-0000-cafe-00000000000f

On the VMware side, too, it shows up with the management interface:

Even though I only migrated the vmknic over.

This configuration works great.  It provides a nice opportunity for the networking team to get with it and start taking back control of the access layer.  And it provides the VMware/Server team a clear path to move VMs back to a network they’re more familiar with if they are not yet comfortable with the 1000v.

Let me know what you think about this set up.

Storage: The new sexy

I was fortunate enough to attend VMworld 2012 in San Francisco this year!  It is indeed a great privilege and I can’t thank Cisco enough for sending me.  There were lots of announcements that were pretty cool from a UCS standpoint like UCS Central and the vCenter plugin. (These were demoed in the Cisco booth) There were cool announcements from VMware about Horizon and the Cloud Suite.  etc.   The sessions were great and the after hours events always entertaining (Although the food at the VMware party sucked compared to 2010).  But among the madness of the myriad of messages there was one that stood out to me more than anything else:  Flash Storage.

Remember when storage was boring?  EMC, NetApp, blah blah blah.  Well those days are gone.  Its not so simple anymore.  The shear number of storage vendors on the showroom floor was a clear illustration that storage is still an open frontier like the wild west, or even like a new season of American Idol… (OK, maybe that last one was a bad analogy)

Sure EMC still leads in market share and NetApp is the fastest growing, but there is plenty of room for disruption.  There were several really good sessions on best practices for storage.  One of my favorite quotes came from Duncan Epping (of YellowBricks and Clustering Deep dive author fame):

We always blame the network for our problems but its usually the storage that is at fault

Indeed we see this in practice quite a lot.  People can’t buy more UCS because they’re constrained by their storage.  The network seems easy enough to blame but as server administrators are getting more comfortable with networking (mostly because they have to since the network access layer is now inside the server at the vSwitch) they’re starting to get that right much more often than the storage.

One of my favorite sessions was a Gartner Storage Best Practices session.  They had the following great messages:

“IOPS resolution is a multi dimensional problem that may not best addressed in the storage array”

This is why putting Fusion-io cards in a server may help with a tiered approach.  (BTW, these were announced to be inside UCS Blades now and should be available before the end of 2012).  It also explains why companies like Atlantis Computing with their ILIO product can offer big performance gains by offloading some of the work the storage array has to do as well as saving space.

This brings up another point from the Gartner session when talking about VDI (aka SHVD: server hosted virtual desktop)

“Storage costs present the #1 barrier to entry”

If you’re wondering when the year of the virtual desktop will come its when the organization has to buy new storage.  Over 40% of data center budget for new equipment goes into storage.  It pretty much pushes server and network equipment to the fringes.  That’s why I don’t ever think there will be a ‘year’ of the virtual desktop.  Instead, we’re in the beginning of the ‘decade of the virtual desktop’

The Tiered approach works as follows:  You have fast disks ( SSDs), slower disks (FC, SAS) and then the slowest disks:  SATA spinning at 7200 RPMS.  Ideally you want the data you use the most (the master copy of a windows VM) sitting on the fast storage, swap files on the mid tier, and lesser used workloads sitting down at the bottom.

The issue with the tiered approach (which NetApp doesn’t really have except for maybe the flash cache which is read only) is that you have to put the workloads where you think they will be.  And there’s a good chance you’ll get it wrong.

And that’s one reason there’s such a huge market out there for storage products that solve the issues of how to store and manage virtual machine files.  It used to be the OS sat on the server and the local disk was the best you could do.  Now with all of those VMs contending for IOPS, the storage is the bottleneck.  The new sexy bottleneck.  Sexy because there’s a lot of money to be made if you can convince people your solution is the best.

Using flash storage seems to be the sexy way to entice your customers.  SSD arrays were all the rage at VMworld.  Tiered solutions that allowed SSDs with FC/SAS/SATA were also quite popular.   I thought I’d go through some of the storage solutions I had a chance to visit with at VMworld.  Most of these are lesser known so it will be interesting to watch and see how the space changes in the next year:

Violin Memory

The Violin 6000 Series flash memory array can do 1 million IOPS with 4Gbps of bandwidth in a 3RU space.  The secret sauce is that they build their own flash memory controllers instead of using the standard SSDs that most flash array vendors use.  This probably isn’t the cheapest but its hard to beat in terms of speed.  This is the storage you buy when money is no object and you just want fast.  Just imagining this connected to some Nexus 5000s and a UCS full of B200 M3s makes me all tingling inside.


I didn’t see Whiptail on the floor but I have helped 2 UCS customers configure this and get it running.  Its cheaper and wicked fast, but offers little intelligence as to what is happening.  Just cram a bunch of SSD drives into the array and lets you go to town.  For some people, speed is all you need and they don’t care about fancy dashboards.


I had a great conversation with a great guy at Tintri at the vFlipCup event on Monday night… Wish I could remember his name.  (BTW, I represented Team JBOD with @CiscoServerGeek and we apparently wrote checks that our team could not cache).   …But I digress… Tintri seemed to be a mix of Atlantis Computing and Whiptail.  Instead of having a VM do the block caching like ILIO does, you instead have that intelligence take place on the controller.  Couple that with an array of SSDs and life looks pretty good.  This seems better than Atlantis in that your chance of VM going down is greater than your chance of the storage array going down.

Pure Storage

Two of the bigger booths of storage companies that I hadn’t heard of were Pure Storage and Tegile.

Pure Storage is a flash array vendor that seemed to have the most sophisticated protocol support including iSCSI (BTW: for the record I really don’t like iSCSI.  When you have UCS, Fibre Channel is so easy and with the UCS Del Mar release later this year you have all the components you need for FCoE without buying any separate equipment)


Tegile is a hybrid array that supports tiered storage.  The value is the deep integration with SSDs.  I would look at this offering as a less confusing EMC offering.  It offers SAN and NAS capabilities as well as data deduplication.  Pretty sweet system.


It seems like all the providers have some great niches and I think most people would be happy with any of these storage solutions.  I’d hate to leave this post without tipping my hat to EMC and NetApp who I work with and do a tremendous job.  There’s a reason they both have so many customers:  They build great products.  I should also call out Hitachi Storage.  Their own team admits they suck at marketing, but in terms of performance and reliability for mission critical apps, its hard to beat their rock solid solutions.  Its truly a company built and run by engineers.  That’s one reason their customers and me like them so much.

So if this post makes you feel all warm inside, that’s because storage is the new sexy and it is good for you to look at on company time.

An adventure in Powershell and the vSphere PowerCLI

I’ve always found a good way to get started with a new language is to just go head into it trying to solve some problem you’re working on, then ask the Internet via a search engine how to do something you want to do.

Today I started for the first time with PowerShell. Here’s the problem I was trying to solve:

I have a lot of VMs that I want to bring up for a class that I’m giving. There could be between 10-50 people at any given class. What I need to do is clone a master VM from a template, change its MAC address to something I have reserved in my DHCP server, and then power it up.

All of this can be done via vCenter.  But its a slow and painful point-and-click-yourself-to-death process. Automation is the way to get this done.

All of this can be done via xCAT.  But I figured why not give it a try.  Other people can live without xCAT.  Maybe I can too.

I have a history of perl. Yes, I have lots of skeletons in the closet with that language. So I was going to do all of this with Perl. But I figured that since I was going to be doing power shell scripts anyway for the UCS emulator portion, why not just do it all with the same thing? So I gave it a shot.

First off, the Windows editors suck. I had to stick with VIM because there’s nothing better for me. Let’s not even argue about that. +1 for me for retaining my dignity.

Next, after installing the VMware PowerCLI tool it was pretty easy.


Here’s the script to clone 9 VMS from a template called UCSPEMaster.  I just change it depending on how many I need.

1..9 | foreach {
    # pad each name with 0s so we have: ucspe08,ucspe09,ucspe10,...
    $strNum = [Convert]::ToString([int]$_)
    $suffix = $strNum.padLeft(2,"0")
    New-VM -VMHost -datastore datastore1 -Template UCSPEMaster \
           -Folder 'UCS-Name UCSPE0$suffix -diskstorageFormat Thin \
           -Location 'UCS Emulators' -runAsync

Notice that each machine will be called UCSPEXX, where XX is the range I specify. That way if
4 more people walk into the class after I’ve configured 14, then I can do 15..8.

I also put all of them on the same host (, the same datastore (datastore1), the same Folder (UCS Emulators), and made them thin clones.

If you have a cluster running Storage DRS (vSphere 5.0+) then you don’t have to specify the datastore and DRS will put it where it sees fit.


The only thing I need to do now is change the MAC address to something that I have reserved.  That way, I can tell the user to just log into the IP address that I’ve set up beforehand.  I’ve configured 60 IP addresses so that I’m ready for a big class.

get-vm UCSPE* | foreach {
    $number = $"UCSPE","")
    $hexNumber = [Convert]::ToString([int]$number, 16)
    $lastPart = $hexNumber.PadLeft(2,"0")
    $adapter = Get-NetworkAdapter $_
    write-host "Changing MAC address for $_"
    Set-NetworkAdapter -NetworkAdapter $adapter \
          -MacAddress 00:50:56:00:01:$lastPart -Confirm:$false

Most of the script is made up of getting the last number part of the VM name. Since each VM is named UCSPE01-UCSPE60 then it grabs the 60, converts it to hex, then adds that as the last part of the MAC address of the adapter. This will work as long as I don’t have more than 255 VMs.

I could have put a Start-VM on the end of this.  Generally, that’s just a powershell one-liner:

Get-VM UCSPE* | Start-VM


The last script will just remove these VMs.  As I tweak around with them, or as people in the class tweak around with them, I just want to erase them and start fresh for the next class.  This is fairly easy:

Get-VM UCSPE* | foreach {
    Stop-VM $_ -confirm:$false
    Remove-VM $_ -DeleteFromDisk -confirm:$false

The only thing I need to update this with is to not power it off if it isn’t on.  It throws some nasty errors if it’s not powered on.  But I’ll do that some other time.

I’ll get this lab working with PowerShell.  It’s not a bad language.  It’s another tool in the handbag.  I still prefer command line scripting with Bash or Perl.  But every now and then its fun to go over and see how the other side lives.  Now, back to xCAT.

vSphere 5 Licensing fun

UPDATE (Aug 3, 2011): VMware changed the licensing model addressing many of the disadvantages laid out in this post. Among them: the monster 1TB VM will not be super expensive and the Enterprise+ license went up to 96GB of vRAM.

The new vSphere 5 licensing information has taken the VMware blogger world by storm.  So I thought I might join in the fun.

What’s happened?  VMware announced vSphere 5 on July 12, 2011.  With all its cool new features, is a new licensing model that is different from the old one.   So let’s examine how this effects your organization.  Most of the material I got from the vSphere Pricing PDF.

What the licensing change is

Before, the license was per CPU socket.  This cost was around $3495 for the Enterprise Plus license.  Now the cost is based on CPU sockets and variable amount of vRAM.  (VMware argues that vRAM is not the same as physical RAM).   One common example is that if you buy 2 Enterprise Plus licenses for a 2 socket system, you’ll get 96GB of vRAM available to use.

Why Licensing had to change

VMware’s white paper spins it as a good thing.  (no mystery why they did this).  But here is what I think the real reason is:  We’ve seen in the last decade a trend of adding more CPU cores per socket.  What started out as a dual core movement is now in less than 10 years at 12 cores per socket.  Imagine the next few years:  24 cores per socket?  64 cores per socket?  128 cores per socket?  Not all that hard to imagine.  Also consider the memory DIMMs:  16GB DIMMs, 32 GB DIMMS (those are already here!)  64 GB DIMMs… you get the idea.

So what does this mean for a software company like VMware that licenses per CPU socket?  It means that in several years, people won’t need as many CPU socket license because they’ll be able to put more VMs on a physical host, so they’ll need less socket licenses.  That means less money for VMware.

VMware like any business is in business to make money, and having profits eroded is not a good thing.  So they decided to take action now and make it so that innovations in physical hardware do not erode their earnings.

Who is this change bad for?

This is bad for customers right now who have 2 CPU socket systems with greater than 96GB of memory.  Hardware vendors like IBM and Cisco have been showing customers how they can save licensing costs on servers by adding more memory to existing 2 socket servers.  For example:

A system like the Cisco B250 which allows up to 384 GB of RAM will now require you pay for 8 licenses instead of 2. This is a huge penalty!

A system like the IBM HX5 which allows up to 256GB of RAM in a single wide blade with two CPU sockets will require  you to pay for 6 licenses instead of 2.

This is also bad for people that over allocate Virtual memory.  Its common practice for people to overcommit memory.   Since you’re paying for vRAM this means that you may be potentially paying more.  This is funny because VMware in the past has said overcommitting memory is one of the good reasons to move to virtualization.

Its also bad for people with Virtual Machines with large memory.  VMware announced that they can now create a VM with 1TB of memory!  That’s great news, but consider the cost in vRAM:  That’s 21 Enterprise Plus licenses at $3,495 = $73,395!  Probably cheaper to use a real machine for that one?

This is bad for people who use Citrix Xen Desktop with VMware vSphere on the backend.  Its interesting that there is a completely different pricing model for people using VMware View with vSphere.  I think this licensing issue will force people to make a choice:  Go with VMware View or go Xen Desktop with Xen hypervisor on the backend.

The above may not be entirely true.  There is also a new product SKU called vSphere 5 Desktop that allows for 100 VM desktops priced at $6500.  This license supersedes the vSphere license so if you run Desktop VMs then your cost for the higher memory systems like the Cisco B250 would not go up.

Who is the change is good for?

Well for now its good for the standard 2 socket systems that have less than 96GB of RAM like the HP Proliant BL2x220c G7 blade which allows up to 96GB of RAM per server instance will require 2 per server, so 4.  No change here.  Same with the Cisco B200s with 16GB DIMMS which goes up to 96GB with 16GB DIMMS.  The problem is this won’t last for long.  As I mentioned before, RAM and CPU core density will only increase meaning you’ll have to pay more for the licenses.  16GB DIMMs will get cheaper and x86 processors will allow more DIMM slots in the future.  (The Cisco B250 already allows 48 DIMM slots on a 2 socket system).

The VMware marketing literature states that the vRAM can be pooled.  So if one system in your datacenter has only 16GB of RAM on a 2 socket system, then you’ll get all kinds of vRAM that you can use on other systems that may have more memory.

A proposed better solution

I’m not opposed to VMware making more money.  As physical server capabilities increase, VMware wants to get more money out of its utilization.  Even though the prices of physical servers will most likely remain the same, it appears VMware software will scale with the number of VMs deployed.

I would propose that there be vRAM entitlements decoupled from the CPU cores that cost less money.  For example, instead of paying for 8 vSphere licenses for a Cisco B250, why not make us pay for 2 vSphere 5 licenses plus 6 48GB vRAM entitlements that are priced at a different rate?  Make these vRents (vRAM entitlements) 1/4th the cost of the socket license?  Here’s the analysis of the B250:

vSphere 4.1 license:  2 licenses: $6,990 (2x$3495)

vSphere 5.0 license: 8 licenses: $27,960 (8x$3495)

my proposal:  vSphere 5.0 license + vRents: $12,233 = $6,990 (2x$3495) + $5,243 (6x$873)

I don’t like that its double the cost of the previous vSphere 4.1 but at least its better than the 400% increase of what we had before.

The Future?

VMware has changed pricing models before.  Just ask my friends in the Data center space where they now have to pay for hosting solutions.  But VMware is so far ahead of anyone else right now, its hard to know what type of backlash they will get.  Sure people will grumble when they need to pay more, but most likely people will just pay it.  So Hyper-V and Xen and KVM, what do you have to say to all of this?  Its your move!

VMworld 2011 Sessions Posted

I am super excited for VMworld 2011 and I’ve already got my tickets, hotel (at the Venetian!) and dates ready to roll!  I was scrolling through the long list of sessions and found tons that seem really interesting to me.

I submitted two sessions:

#2815 “Secrets of ESXi Automated Installations”


#2642 “Pimp my ESXi Kickstart Install”

They both will talk about mainly the same thing:  Getting ESXi onto bare metal and look at some free tools to do it with but the focus will mostly be on the configuration parameters inside the %post and %firstboot sections and how to use them.  We’ll also talk about the mod.tgz secrets that I use to deploy stateless ESXi and how it works when the machine reboots.  I actually didn’t think VMware would approve the title “Pimp my ESXi Kickstart Install” but hey, it got there!  So I guess we’ll see if I have to deliver any of them or not.  Either way, presenting or attending, I will enjoy attending VMworld 2011.

So if you don’t vote for me there were some others that looked pretty cool.  #l1940 10 Best Free Tools for vSphere Management in 2011 I’m sure will make the list.  (And I’m bummed that they’ll probably not look at xCAT which to me has more power behind it than most tools… the problem is its too undocumented… )

1956 The ESXi Quiz show is something I’m sure I’ll enjoy. and I’ll probably attend a few on power cli to see what the fuss is about.  (I’m a Linux guy so this shows how open minded I am.) and #2964 looks really good “Building products on top of VMware”.  I think I’d demand they present it if they rename it “Build products on top of VMware so that VMware will want to buy your company”.

Anyway, happy voting!

Virtual Machine Song

We wrote a song called “Virtual Machine” that can be freely downloaded here. (right click to download)

Some of my friends and I have been writing and performing songs for over 20 years.  Occasionally we’ll get together and work on something even though I’m separated from them by about 1,000 miles.  This song we had sitting around for a while and we didn’t get a chance to redo the vocals and add a bunch of other stuff we had originally planned, but I figured just get it out there and we’ll move on to the next hit song.

I’d love to play this song live along with a few other cloud/virtual songs we’ve got on the back burner at VMworld 2011.  Even better if we could do it on stage with Vittorio Viarengo.  But for now I’m just happy that its complete enough to listen to.

The song is probably too tech heavy but I think its still fun.  Here’s the silly lyrics:

I am a virtual machine, an instance of an operating system.  Just a program running a program.  To enable all your applications.

I’m a thin clone of the virtual machine and so we are tightly tethered.  We share the same disk image and write changes to a delta file

I’m a thick clone of a virtual machine I’m an independent image we run together on the same hardware and provide maximum utilization.

Baby you can try my virtual machine all you need is a hypervisor.  You can run it in the cloud or in the data center baby you could use some virtual love

I don’t want to disrespect your physical machine I just want to make you lots of money.  You’ve been idling all the time acting so inefficiently, baby you could use some virtual love.

We are virtual machines!  We can stop and suspend…

We can migrate to other systems or save states with snapshot files.

You’ve got lots of problems:  9 nines and system failures.  You got SAAS and PAAS to deliver on an SLA with penalty clauses.  You need virtual machines!  We can solve your desktop issues.  We can reduce your capital expenses and consolidate your infrastructure.

Baby you can try my virtual machine all you need is a hypervisor, you can run it in the cloud or in your datacenter baby you could use some virtual love.

I don’t want to disrespect your physical machine I just want to make you lots of money.  (I’m a free bit baby!)  You’ve been idling all the time acting so inefficiently baby you could use some virtual love…

Maybe you could take this virtual machine and shove it up inside your datacenter.  You can consolidate the system on commodity hardware baby you could use some virtual love…

I don’t want to disrespect your physical machine, I just want to make you lots of money.  You’ve been idling all the time acting too inefficiently, baby you got to use this virtual love…

ESXi 4.1 and HP BL460c G6 with Mezzenine card

Had an issue where I would install CentOS on these HP blades and I would be able to see 16 nics.  But when I installed ESXi 4.1 I only saw 8 nics.  16 is the right number because each flexNIC has 4 vNics.  So with 4 of these, I wanted to see some serious bandwidth.  After fumbling around we finally came to the conclusion that the be2net driver was not loaded on the hypervisor.

My Mezzanine card is a HP NC550m Dual Port Flex-10 10GbE BL-c Adapter.  My HP rep said that these were not going to be supported by HP on ESXi 4.1 until November and that I could drop back to 4.0 or he could try to get me some beta code.

I found that you can just download the driver here.  I tried a similar route by installing the hp-esxi4.1uX-bundle from HPs website but that just gave me stuff I didn’t need (like iLo drivers).

The link above is an ISO image.  The easiest way for me to install it on a running machine was to open the ISO on a linux machine and then copy the files to the ESX hosts:

# mkdir foo
# mount vmware-esx-drivers-net-be2net_400.2.102.440.0-1vm* foo -o loop
# cd foo/offline-bundle
# scp vhost001:/

Then you just need to install it.  The only problem with this is that it involves a entering maintenance mode and then a reboot.  Is this windows xp or something?  We’re just talking about a driver here…

Anyway, SSH to the ESXi 4.1 (or use VUG if you want to pay $500 bucks instead).  Since I use xCAT, I have passwordless SSH set up:

# vim-cmd hostsvc/maintenance_mode_enter
# esxupdate update --bundle /
# vim-cmd hostsvc/maintenance_mode_exit
# reboot; exit

After the node reboots you can run:

esxcfg-nics -l

you’ll be able to see all 16 nics.

Hope that saves you time as it took me a while to figure this out…

My next post will talk about how to integrate this into the kickstart file so you don’t have to do any after-the-install junk.

ImageX Windows 2008 with vCenter Server

With xCAT I used the imagex capabilities to clone a machine (virtual machine) with vCenter on it and now I’m installing that captured image to another virtual machine.  One reason I do it this way as opposed to creating a VM Template is that now I’m able to deploy to physical and virtual servers.  In addition I can deploy to KVM VMs and VMware VMs.

Anyway, when I restarted the cloned disk, I couldn’t start vCenter Server on it. This is something that I assume happens with all windows images that you run sysprep on regardless or not whether its xCAT induced.

There were a lot of error messages as I trolled through the logs and I searched on all of them:

“Windows could not start the VMware VirtualCenter Server on Local Computer.  For more information, review the System Event Log.”

“the system cannot find the path specified. c:\Program Files (x86)\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\master.mdf” (odd to me, since the file did actually exist!)

“could not start the SQL Server (SQLEXP_VIM) on Local Computer”

I tried all these google terms with little success:

“Virtual Center will not start after reboot”

Finally, I found this post: that finally made sense to me, so I gave it a shot.  Here’s exactly what needs to be done:

Step 1

Log into Windows Server 2008 machine and go to ‘Services’

Step 2

Right click on ‘SQL Server (SQLEXP_VIM)’  and click on Properties

Step 3

Under ‘Log On’ check the Local System account and also check ‘Allow service to interact with desktop’

Step 4

Start the ‘SQL Server (SQLEXP_VIM)’ service.  It should start up without errors.

Step 5

Do the same procedure (Steps 2-4) to the ‘VMware VirtualCenter Server’ service.  This should now start up and you should be able to connect to Virtual Center.

Well, that wasted about 4 hours of my time, but its nice to have a happy ending.

ESXi 4.1 command line awesomeness

I spend a lot of time on the command line of ESXi 4.1 due to my development duties at Sumavi.  There are quite a few things you can do on the command line that make it pretty cool to work with.  As such, in many instances I don’t install vSphere Server nor vSphere client.  I just log in and do my duties.  Notice that everything I post below you can do without vSphere Server nor vSphere client.  Let me know if these are useful to you!

1.  Get a list of all VMs on the Hypervisor:

# vim-cmd vmsvc/getallvms
Vmid     Name                      File                          Guest OS          Version   Annotation
16     Vcenter1   [Storage1-ESX01] Vcenter1/Vcenter1.vmx   windows7Server64Guest   vmx-07              
32     Vcenter    [Storage1-ESX01] Vcenter/Vcenter.vmx     winLonghorn64Guest      vmx-07

Notice the vmid.  That vmid is used in many commands that follow when you want to perform actions on individual VMs.

2.  Check which Physical NICS are up

There are a few commands in the esxcfg-* family that are used to configure the hypervisor network.  For example, if you want to see which NICs have network connections, you can use:

~ # esxcfg-nics -l
Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description                   
vmnic0  0000:02:00.00 bnx2x       Up   1000Mbps  Full   d8:d3:85:da:f8:30 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet
vmnic1  0000:02:00.01 bnx2x       Down 0Mbps     Half   d8:d3:85:da:f8:34 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet
vmnic2  0000:02:00.02 bnx2x       Up   9000Mbps  Full   d8:d3:85:da:f8:31 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet
vmnic3  0000:02:00.03 bnx2x       Down 0Mbps     Half   d8:d3:85:da:f8:35 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet
vmnic4  0000:02:00.04 bnx2x       Down 0Mbps     Half   d8:d3:85:da:f8:32 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet
vmnic5  0000:02:00.05 bnx2x       Down 0Mbps     Half   d8:d3:85:da:f8:36 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet
vmnic6  0000:02:00.06 bnx2x       Down 0Mbps     Half   d8:d3:85:da:f8:33 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet
vmnic7  0000:02:00.07 bnx2x       Down 0Mbps     Half   d8:d3:85:da:f8:37 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet

Notice that only vmnic0 and vmnic2 are up.  This mostly has to do with the way I configured my blades with the Flex-10 Virtual connect.  (A feature of HP Blades).  If I am now to configure the network, its best that I do only vmnic0 and vmnic2 since they’re the only ones that have a link.  For you Linux masters out there, there’s no ‘service network status’ nor restart that you can do.  It just always seems to be on.

3.  Creating a quick network connection

Since we know vmnic2 is up, let’s make a connection to it so that we can SSH into it, or at least ping out of it:

# add vSwitch1
esxcfg-vswitch-a vSwitch1
# link vSwitch1 to vmnic2 as an uplink
esxcfg-vswitch -L vmnic2 vSwitch1
# add the DATA portgroup to this switch
esxcfg-vswitch -A DATA vSwtich1
# uplink DATA to vmnic2
esxcfg-vswitch -M vmnic2 -p DATA vSwitch1
# put DATA on VLAN70
esxcfg-vswitch -v 70 -p DATA vSwitch1

4.  Create a new vmdk and add to existing VM

Here we have a VM (vmid 32 ) that we want to add a 60GB vmdk to.  We run:

# create the 60GB hard drive in foo's directory
vmkfstools -c 60G /vmfs/volumes/datastore1/foo/newfoo.vmdk
# add the disk to foo's inventory. (The 0 and 1 is the scsi ID numbers
vim-cmd vmsvc/device.diskaddexisting 32 /vmfs/volumes/datastore1/foo/newfoo.vmdk 0 1

5.  Check/Toggle VM power stat

You can turn nodes off and on and check power status.  You need to know the vmid as shown in #1 above:

# get power stat
vim-cmd vmsvc/power.getstat 16
# turn off
vim-cmd vmsvc/ 16
# turn on
vim-cmd vmsvc/power.on 16

6.  Add the Physical USB Device to a VM

If you want to add the USB device that’s plugged into the physical machine to the virtual machine with vmid 16 you can do this:

#make sure usbarbitrator is started
/etc/init.d/usbarbitrator start
# add to vmid 16
vim-cmd vmsvc/device.connusbdev 16 "path:1/0/0 autoclean:1"

Note that the VM should be powered off when you do this for best results.

7.  Register a VM to the Hypervisor

If you copied all the vmx and vmdk files to an ESXi 4.1 hypervisor you can simply register them with that hypervisor and turn them on:

vim-cmd solo/registervm /vmfs/volumes/datastore1/foo/foo.vmx
# then turn it on using the stuff in tip 5!

8.  Enable SSH from the command line

This is an easy one:

/etc/init.d/TSM-SSH start

9.  Add the license to the ESXi 4.1 hypervisor

This came  up in a few places and I already documented it in this blog, but figured I’d do it again.  If you have a license and you want to add it to your hypervisor because its about to expire in 1 day you can log in and just run:

vim-cmd vimsvc/license --set XXXXX-XXXXX-XXXXX-XXXXX-XXX11

10.  Writing output to main console

In your kickstart files, you may want to redirect output to the main console that people watch as an installation takes place.  This is /dev/tty2.  Therefore, if in your kickstart file you are cloning a vmdk using vmdkfstools, you can let people see how the progress is going by just piping it out.  Here’s an example:

vmkfstools -i $DATASTOREREMOTE/stage/stage.vmdk -d zeroedthick $DATASTOREOTHER/new/new.vmdk | tee -a /dev/tty2

This is cool in that you’ll see the percentage points pop up as you go along.  The thing to remember is that you’ll have to send some carriage return escape sequences a la echo -e “\r\n” to line things up better.