Category Archives: ESXi 4.1

ESXi 4.1 and HP BL460c G6 with Mezzenine card

Had an issue where I would install CentOS on these HP blades and I would be able to see 16 nics.  But when I installed ESXi 4.1 I only saw 8 nics.  16 is the right number because each flexNIC has 4 vNics.  So with 4 of these, I wanted to see some serious bandwidth.  After fumbling around we finally came to the conclusion that the be2net driver was not loaded on the hypervisor.

My Mezzanine card is a HP NC550m Dual Port Flex-10 10GbE BL-c Adapter.  My HP rep said that these were not going to be supported by HP on ESXi 4.1 until November and that I could drop back to 4.0 or he could try to get me some beta code.

I found that you can just download the driver here.  I tried a similar route by installing the hp-esxi4.1uX-bundle from HPs website but that just gave me stuff I didn’t need (like iLo drivers).

The link above is an ISO image.  The easiest way for me to install it on a running machine was to open the ISO on a linux machine and then copy the files to the ESX hosts:

# mkdir foo
# mount vmware-esx-drivers-net-be2net_400.2.102.440.0-1vm* foo -o loop
# cd foo/offline-bundle
# scp SVE-be2net-2.102.440.0-offline_bundle-292682.zip vhost001:/

Then you just need to install it.  The only problem with this is that it involves a entering maintenance mode and then a reboot.  Is this windows xp or something?  We’re just talking about a driver here…

Anyway, SSH to the ESXi 4.1 (or use VUG if you want to pay $500 bucks instead).  Since I use xCAT, I have passwordless SSH set up:

# vim-cmd hostsvc/maintenance_mode_enter
# esxupdate update --bundle /SVE-be2net-2.102.440.0-offline_bundle-292682.zip
# vim-cmd hostsvc/maintenance_mode_exit
# reboot; exit

After the node reboots you can run:

esxcfg-nics -l

you’ll be able to see all 16 nics.

Hope that saves you time as it took me a while to figure this out…

My next post will talk about how to integrate this into the kickstart file so you don’t have to do any after-the-install junk.

ESXi 4.1 command line awesomeness

I spend a lot of time on the command line of ESXi 4.1 due to my development duties at Sumavi.  There are quite a few things you can do on the command line that make it pretty cool to work with.  As such, in many instances I don’t install vSphere Server nor vSphere client.  I just log in and do my duties.  Notice that everything I post below you can do without vSphere Server nor vSphere client.  Let me know if these are useful to you!

1.  Get a list of all VMs on the Hypervisor:

# vim-cmd vmsvc/getallvms
Vmid     Name                      File                          Guest OS          Version   Annotation
16     Vcenter1   [Storage1-ESX01] Vcenter1/Vcenter1.vmx   windows7Server64Guest   vmx-07              
32     Vcenter    [Storage1-ESX01] Vcenter/Vcenter.vmx     winLonghorn64Guest      vmx-07

Notice the vmid.  That vmid is used in many commands that follow when you want to perform actions on individual VMs.

2.  Check which Physical NICS are up

There are a few commands in the esxcfg-* family that are used to configure the hypervisor network.  For example, if you want to see which NICs have network connections, you can use:

~ # esxcfg-nics -l
Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description                   
vmnic0  0000:02:00.00 bnx2x       Up   1000Mbps  Full   d8:d3:85:da:f8:30 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet
vmnic1  0000:02:00.01 bnx2x       Down 0Mbps     Half   d8:d3:85:da:f8:34 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet
vmnic2  0000:02:00.02 bnx2x       Up   9000Mbps  Full   d8:d3:85:da:f8:31 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet
vmnic3  0000:02:00.03 bnx2x       Down 0Mbps     Half   d8:d3:85:da:f8:35 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet
vmnic4  0000:02:00.04 bnx2x       Down 0Mbps     Half   d8:d3:85:da:f8:32 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet
vmnic5  0000:02:00.05 bnx2x       Down 0Mbps     Half   d8:d3:85:da:f8:36 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet
vmnic6  0000:02:00.06 bnx2x       Down 0Mbps     Half   d8:d3:85:da:f8:33 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet
vmnic7  0000:02:00.07 bnx2x       Down 0Mbps     Half   d8:d3:85:da:f8:37 1500   Broadcom Corporation NetXtreme II 57711E/NC532i 10Gigabit Ethernet

Notice that only vmnic0 and vmnic2 are up.  This mostly has to do with the way I configured my blades with the Flex-10 Virtual connect.  (A feature of HP Blades).  If I am now to configure the network, its best that I do only vmnic0 and vmnic2 since they’re the only ones that have a link.  For you Linux masters out there, there’s no ‘service network status’ nor restart that you can do.  It just always seems to be on.

3.  Creating a quick network connection

Since we know vmnic2 is up, let’s make a connection to it so that we can SSH into it, or at least ping out of it:

# add vSwitch1
esxcfg-vswitch-a vSwitch1
# link vSwitch1 to vmnic2 as an uplink
esxcfg-vswitch -L vmnic2 vSwitch1
# add the DATA portgroup to this switch
esxcfg-vswitch -A DATA vSwtich1
# uplink DATA to vmnic2
esxcfg-vswitch -M vmnic2 -p DATA vSwitch1
# put DATA on VLAN70
esxcfg-vswitch -v 70 -p DATA vSwitch1

4.  Create a new vmdk and add to existing VM

Here we have a VM (vmid 32 ) that we want to add a 60GB vmdk to.  We run:

# create the 60GB hard drive in foo's directory
vmkfstools -c 60G /vmfs/volumes/datastore1/foo/newfoo.vmdk
# add the disk to foo's inventory. (The 0 and 1 is the scsi ID numbers
vim-cmd vmsvc/device.diskaddexisting 32 /vmfs/volumes/datastore1/foo/newfoo.vmdk 0 1

5.  Check/Toggle VM power stat

You can turn nodes off and on and check power status.  You need to know the vmid as shown in #1 above:

# get power stat
vim-cmd vmsvc/power.getstat 16
# turn off
vim-cmd vmsvc/power.off 16
# turn on
vim-cmd vmsvc/power.on 16

6.  Add the Physical USB Device to a VM

If you want to add the USB device that’s plugged into the physical machine to the virtual machine with vmid 16 you can do this:

#make sure usbarbitrator is started
/etc/init.d/usbarbitrator start
# add to vmid 16
vim-cmd vmsvc/device.connusbdev 16 "path:1/0/0 autoclean:1"

Note that the VM should be powered off when you do this for best results.

7.  Register a VM to the Hypervisor

If you copied all the vmx and vmdk files to an ESXi 4.1 hypervisor you can simply register them with that hypervisor and turn them on:

vim-cmd solo/registervm /vmfs/volumes/datastore1/foo/foo.vmx
# then turn it on using the stuff in tip 5!

8.  Enable SSH from the command line

This is an easy one:

/etc/init.d/TSM-SSH start

9.  Add the license to the ESXi 4.1 hypervisor

This came  up in a few places and I already documented it in this blog, but figured I’d do it again.  If you have a license and you want to add it to your hypervisor because its about to expire in 1 day you can log in and just run:

vim-cmd vimsvc/license --set XXXXX-XXXXX-XXXXX-XXXXX-XXX11

10.  Writing output to main console

In your kickstart files, you may want to redirect output to the main console that people watch as an installation takes place.  This is /dev/tty2.  Therefore, if in your kickstart file you are cloning a vmdk using vmdkfstools, you can let people see how the progress is going by just piping it out.  Here’s an example:

vmkfstools -i $DATASTOREREMOTE/stage/stage.vmdk -d zeroedthick $DATASTOREOTHER/new/new.vmdk | tee -a /dev/tty2

This is cool in that you’ll see the percentage points pop up as you go along.  The thing to remember is that you’ll have to send some carriage return escape sequences a la echo -e “\r\n” to line things up better.

VMware API code to mount NFS Datastore on ESX host.

Since I posted this to the VMware user group today I figured I might as well post this to my blog as well:

Lets say you have a datastore 10.3.0.101 that has a mount point /install/vm directory. You want to mount that datastore onto host vhost04 so you can start creating VMs. Here is how it is done:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
use Data::Dumper;
require VMware::VIRuntime;
VMware::VIRuntime->import();
use strict;

#  log into a node:
my $conn;
my $esx = shift || 'vhost04';
my $server = '10.3.0.101';
my $serverpath = '/install/vm';
my $location = 'nfs_10.3.0.101_install_vm';

eval {
$conn = Vim->new(service_url=>"https://$esx/sdk");
$conn->login(user_name=>'root',password=>'c1ust3r');
};
if($@){
print Dumper($@);
print $@->;fault_string . "\n";
}

my $hv = $conn->find_entity_view(view_type => 'HostSystem');

my $nds = HostNasVolumeSpec->new(accessMode=>'readWrite',
remoteHost=>$server,
localPath=>$location,
remotePath=>$serverpath);
my $dsmv = $hv->{vim}->get_view(mo_ref=>$hv->configManager->datastoreSystem);

eval {
$dsmv->CreateNasDatastore(spec=>$nds);
};

if ($@) {
print $@->fault_string . "\n";
print Dumper($@);
}

$conn->logout;

ESXi 4.1: setting license via command line

Setting the license at the command line is done with the following command:

1
vim-cmd vimsvc/license

If you run it you’ll see that one of the options is to –set=string.
But if you run it like this:

1
vim-cmd vimsvc/license --set=XXXXX-XXXXX-XXXXX-XXXXX-XXX11

You will get an error.

The right way to do it is:

1
vim-cmd vimsvc/license --set XXXXX-XXXXX-XXXXX-XXXXX-XXX11

Notice there is no ‘=’. I would ask that someone please update the help message here.

You can run it again with the –show option and you’ll see your license has been accepted :-)

VMware API in Perl

Here’s a simple example to connect to a hypervisor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
#!/usr/bin/perl

use Data::Dumper;
require VMware::VIRuntime;
VMware::VIRuntime->import();
use strict;

# try logging into a node:
my $conn;
my $hyp = shift || 'vhost31';
print "performing action on $hyp\n";
eval {
        $conn = Vim->new(service_url=>"https://$hyp/sdk");
        $conn->login(user_name=>'root',password=>'cluster');
};

Now you probably want to do something since you’re connected. The best way is to go over and read the VMware API documentation. The Reference Guide seems to be the best. You have to do a lot of guessing since it isn’t necessarily written for any language. Hopefully I’ll be able to post more on using this later. If you want to huge example, you can look at the ESX plugin in the xCAT source tree. We do pretty much everything you could think of with it. Since its open source, you can use it however you want.

More notes on ESXi 4.1 Kickstart

ESXi 4.1 kickstart is adequate for most things but I still have several issues with it that I consider ‘bugs':

1.  If you’re not connected to a network, it doesn’t work.  This is fine since most people will be on a network with VMware anyway right?  Fine, I’ll let this one slide.  But if you just have a machine and a usb stick, then why do you need the network?  Sure you’ll have it eventually but I just want to test it on my server on my desk…

2.  The kickstart file likes to stop and give you alerts even if everything is ok.  As an example:  In the post install script if I don’t put the interpreter it stops and gives me a note:  “Interpreter not specified, using busybox”  That’s fine, that’s what the default is.  Why stop me?  The docs state clearly that the default is busybox.

3.  Name resolution doesn’t work in postscripts.  If you’re trying to get information from other hosts, it doesn’t work.  Forget it.  Just put in the IP address in your post install script.

4.  USB installations without kickstart don’t work.  You need to have a CD/DVD image.  This is lame.  In an era where most servers I deal with don’t have DVD roms, why make me buy a usb DVD drive?  A $10 usb stick should do this just fine.

5.  Lack of mount support.  This kills me.  I want to have a USB drive boot up ESXi 4.1 in kickstart and then boot up with a virtual machine.  Problem is my virtual machine is 60GB.  After digging around, I see that ESXi 4.1 can get files from a FAT32 filesystem by using the mcopy command.  (It doesn’t do a mount).  But what I really want is ext3 support so that I can copy 60GB files onto a hard disk.  I’m thinking about hacking an ext3 driver for busybox, but I don’t know how difficult that will be.   Right now, my options seem to break up my disk image into 2GB chunks so they fit on the FAT 32 partition… lame.  Anyway, please don’t tell me to consider NFS and all that stuff, because I know that’s the optimal solution.  This project is a little different than what you may be thinking of.

Anyway, I don’t want to keep complaining, so here are some nice things:

Enable SSH on the TSM.

We want SSH on our machines, even if its not supported.  So we add this to our kickstart file:

%firstboot –interpreter=busybox –unsupported –level=47

sed -ie ‘s/#ssh/ssh/’ /etc/inetd.conf

Mounting USB drives

As I mentioned you can’t mount USB drives on ESXi 4.1.  (At least I haven’t figured it out yet).  You can do passthrough with the USB drives so that the VMs can mount them, but you can’t actually mount it on the hypervisor.

However you can copy files from the FAT32 partition.  Here is an example of a command to use in a kickstart file:

mcopy -i /dev/disks/mpx.vmhba32:C0:T0:L0:1 \::IMAGEDD.BZ2 /install_cache/IMAGEDD.BZ2

(In fact, this is the exact command used by the installer to grab the bz2 image from the fat 32 partition)

So if you had a file named foo on there, you could substitute it in for the IMAGEDD.BZ2 file name and copy it onto your hypervisor.  I would do this for copying *vmx files or things like that.

There’s one catch:  The mcopy command is available during installation, but upon reboot, there is no mcopy command!  So if you want it, then a good idea is to copy it during the kickstart file to some place where you can get it after its installed.

Anyway, happy VMware VMworld to all you who are going.