Nexus 1000v Layer 3

Layer 3 mode is the recommended way to configure VSM to VEM communication in the Nexus 1000v.   Layer 3 mode keeps things simple and easier to troubleshoot.

I kept my design very simple.  There’s one VLAN (509) that I run my ESXi hosts on.  The IP addresses are 192.168.40.xxx.  Just to give you an example:

vCenter: 192.168.40.2
ESXi Host1: 192.168.40.101
ESXi Host2: 192.168.40.102

N1kv: 192.168.40.31

Using this I had a simple uplink port-profile defined:

nexus1000v(config-port-prof)# show port-profile name uplink

port-profile uplink
 type: Ethernet
 description: 
 status: enabled
 max-ports: 32
 min-ports: 1
 inherit:
 config attributes:
  switchport mode trunk
  switchport trunk allowed vlan 1,501,506,509-510,714,3967
  channel-group auto mode on mac-pinning
  no shutdown
 evaluated config attributes:
  switchport mode trunk
  switchport trunk allowed vlan 1,501,506,509-510,714,3967
  channel-group auto mode on mac-pinning
  no shutdown
 assigned interfaces:
  port-channel1
  port-channel2
  Ethernet3/1
  Ethernet3/2
  Ethernet4/1
  Ethernet4/2
 port-group: uplink
 system vlans: 1,501,506,509-510,714
 capability l3control: no
 capability iscsi-multipath: no
 capability vxlan: no
 capability l3-vn-service: no
 port-profile role: none
 port-binding: static

And a simple management port-profile:

nexus1000v(config-port-prof)# show port-profile name management

port-profile management
 type: Vethernet
 description: 
 status: enabled
 max-ports: 32
 min-ports: 1
 inherit:
 config attributes:
  switchport mode access
  switchport access vlan 509
  no shutdown
 evaluated config attributes:
  switchport mode access
  switchport access vlan 509
  no shutdown
 assigned interfaces:
  Vethernet1
  Vethernet4
 port-group: management
 system vlans: 509
 capability l3control: yes
 capability iscsi-multipath: no
 capability vxlan: no
 capability l3-vn-service: no
 port-profile role: none
 port-binding: static

I had everything set up right… I thought.  The only problem was (before, not in the output above) is that I couldn’t see my VEMs! They were all hooked up in vCenter and I was even running traffic through them. But no VEMs:

nexus1000v(config)# show module vem
No Virtual Ethernet Modules found.

I finally stumbled upon this nice document and realized I hadn’t enabled l3control.  Doing that:

nexus1000v(config-port-prof)# capability l3control

And Bam!  Everything worked:

nexus1000v(config-port-prof)# show module vem
Mod  Ports  Module-Type                       Model               Status
---  -----  --------------------------------  ------------------  ------------
3    248    Virtual Ethernet Module           NA                  ok
4    248    Virtual Ethernet Module           NA                  ok

Mod  Sw                  Hw      
---  ------------------  ------------------------------------------------  
3    4.2(1)SV1(5.1a)     VMware ESXi 5.0.0 Releasebuild-469512 (3.0)      
4    4.2(1)SV1(5.1a)     VMware ESXi 5.0.0 Releasebuild-469512 (3.0)      

Mod  MAC-Address(es)                         Serial-Num
---  --------------------------------------  ----------
3    02-00-0c-00-03-00 to 02-00-0c-00-03-80  NA
4    02-00-0c-00-04-00 to 02-00-0c-00-04-80  NA

Mod  Server-IP        Server-UUID                           Server-Name
---  ---------------  ------------------------------------  --------------------
3    192.168.40.101   00000000-0000-0000-cafe-00000000000f  192.168.40.101
4    192.168.40.102   00000000-0000-0000-cafe-00000000000e  192.168.40.102