Category Archives: Nexus 5000

A few Nexus notes

I’ve been working with OTV and Fabricpath and I thought I’d put a few pointers down that I learned that had me scratching my head for a while.

1.  To makes sure that OTV is working, the join interfaces on each side must be able to ping each other.  Let’s say that site A has join interface 192.168.101.1 and site B has a join interface 192.168.102.1.  Before OTV can work, from the OTV VDC, you need to make sure that they can ping each other.  This will usually mean that routing is set up properly.

2.  For Fabricpath to work with OTV, I had my main aggregation VDC connected to the OTV VDC through an M1 interface.  This doesn’t work.  Instead, connect the OTV VDC to the aggregation VDC through an F1 interface.  This makes it so Fabricpath is terminated and moved into classical ethernet.  I scratched my head for probably 4 hours last night until I thought of trying that this morning.  Lessons learned.

Hopefully that helps someone.

QoS and Jumbo Frames on the Nexus 5500

Nexus 5548 UP

 

I’ve had the fortunate opportunity to have two Nexus 5548UPs in my lab to help test upgrade problems for one of my customers.  Its been great to have some gear to play with and really try to understand how it all works together.

One of the issues I’ve run up against in the past (and you may have too) is configuring jumbo frames on network switches.  Jumbo frames enable more bytes to be sent more efficiently through the data center.  The default maximum transmission unit size (MTU) on nearly all networks and server NICs is 1500 bytes.  That means if you want to send more traffic, then you need to send more frames.  When you increase the packet size to 9000 bytes then you send less headers, less frames, and more data.  The result is that it is supposed to decrease CPU load.  The adverse effects are that you may end up with higher latency and you may end up configuring a lot of end points.  That can get really complicated.

As an example of configuring jumbo frames in a data center consider all the endpoints that has to be configured:

  • Operating System:  The NIC must be set to MTU 9000.  On VMware, the vSwitch has to be set to this as well as the VMs if they will be supporting jumbo frames.
  • On UCS, the vNIC has to be mapped to a jumbo frames QoS policy.
  • The ports on the network switch must have jumbo frames enabled on the uplink.
  • The ports on the storage controller must have jumbo frames enabled.
  • The storage network interfaces must have jumbo frames configured.

So you can see there is a great deal of orchestration between several teams in the data center.  Everybody has to know what they are doing.

You can test if your jumbo frames are enabled on Linux by sending a simple ping:

1
ping -M do -s 8972 <node>

If that goes through without errors, then congratulations!  You have jumbo frames enabled from node to node.

Jumbo Frames on the Nexus 5500

There is a guide on Cisco’s web page that talks about enabling Jumbo frames.  But to do it, you have to do things like policy maps and class maps.  I’ve often thought:  Why is this so hard to do?  It seems like just an easy command would be more sufficient.

The reason goes back to the standard trade offs engineers make.  “Make it Easy” vs. “Make it Flexible”.   You can’t really have flexibility without more nerd knobs to turn.  Most of this is probably more applicable to application traffic.

The other problem I always think of:  Why can’t you just apply the MTU to the interface?  With the Nexus 5500 you don’t typically set it to the interface.  But can you?  Kind of.

QoS Class Map and Policy Maps

The architecture of Nexus 5500s is a bit different than that of Catalyst switches.  The buffering is done more on the ingress ports.  (The ports where traffic enters).  As such the first thing you’ll want to do is “tag” or “classify” the traffic that comes into the port.  You do this with a class-map command of type QoS.  How do you identify traffic?

The most common way to match traffic is with either an IP Access List, or by the protocol.  Let’s do it:

1
2
3
4
5k-top# conf
Enter configuration commands, one per line. End with CNTL/Z.
n5k-top(config)# class-map myTraffic
n5k-top(config-cmap-qos)# match protocol iscsi

Here we’re just matching iSCSI traffic. That’s one that we might want to do Jumbo Frames on. But we could also do something for IP addresses. Let’s say that all hosts on the 172.20.0.0/24 network should have jumbo frames. That would make sense if this network was for storage (NFS, iSCSI or whatever).

To do that we would use an access list:

1
2
n5k-top(config)# ip access-list jumbo-list
n5k-top(config-acl)# permit ip 127.20.0.0/24 any

Now we can put that on our QoS group:

1
2
3
4
n5k-top(config-acl)# class-map type qos myTraffic
n5k-top(config-cmap-qos)# no match protocol iscsi
n5k-top(config-cmap-qos)# match access-group name jumbo-list
n5k-top(config-cmap-qos)# sh class-map type qos myTraffic
1
2
3
4
Type qos class-maps
===================
class-map type qos match-all myTraffic
match access-group name jumbo-list

 

Great! Now we need to put this into one of the qos groups that we can work on. There’s a lot more we can do, but for simplicity, we’ll just put this into qos group 2.  That is a good place to be since this traffic could be NFS or iSCSI and may be more important than normal traffic.  We need to put it in a qos group because there are 2 other QoS types that we haven’t talked about, and those QoS types classify by qos-group numbers.   Here’s the other two:

network-qos: This is the QoS type that allows for Jumbo Frames (mtu), multicast, pause-no drop, and other settings that show how a packet gets through the network.  The shape of it and how it reacts.  The others do more for what happens when a packet enters or leaves the switch.

queuing: This is the last type of QoS and does things like allocate how much bandwidth certain traffic gets as it goes through the network.  By default, 100 percent of the bandwidth goes to the default class.

So now we know:  We have 3 types of QoS settings and each of these settings requires a class-map (to tag the traffic) and a policy-map: what to do with all the traffic when it comes in.  With policy-maps on each of these classes, you’ll associate multiple class-maps with behaviors.  Finally, once we have these policy-maps for each of the three types of classes, we assign this to the system QoS.

Let’s finish the qos type.  So far we’ve only made a class-map for it.  But now, we want a policy-map.  We may have three types of traffic:  Default traffic, myTraffic, and fcoe.  FCoE and the default traffic are there “by default”.  So let’s make a policy-map with those traffic types:

1
2
3
4
5
n5k-top(config-cmap-nq)# policy-map type qos myQoS-policy
n5k-top(config-pmap-qos)# class type qos myTraffic
n5k-top(config-pmap-c-qos)# set qos-group 2
n5k-top(config-pmap-c-qos)# class type qos class-fcoe
n5k-top(config-pmap-c-qos)# set qos-group 1

There, now we have three traffic lanes marked. The default was put in there for us. Check it out:

1
n5k-top(config-pmap-c-qos)# sh policy-map type qos myQoS-policy

Type qos policy-maps
====================

policy-map type qos myQoS-policy
class type qos myTraffic
set qos-group 2
class type qos class-fcoe
set qos-group 1
class type qos class-default
set qos-group 0

Ok, now we want to set our jumbo frames. That means we need to create a network-qos type of QoS. First we have to mark what we want. Our only options are to classify by qos-groups for the network-qos type of QoS. So let’s create a class-map:

1
2
n5k-top(config-cmap-que)# class-map type network-qos myTraffic
n5k-top(config-cmap-nq)# match qos-group 2

Notice that I kept the name the same in the network-qos type of QoS as I did for the qos type of QoS. This makes things a bit easier. Now we just need to create a policy-map using this class-map, as well as the fcoe class-map (the fcoe class-map is created by default)

1
2
3
4
5
6
7
n5k-top(config-cmap-nq)# policy-map type network-qos myNetwork-QoS-policy
n5k-top(config-pmap-nq)# class type network-qos myTraffic
n5k-top(config-pmap-nq-c)# mtu 9216
n5k-top(config-pmap-nq-c)# class type network-qos class-fcoe
n5k-top(config-pmap-nq-c)# pause no-drop
n5k-top(config-pmap-nq-c)# mtu 2158
n5k-top(config-pmap-nq-c)# sh policy-map type network-qos myNetwork-QoS-policy

Type network-qos policy-maps
===============================

policy-map type network-qos myNetwork-QoS-policy
class type network-qos myTraffic

mtu 9216
class type network-qos class-fcoe

pause no-drop
mtu 2158
class type network-qos class-default

mtu 1500
multicast-optimize

Ok! 2 down, 1 to go. The queueing policy. We just need to split these loads. First we need to mark the traffic. Again, we can only use qos-groups to do that. So this time we’ll create a queuing type of traffic. We’ll call the name the same as we did the last two class-maps:

1
2
n5k-top(config-pmap-c-qos)# class-map type queuing myTraffic
n5k-top(config-cmap-que)# match qos-group 2

Now that we have that, lets associate it to a policy map. (just like the others). Here we can just split it in half. We don’t really know what our traffic will be. (Maybe you do in your network). So we’re just going to give most of it to our jumbo frame traffic (50%), and then we’ll give 25% to the other two types (fcoe and default).

1
2
3
4
5
6
7
8
n5k-top(config-pmap-nq-c)# policy-map type queuing myQueuing-policy
n5k-top(config-pmap-que)# class type queuing myTraffic
n5k-top(config-pmap-c-que)# bandwidth percent 50
n5k-top(config-pmap-c-que)# class type queuing class-fcoe
n5k-top(config-pmap-c-que)# bandwidth percent 25
n5k-top(config-pmap-c-que)# class type queuing class-default
n5k-top(config-pmap-c-que)# bandwidth percent 25
n5k-top(config-pmap-c-que)# sh policy-map type queuing myQueuing-policy

Type queuing policy-maps
========================

policy-map type queuing myQueuing-policy
class type queuing myTraffic
bandwidth percent 50
class type queuing class-fcoe
bandwidth percent 25
class type queuing class-default
bandwidth percent 25

Boom! Just like that, we now have our three QoS policies created. One for type qos called myQos-policy. One for type network-qos called myNetwork-QoS-policy. And finally, the one we just created for type queuing called myQueuing-policy.

What’s left? Now we just need to apply this to the system:

1
2
3
4
n5k-top(config-sys-qos)# service-policy type qos input myQoS-policy
n5k-top(config-sys-qos)# service-policy type network-qos myNetwork-QoS-policy
n5k-top(config-sys-qos)# service-policy type queuing input myQueuing-policy
n5k-top(config-sys-qos)# service-policy type queuing output myQueuing-policy

Notice for the Queuing policy we applied it twice. Once to the input and once to the output. The QoS type is only applied to the input, because that’s where traffic comes in and is marked. The network-qos effects input and output but should be the same for all, so we only configure it once.

Well, hopefully that wasn’t too confusing.  You can obviously see the power in this.  If we had voice, video, and other types of traffic coming through here, we could add it to our policy-maps after we tag it.  With QoS, the settings should be the same on all switches in the datacenter.  If you are running a VPC, you’ll have to make sure that the other switch has the same set up, otherwise you’ll get a Type two error:

1
2
3
n5k-top(config-sys-qos)# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : failed
Type-2 inconsistency reason : QoSMgr Network QoS configuration incompatible
vPC role : primary
Number of vPCs configured : 3
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Disabled

vPC Peer-link status
———————————————————————
id Port Status Active vlans
– —- —— ————————————————–
1 Po1 up 1,500

vPC status
—————————————————————————-
id Port Status Consistency Reason Active vlans
—— ———– —— ———– ————————– ———–
10 Po10 up success success 1,100,500
20 Po20 up success success 1,100,500
80 Po80 up success success 1,100,500

See? Nobody wants a type 2 error. Apply settings throughout the data center!

Hope this helps someone struggling with getting jumbo frames on a Nexus 5k.

FCoE with UCS C-Series

I have in my lab a C210 that I want to turn into an FCoE target storage.  I’ll write more on that in another post.  The first challenge was to get it up with FCoE.  Its attached to a pair of Nexus 5548s.  I installed RedHat Linux 6.5 on the C210 and booted up.  The big issue I had was that even though RedHat Linux 6.5 comes with the fnic and enic drivers, the FCoE never happened.  It wasn’t until I installed the updated drivers from Cisco that I finally saw a flogi.  But there were other tricks that you had to do to make the C210 actually work with FCoE.

C210 CIMC

The first part to start is looking in the CIMC (with the machine powered on) and configure the vHBAs. From the GUI go to:

Server -> Inventory

Then on the work pane, the ‘Network Adapters’ tab, then down below select vHBAs.  Here you will see two vHBAs by default.  From here you have to set the VLAN that the vHBA will go over.  Clicking the ‘Properties’ on the interface you have to select the VLAN.  I set the MAC address to ‘AUTO’ based on a TAC case I looked at, but this never persisted.  From there I entered the VLAN.  VLAN 10 for the first interface and VLAN 20 for the second interface.  This VLAN 10 matches the FCoE VLAN and VSAN that I created on the Nexus 5548.  On the other Nexus I creed VLAN 20 to match FCoE VLAN 20 and VSAN 20.

This then seemed to require a reboot of the Linux Server for the VLANs to take effect.  In hindsight this is something I probably should have done first.

RedHat Linux 6.5

This needs to have the Cisco drivers for the fnic.  You might want to install the enic drivers as well.  I got these from cisco.com.  I used the B series drivers and it was a 1.2GB file that I had to download all to get a 656KB driver package.  I installed the kmod-fnic-1.6.0.6-1 RPM.  I had a customer who had updated to a later kernel and he had to install the kernel-devel rpm and recompile the driver.  After it came up, it worked for him.

With the C210 I wanted to bond the 10Gb NICs into a vPC.  So I did an LACP bond with Linux.  This was done as follows:

Created file: /etc/modprobe.d/bond.conf

alias bond0 bonding
options bonding mode=4 miimon=100 lacp_rate=1

Created file: /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
IPADDR=172.20.1.1
ONBOOT=yes
NETMASK=255.255.0.0
STARTMODE=onboot
MTU=9000

Edited the /etc/sysconfig/network-scripts/ifcfg-eth2

DEVICE=eth2
MASTER=bond0
SLAVE=yes
HWADDR=58:8D:09:0F:14:BE
TYPE=Ethernet
UUID=8bde8c1f-926f-4960-87ff-c0973f5ef921
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none

Edited the /etc/sysconfig/network-scripts/ifcfg-eth3

DEVICE=eth3
MASTER=bond0
SLAVE=yes
HWADDR=58:8D:09:0F:14:BF
TYPE=Ethernet
UUID=6e2e7493-c1a1-4164-9215-04f0584b338c
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none

Next restart the network and you should have a bond. You may need to restart this after you configure the Nexus 5548 side.

service network restart

Nexus 5548 Top
Log in and create VPCs and stuff.  Also don’t forget to do the MTU 9000 system class.  I use this for jumbo frames in the data center.

policy-map type network-qos jumbo
class type network-qos class-default
mtu 9216
multicast-optimize
system qos
service-policy type network-qos jumbo

One thing that drives me crazy is that you can’t do sh int po 4 to see that the MTU is 9000. From the documents, you have to do

sh queuing int po 4

to see that your jumbo frames are enabled.

The C210 is attached to ethernet port 1 on each of the switches.  Here’s the Ethernet configuration:

The ethernet:

interface Ethernet1/1
switchport mode trunk
switchport trunk allowed vlan 1,10
spanning-tree port type edge trunk
channel-group 4

The port channel:

interface port-channel4
switchport mode trunk
switchport trunk allowed vlan 1,10
speed 10000
vpc 4

As you can see VLAN 10 is the VSAN. We need to create the VSAN info for that.

feature fcoe
vsan database
vsan 10
vlan 10
fcoe vsan 10

Finally, we need to create the vfc for the interface:

interface vfc1
bind interface Ethernet1/1
switchport description Connection to NFS server FCoE
no shutdown
vsan database
vsan 10 interface vfc1

Nexus 5548 Bottom
The other Nexus is similar configuration.  The difference is that instead of VSAN 10, VLAN 10, we use VSAN20, VLAN 20 and bind the FCoE to VSAN 20.  In the SAN world, we don’t cross the streams.  You’ll see that the VLANS are not the same in the two switches.

Notice that in the below configuration, VLAN 20 nor 10 is defined for through the peer link so you’ll only see VLAN 1 enabled on the vPC:

N5k-bottom# sh vpc consistency-parameters interface po 4

Legend:
Type 1 : vPC will be suspended in case of mismatch

Name Type Local Value Peer Value
————- —- ———————- ———————–
Shut Lan 1 No No
STP Port Type 1 Default Default
STP Port Guard 1 None None
STP MST Simulate PVST 1 Default Default
mode 1 on on
Speed 1 10 Gb/s 10 Gb/s
Duplex 1 full full
Port Mode 1 trunk trunk
Native Vlan 1 1 1
MTU 1 1500 1500
Admin port mode 1
lag-id 1
vPC card type 1 Empty Empty
Allowed VLANs – 1 1
Local suspended VLANs – – -

But on the individual nodes you’ll see that the VLAN is enabled in the VPC. VLAN 10 is carrying storage traffic.

# sh vpc 4

vPC status
—————————————————————————-
id Port Status Consistency Reason Active vlans
—— ———– —— ———– ————————– ———–
4 Po4 up success success 1,10

Success?

How do you know you succeeded?

N5k-bottom# sh flogi database
——————————————————————————–
INTERFACE VSAN FCID PORT NAME NODE NAME
——————————————————————————–
vfc1 10 0x2d0000 20:00:58:8d:09:0f:14:c1 10:00:58:8d:09:0f:14:c1

Total number of flogi = 1.

You’ll see the login. If not, then try restarting the interface on the Linux side. You should see a different WWPN in each Nexus. Another issue you might have is that the VLANS may be mismatched, so make sure you have the right node on the right server.

Let me know how it worked for you!

Change a Fabric Interconnect into a Nexus Switch

I got a Nexus 5010 from our spare parts department.  When I booted it up, lo and behold, it thought it was a UCS Fabric Interconnect 6020!

As most people know the 6120XP is the same hardware as the Nexus 5010.  Only difference is that its spray painted green.  Well this particular model I got was gray and said it was a Nexus 5010.  So I was bound and determined to get it back.  I got pretty close, and wanted to write down the steps I took.

I’m sad to say, however, that I didn’t get it to work all the way.

Here’s what I did:

Step 1. Get TFTP server setup (This explains how to do it for a MacBook Pro)
I’m running Mountain Lion OSX. Turns out there is a default tftp server installed with it. Getting it running is pretty easy. Just run:

sudo launchctl load -F /System/Library/LaunchDaemons/tftp.plist

(Turning it off is done with:

sudo launchctl unload -F /System/Library/LaunchDaemons/tftp.plist

)
(To see if its running run:

sudo launchctl list | grep tftp,

if you see output its running, if not, its not)

From there you need to put the files you need into the /private/tftpboot/

I went to Cisco’s support page and easily found two files:
n5000-uk9.5.2.1.N1.5.bin < the software file
and
n5000-uk9-kickstart.5.2.1.N1.5.bin < the kickstart file

I had to copy them with sudo since you’re going into a privileged directory.

You should test your tftp server to make sure it works. No use yelling at the Nexus 5000 for telling you it can’t access the file.

From the command prompt on the mac:

cd ~/Desktop
tftp localhost
get n5000-uk9.5.2.1.N1.5.bin

If that works, you are in business.

Step 2. Load the Nexus 5000 (that thinks its a 6100) into the loader prompt.

When the machine started booting, I had to do

Ctrl+Shift+R

right as it was loading the UCS kickstart file. Doing this got me to a
loader>

prompt.

From here, we don’t have a lot of options. But all we need to do is set the mgmt0 interface and kickstart from our Nexus image that we have on tftp.
(Incidently, at this point I ran the dir command to see if there were any nexus images, and there wasn’t! Only UCS images. )

Here’s how we set that:

loader> set ip 192.168.1.99 255.255.255.0

Then it confirmed that this was good. Now, to load up the kickstart file:

loader> boot tftp://192.168.1.234/private/tftpboot/n5000-u9-kickstart.5.2.1.N1.5.bin
Address: 192.168.1.99
Netmask: 255.255.255.0
Server: 192.168.1.234
Gateway: 0.0.0.0
Booting: /private/tftpboot/n5000-uk9-kickstart.5.2.1.N1.5.bin console=ttyS0,960
0n8nn

the system then boots up. Does some image verification and loads into a boot prompt:

Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2013, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and

http://www.opensource.org/licenses/lgpl-2.1.php

switch(boot)#

Step 3: Copy files and continue booting

Now we just need to get the files on the device.

switch(boot)# con t
switch(boot)(config)# inter mgmt 0
switch(boot)(config-if)# ip address 192.168.1.99 255.255.255.0
switch(boot)(config-if)# no shutdown
switch(boot)(config-if)# exit
switch(boot)(config)# exit
switch(boot)# copy tftp: boot flash:
switch(boot)# copy tftp: bootflash:
Enter source filename: /private/tftpboot/n5000-uk9.5.2.1.N1.5.bin
Enter hostname for the tftp server: 192.168.1.234
Trying to connect to tftp server……
Connection to server Established. Copying Started…..

At this point I went downstairs and had some chips to eat. I got back and had to wait like 15-20 min for it to copy. Shesh! Finally, when I was about to cancel it, I saw:

TFTP get operation was successful
Copy complete, now saving to disk (please wait)…

Now we need to get the kickstart file:

switch(boot)# copy tftp://192.168.1.234/n5000-uk9-kickstart.5.2.1.N1.5.bin boot flash:

So I waited some more, this one didn’t take as long.

Then I deleted a bunch of UCS files:

switch(boot)# delete bootflash:ucs-6100-k9-system.4.0.1a.N2.1.0.1036.gbin
switch(boot)# delete bootflash:cisco_nexus_1000v_certificate.pem
switch(boot)# delete bootflash:ucs-6100-k9-kickstart.4.0.1a.N2.1.0.1036.gbin
switch(boot)# delete bootflash:ucs-6100-k9-kickstart.4.0.1a.N2.1.0.1056d.gbin
switch(boot)# delete bootflash:ucs-6100-k9-system.4.0.1a.N2.1.0.1056d.gbin
switch(boot)# delete bootflash:ucs-manager-k9.1.0.0.1036.gbin
switch(boot)# delete bootflash:ucs-manager-k9.1.0.0.1056d.gbin

Then I booted the image:

switch(boot)# load n5000-uk9.5.2.1.N1.5.bin

This set me to the boot prompt again. So I hit exit:

boot switch(boot)# exit

It kept rebooting to stored images of UCS manager. So I found this command:

init system check-filesystem

From here, I repeated the operation of downloading the 2 Nexus images.  At least now it didn’t boot up into UCS Fabric Interconnect, but I could never get it to go to regular Cisco Nexus 5010.  It may be that there was something wrong with the hardware.  It certainly looks a little beat if you look at this hardware.  If nothing else, I learned a little more about the boot files in the Nexus 5000.