I’ve had the fortunate opportunity to have two Nexus 5548UPs in my lab to help test upgrade problems for one of my customers. Its been great to have some gear to play with and really try to understand how it all works together.
One of the issues I’ve run up against in the past (and you may have too) is configuring jumbo frames on network switches. Jumbo frames enable more bytes to be sent more efficiently through the data center. The default maximum transmission unit size (MTU) on nearly all networks and server NICs is 1500 bytes. That means if you want to send more traffic, then you need to send more frames. When you increase the packet size to 9000 bytes then you send less headers, less frames, and more data. The result is that it is supposed to decrease CPU load. The adverse effects are that you may end up with higher latency and you may end up configuring a lot of end points. That can get really complicated.
As an example of configuring jumbo frames in a data center consider all the endpoints that has to be configured:
- Operating System: The NIC must be set to MTU 9000. On VMware, the vSwitch has to be set to this as well as the VMs if they will be supporting jumbo frames.
- On UCS, the vNIC has to be mapped to a jumbo frames QoS policy.
- The ports on the network switch must have jumbo frames enabled on the uplink.
- The ports on the storage controller must have jumbo frames enabled.
- The storage network interfaces must have jumbo frames configured.
So you can see there is a great deal of orchestration between several teams in the data center. Everybody has to know what they are doing.
You can test if your jumbo frames are enabled on Linux by sending a simple ping:
ping -M do -s 8972
If that goes through without errors, then congratulations! You have jumbo frames enabled from node to node.
Jumbo Frames on the Nexus 5500
There is a guide on Cisco’s web page that talks about enabling Jumbo frames. But to do it, you have to do things like policy maps and class maps. I’ve often thought: Why is this so hard to do? It seems like just an easy command would be more sufficient.
The reason goes back to the standard trade offs engineers make. “Make it Easy” vs. “Make it Flexible”. You can’t really have flexibility without more nerd knobs to turn. Most of this is probably more applicable to application traffic.
The other problem I always think of: Why can’t you just apply the MTU to the interface? With the Nexus 5500 you don’t typically set it to the interface. But can you? Kind of.
QoS Class Map and Policy Maps
The architecture of Nexus 5500s is a bit different than that of Catalyst switches. The buffering is done more on the ingress ports. (The ports where traffic enters). As such the first thing you’ll want to do is “tag” or “classify” the traffic that comes into the port. You do this with a class-map command of type QoS. How do you identify traffic?
The most common way to match traffic is with either an IP Access List, or by the protocol. Let’s do it:
5k-top# conf
Enter configuration commands, one per line. End with CNTL/Z.
n5k-top(config)# class-map myTraffic
n5k-top(config-cmap-qos)# match protocol iscsi
Here we’re just matching iSCSI traffic. That’s one that we might want to do Jumbo Frames on. But we could also do something for IP addresses. Let’s say that all hosts on the 172.20.0.0/24 network should have jumbo frames. That would make sense if this network was for storage (NFS, iSCSI or whatever).
To do that we would use an access list:
n5k-top(config)# ip access-list jumbo-list
n5k-top(config-acl)# permit ip 127.20.0.0/24 any
Now we can put that on our QoS group:
n5k-top(config-acl)# class-map type qos myTraffic
n5k-top(config-cmap-qos)# no match protocol iscsi
n5k-top(config-cmap-qos)# match access-group name jumbo-list
n5k-top(config-cmap-qos)# sh class-map type qos myTraffic
Type qos class-maps
===================
class-map type qos match-all myTraffic
match access-group name jumbo-list
Great! Now we need to put this into one of the qos groups that we can work on. There’s a lot more we can do, but for simplicity, we’ll just put this into qos group 2. That is a good place to be since this traffic could be NFS or iSCSI and may be more important than normal traffic. We need to put it in a qos group because there are 2 other QoS types that we haven’t talked about, and those QoS types classify by qos-group numbers. Here’s the other two:
network-qos: This is the QoS type that allows for Jumbo Frames (mtu), multicast, pause-no drop, and other settings that show how a packet gets through the network. The shape of it and how it reacts. The others do more for what happens when a packet enters or leaves the switch.
queuing: This is the last type of QoS and does things like allocate how much bandwidth certain traffic gets as it goes through the network. By default, 100 percent of the bandwidth goes to the default class.
So now we know: We have 3 types of QoS settings and each of these settings requires a class-map (to tag the traffic) and a policy-map: what to do with all the traffic when it comes in. With policy-maps on each of these classes, you’ll associate multiple class-maps with behaviors. Finally, once we have these policy-maps for each of the three types of classes, we assign this to the system QoS.
Let’s finish the qos type. So far we’ve only made a class-map for it. But now, we want a policy-map. We may have three types of traffic: Default traffic, myTraffic, and fcoe. FCoE and the default traffic are there “by default”. So let’s make a policy-map with those traffic types:
n5k-top(config-cmap-nq)# policy-map type qos myQoS-policy
n5k-top(config-pmap-qos)# class type qos myTraffic
n5k-top(config-pmap-c-qos)# set qos-group 2
n5k-top(config-pmap-c-qos)# class type qos class-fcoe
n5k-top(config-pmap-c-qos)# set qos-group 1
There, now we have three traffic lanes marked. The default was put in there for us. Check it out:
n5k-top(config-pmap-c-qos)# sh policy-map type qos myQoS-policy
Type qos policy-maps
====================
policy-map type qos myQoS-policy
class type qos myTraffic
set qos-group 2
class type qos class-fcoe
set qos-group 1
class type qos class-default
set qos-group 0
Ok, now we want to set our jumbo frames. That means we need to create a network-qos type of QoS. First we have to mark what we want. Our only options are to classify by qos-groups for the network-qos type of QoS. So let’s create a class-map:
n5k-top(config-cmap-que)# class-map type network-qos myTraffic
n5k-top(config-cmap-nq)# match qos-group 2
Notice that I kept the name the same in the network-qos type of QoS as I did for the qos type of QoS. This makes things a bit easier. Now we just need to create a policy-map using this class-map, as well as the fcoe class-map (the fcoe class-map is created by default)
n5k-top(config-cmap-nq)# policy-map type network-qos myNetwork-QoS-policy
n5k-top(config-pmap-nq)# class type network-qos myTraffic
n5k-top(config-pmap-nq-c)# mtu 9216
n5k-top(config-pmap-nq-c)# class type network-qos class-fcoe
n5k-top(config-pmap-nq-c)# pause no-drop
n5k-top(config-pmap-nq-c)# mtu 2158
n5k-top(config-pmap-nq-c)# sh policy-map type network-qos myNetwork-QoS-policy
Type network-qos policy-maps
===============================
policy-map type network-qos myNetwork-QoS-policy
class type network-qos myTraffic
mtu 9216
class type network-qos class-fcoe
pause no-drop
mtu 2158
class type network-qos class-default
mtu 1500
multicast-optimize
Ok! 2 down, 1 to go. The queueing policy. We just need to split these loads. First we need to mark the traffic. Again, we can only use qos-groups to do that. So this time we’ll create a queuing type of traffic. We’ll call the name the same as we did the last two class-maps:
n5k-top(config-pmap-c-qos)# class-map type queuing myTraffic
n5k-top(config-cmap-que)# match qos-group 2
Now that we have that, lets associate it to a policy map. (just like the others). Here we can just split it in half. We don’t really know what our traffic will be. (Maybe you do in your network). So we’re just going to give most of it to our jumbo frame traffic (50%), and then we’ll give 25% to the other two types (fcoe and default).
n5k-top(config-pmap-nq-c)# policy-map type queuing myQueuing-policy
n5k-top(config-pmap-que)# class type queuing myTraffic
n5k-top(config-pmap-c-que)# bandwidth percent 50
n5k-top(config-pmap-c-que)# class type queuing class-fcoe
n5k-top(config-pmap-c-que)# bandwidth percent 25
n5k-top(config-pmap-c-que)# class type queuing class-default
n5k-top(config-pmap-c-que)# bandwidth percent 25
n5k-top(config-pmap-c-que)# sh policy-map type queuing myQueuing-policy
Type queuing policy-maps
========================
policy-map type queuing myQueuing-policy
class type queuing myTraffic
bandwidth percent 50
class type queuing class-fcoe
bandwidth percent 25
class type queuing class-default
bandwidth percent 25
Boom! Just like that, we now have our three QoS policies created. One for type qos called myQos-policy. One for type network-qos called myNetwork-QoS-policy. And finally, the one we just created for type queuing called myQueuing-policy.
What’s left? Now we just need to apply this to the system:
n5k-top(config-sys-qos)# service-policy type qos input myQoS-policy
n5k-top(config-sys-qos)# service-policy type network-qos myNetwork-QoS-policy
n5k-top(config-sys-qos)# service-policy type queuing input myQueuing-policy
n5k-top(config-sys-qos)# service-policy type queuing output myQueuing-policy
Notice for the Queuing policy we applied it twice. Once to the input and once to the output. The QoS type is only applied to the input, because that’s where traffic comes in and is marked. The network-qos effects input and output but should be the same for all, so we only configure it once.
Well, hopefully that wasn’t too confusing. You can obviously see the power in this. If we had voice, video, and other types of traffic coming through here, we could add it to our policy-maps after we tag it. With QoS, the settings should be the same on all switches in the datacenter. If you are running a VPC, you’ll have to make sure that the other switch has the same set up, otherwise you’ll get a Type two error:
n5k-top(config-sys-qos)# sh vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : failed
Type-2 inconsistency reason : QoSMgr Network QoS configuration incompatible
vPC role : primary
Number of vPCs configured : 3
Peer Gateway : Disabled
Dual-active excluded VLANs : –
Graceful Consistency Check : Enabled
Auto-recovery status : Disabled
vPC Peer-link status
———————————————————————
id Port Status Active vlans
— —- —— ————————————————–
1 Po1 up 1,500
vPC status
—————————————————————————-
id Port Status Consistency Reason Active vlans
—— ———– —— ———– ————————– ———–
10 Po10 up success success 1,100,500
20 Po20 up success success 1,100,500
80 Po80 up success success 1,100,500
See? Nobody wants a type 2 error. Apply settings throughout the data center!
Hope this helps someone struggling with getting jumbo frames on a Nexus 5k.
Comments are closed.