{"id":831,"date":"2014-01-31T14:06:20","date_gmt":"2014-01-31T20:06:20","guid":{"rendered":"http:\/\/benincosa.com\/blog\/?p=831"},"modified":"2014-11-19T11:23:55","modified_gmt":"2014-11-19T17:23:55","slug":"qos-and-jumbo-frames-on-the-nexus-5500","status":"publish","type":"post","link":"https:\/\/benincosa.com\/?p=831","title":{"rendered":"QoS and Jumbo Frames on the Nexus 5500"},"content":{"rendered":"<figure style=\"width: 600px\" class=\"wp-caption aligncenter\"><img decoding=\"async\" loading=\"lazy\" src=\"http:\/\/trouter.co.uk\/images\/nexus_5548up_switch_large.png\" alt=\"\" width=\"600\" height=\"125\" \/><figcaption class=\"wp-caption-text\">Nexus 5548 UP<\/figcaption><\/figure>\n<p>&nbsp;<\/p>\n<p>I&#8217;ve had the fortunate opportunity to have two Nexus 5548UPs in my lab to help test upgrade problems for one of my customers. \u00a0Its been great to have some gear to play with and really try to understand how it all works together.<\/p>\n<p>One of the issues I&#8217;ve run up against in the past (and you may have too) is configuring <a href=\"http:\/\/en.wikipedia.org\/wiki\/Jumbo_frame\">jumbo frames<\/a> on network switches. \u00a0Jumbo frames enable more bytes to be sent more efficiently through the data center. \u00a0The default maximum transmission unit size (MTU) on nearly all networks and server NICs is 1500 bytes. \u00a0That means if you want to send more traffic, then you need to send more frames. \u00a0When you increase the packet size to 9000 bytes then you send less headers, less frames, and more data. \u00a0The result is that it is supposed to decrease CPU load. \u00a0The adverse effects are that you may end up with higher latency and you may end up configuring a lot of end points. \u00a0That can get really complicated.<\/p>\n<p>As an example of configuring jumbo frames in a data center consider all the endpoints that has to be configured:<\/p>\n<ul>\n<li>Operating System: \u00a0The NIC must be set to MTU 9000. \u00a0On VMware, the vSwitch has to be set to this as well as the VMs if they will be supporting jumbo frames.<\/li>\n<li>On UCS, the vNIC has to be mapped to a jumbo frames QoS policy.<\/li>\n<li>The ports on the network switch must have jumbo frames enabled on the uplink.<\/li>\n<li>The ports on the storage controller must have jumbo frames enabled.<\/li>\n<li>The storage network interfaces must have jumbo frames configured.<\/li>\n<\/ul>\n<p>So you can see there is a great deal of orchestration between several teams in the data center. \u00a0Everybody has to know what they are doing.<\/p>\n<p>You can test if your jumbo frames are enabled on Linux by sending a simple ping:<\/p>\n<p><code>ping -M do -s 8972 <node><\/code><br \/>\nIf that goes through without errors, then congratulations! \u00a0You have jumbo frames enabled from node to node.<\/p>\n<p><strong>Jumbo Frames on the Nexus 5500<\/strong><\/p>\n<p>There is a <a href=\"http:\/\/www.cisco.com\/en\/US\/docs\/switches\/datacenter\/nexus5000\/sw\/qos\/Cisco_Nexus_5000_Series_NX-OS_Quality_of_Service_Configuration_Guide_chapter3.html#con_1150612\">guide<\/a> on Cisco&#8217;s web page that talks about enabling Jumbo frames. \u00a0But to do it, you have to do things like policy maps and class maps. \u00a0I&#8217;ve often thought: \u00a0Why is this so hard to do? \u00a0It seems like just an easy command would be more sufficient.<\/p>\n<p>The reason goes back to the standard trade offs engineers make. \u00a0&#8220;Make it Easy&#8221; vs. &#8220;Make it Flexible&#8221;. \u00a0 You can&#8217;t really have flexibility without more nerd knobs to turn. \u00a0Most of this is probably more applicable to application traffic.<\/p>\n<p>The other problem I always think of: \u00a0Why can&#8217;t you just apply the MTU to the interface? \u00a0With the Nexus 5500 you don&#8217;t typically set it to the interface. \u00a0But can you? \u00a0Kind of.<\/p>\n<p><strong>QoS Class Map and Policy Maps<\/strong><\/p>\n<p>The architecture of Nexus 5500s is a bit different than that of Catalyst switches. \u00a0The buffering is done more on the ingress ports. \u00a0(The ports where traffic enters). \u00a0As such the first thing you&#8217;ll want to do is &#8220;tag&#8221; or &#8220;classify&#8221; the traffic that comes into the port. \u00a0You do this with a class-map command of type QoS. \u00a0How do you identify traffic?<\/p>\n<p>The most common way to match traffic is with either an IP Access List, or by the protocol. \u00a0Let&#8217;s do it:<br \/>\n<code><br \/>\n5k-top# conf<br \/>\nEnter configuration commands, one per line. End with CNTL\/Z.<br \/>\nn5k-top(config)# class-map myTraffic<br \/>\nn5k-top(config-cmap-qos)# match protocol iscsi<br \/>\n<\/code><\/p>\n<p>Here we&#8217;re just matching iSCSI traffic. That&#8217;s one that we might want to do Jumbo Frames on. But we could also do something for IP addresses. Let&#8217;s say that all hosts on the 172.20.0.0\/24 network should have jumbo frames. That would make sense if this network was for storage (NFS, iSCSI or whatever).<\/p>\n<p>To do that we would use an access list:<\/p>\n<p><code><br \/>\nn5k-top(config)# ip access-list jumbo-list<br \/>\nn5k-top(config-acl)# permit ip 127.20.0.0\/24 any<br \/>\n<\/code><\/p>\n<p>Now we can put that on our QoS group:<br \/>\n<code><br \/>\nn5k-top(config-acl)# class-map type qos myTraffic<br \/>\nn5k-top(config-cmap-qos)# no match protocol iscsi<br \/>\nn5k-top(config-cmap-qos)# match access-group name jumbo-list<br \/>\nn5k-top(config-cmap-qos)# sh class-map type qos myTraffic<br \/>\n<\/code><br \/>\n<code><br \/>\nType qos class-maps<br \/>\n===================<br \/>\nclass-map type qos match-all myTraffic<br \/>\nmatch access-group name jumbo-list<\/code><\/p>\n<p>&nbsp;<\/p>\n<p>Great! Now we need to put this into one of the qos groups that we can work on. There&#8217;s a lot more we can do, but for simplicity, we&#8217;ll just put this into qos group 2. \u00a0That is a good place to be since this traffic could be NFS or iSCSI and may be more important than normal traffic. \u00a0We need to put it in a qos group because there are 2 other QoS types that we haven&#8217;t talked about, and those QoS types classify by qos-group numbers. \u00a0 Here&#8217;s the other two:<\/p>\n<p><strong>network-qos<\/strong>: This is the QoS type that allows for Jumbo Frames (mtu), multicast, pause-no drop, and other settings that show how a packet gets through the network. \u00a0The shape of it and how it reacts. \u00a0The others do more for what happens when a packet enters or leaves the switch.<\/p>\n<p><strong>queuing:\u00a0<\/strong>This is the last type of QoS and does things like allocate how much bandwidth certain traffic gets as it goes through the network. \u00a0By default, 100 percent of the bandwidth goes to the default class.<\/p>\n<p>So now we know: \u00a0We have 3 types of QoS settings and each of these settings requires a class-map (to tag the traffic) and a policy-map: what to do with all the traffic when it comes in. \u00a0With policy-maps on each of these classes, you&#8217;ll associate multiple class-maps with behaviors. \u00a0Finally, once we have these policy-maps for each of the three types of classes, we assign this to the system QoS.<\/p>\n<p>Let&#8217;s finish the\u00a0<strong>qos<\/strong> type. \u00a0So far we&#8217;ve only made a class-map for it. \u00a0But now, we want a policy-map. \u00a0We may have three types of traffic: \u00a0Default traffic, myTraffic, and fcoe. \u00a0FCoE and the default traffic are there &#8220;by default&#8221;. \u00a0So let&#8217;s make a policy-map with those traffic types:<br \/>\n<code><br \/>\nn5k-top(config-cmap-nq)# policy-map type qos myQoS-policy<br \/>\nn5k-top(config-pmap-qos)# class type qos myTraffic<br \/>\nn5k-top(config-pmap-c-qos)# set qos-group 2<br \/>\nn5k-top(config-pmap-c-qos)# class type qos class-fcoe<br \/>\nn5k-top(config-pmap-c-qos)# set qos-group 1<br \/>\n<\/code><br \/>\nThere, now we have three traffic lanes marked. The default was put in there for us. Check it out:<br \/>\n<code><br \/>\nn5k-top(config-pmap-c-qos)# sh policy-map type qos myQoS-policy<\/code><\/p>\n<p>Type qos policy-maps<br \/>\n====================<\/p>\n<p>policy-map type qos myQoS-policy<br \/>\nclass type qos myTraffic<br \/>\nset qos-group 2<br \/>\nclass type qos class-fcoe<br \/>\nset qos-group 1<br \/>\nclass type qos class-default<br \/>\nset qos-group 0<\/p>\n<p>Ok, now we want to set our jumbo frames. That means we need to create a <strong>network-qos<\/strong> type of QoS. First we have to mark what we want. Our only options are to classify by qos-groups for the network-qos type of QoS. So let&#8217;s create a class-map:<br \/>\n<code><br \/>\nn5k-top(config-cmap-que)# class-map type network-qos myTraffic<br \/>\nn5k-top(config-cmap-nq)# match qos-group 2<br \/>\n<\/code><br \/>\nNotice that I kept the name the same in the network-qos type of QoS as I did for the qos type of QoS. This makes things a bit easier. Now we just need to create a policy-map using this class-map, as well as the fcoe class-map (the fcoe class-map is created by default)<\/p>\n<p><code><br \/>\nn5k-top(config-cmap-nq)# policy-map type network-qos myNetwork-QoS-policy<br \/>\nn5k-top(config-pmap-nq)# class type network-qos myTraffic<br \/>\nn5k-top(config-pmap-nq-c)# mtu 9216<br \/>\nn5k-top(config-pmap-nq-c)# class type network-qos class-fcoe<br \/>\nn5k-top(config-pmap-nq-c)# pause no-drop<br \/>\nn5k-top(config-pmap-nq-c)# mtu 2158<br \/>\nn5k-top(config-pmap-nq-c)# sh policy-map type network-qos myNetwork-QoS-policy<\/code><\/p>\n<p>Type network-qos policy-maps<br \/>\n===============================<\/p>\n<p>policy-map type network-qos myNetwork-QoS-policy<br \/>\nclass type network-qos myTraffic<\/p>\n<p>mtu 9216<br \/>\nclass type network-qos class-fcoe<\/p>\n<p>pause no-drop<br \/>\nmtu 2158<br \/>\nclass type network-qos class-default<\/p>\n<p>mtu 1500<br \/>\nmulticast-optimize<\/p>\n<p>Ok! 2 down, 1 to go. The queueing policy. We just need to split these loads. First we need to mark the traffic. Again, we can only use qos-groups to do that. So this time we&#8217;ll create a <strong>queuing<\/strong> type of traffic. We&#8217;ll call the name the same as we did the last two class-maps:<\/p>\n<p><code><br \/>\nn5k-top(config-pmap-c-qos)# class-map type queuing myTraffic<br \/>\nn5k-top(config-cmap-que)# match qos-group 2<br \/>\n<\/code><\/p>\n<p>Now that we have that, lets associate it to a policy map. (just like the others). Here we can just split it in half. We don&#8217;t really know what our traffic will be. (Maybe you do in your network). So we&#8217;re just going to give most of it to our jumbo frame traffic (50%), and then we&#8217;ll give 25% to the other two types (fcoe and default).<\/p>\n<p><code><br \/>\nn5k-top(config-pmap-nq-c)# policy-map type queuing myQueuing-policy<br \/>\nn5k-top(config-pmap-que)# class type queuing myTraffic<br \/>\nn5k-top(config-pmap-c-que)# bandwidth percent 50<br \/>\nn5k-top(config-pmap-c-que)# class type queuing class-fcoe<br \/>\nn5k-top(config-pmap-c-que)# bandwidth percent 25<br \/>\nn5k-top(config-pmap-c-que)# class type queuing class-default<br \/>\nn5k-top(config-pmap-c-que)# bandwidth percent 25<br \/>\nn5k-top(config-pmap-c-que)# sh policy-map type queuing myQueuing-policy<\/code><\/p>\n<p>Type queuing policy-maps<br \/>\n========================<\/p>\n<p>policy-map type queuing myQueuing-policy<br \/>\nclass type queuing myTraffic<br \/>\nbandwidth percent 50<br \/>\nclass type queuing class-fcoe<br \/>\nbandwidth percent 25<br \/>\nclass type queuing class-default<br \/>\nbandwidth percent 25<\/p>\n<p>Boom! Just like that, we now have our three QoS policies created. One for type <strong>qos<\/strong> called myQos-policy. One for type <strong>network-qos<\/strong> called myNetwork-QoS-policy. And finally, the one we just created for type <strong>queuing<\/strong> called myQueuing-policy.<\/p>\n<p>What&#8217;s left? Now we just need to apply this to the system:<br \/>\n<code><br \/>\nn5k-top(config-sys-qos)# service-policy type qos input myQoS-policy<br \/>\nn5k-top(config-sys-qos)# service-policy type network-qos myNetwork-QoS-policy<br \/>\nn5k-top(config-sys-qos)# service-policy type queuing input myQueuing-policy<br \/>\nn5k-top(config-sys-qos)# service-policy type queuing output myQueuing-policy<br \/>\n<\/code><\/p>\n<p>Notice for the Queuing policy we applied it twice. Once to the input and once to the output. The QoS type is only applied to the input, because that&#8217;s where traffic comes in and is marked. The network-qos effects input and output but should be the same for all, so we only configure it once.<\/p>\n<p>Well, hopefully that wasn&#8217;t too confusing. \u00a0You can obviously see the power in this. \u00a0If we had voice, video, and other types of traffic coming through here, we could add it to our policy-maps after we tag it. \u00a0With QoS, the settings should be the same on all switches in the datacenter. \u00a0If you are running a VPC, you&#8217;ll have to make sure that the other switch has the same set up, otherwise you&#8217;ll get a Type two error:<\/p>\n<p><code><br \/>\nn5k-top(config-sys-qos)# sh vpc<br \/>\nLegend:<br \/>\n(*) - local vPC is down, forwarding via vPC peer-link<\/code><\/p>\n<p>vPC domain id : 1<br \/>\nPeer status : peer adjacency formed ok<br \/>\nvPC keep-alive status : peer is alive<br \/>\nConfiguration consistency status : success<br \/>\nPer-vlan consistency status : success<br \/>\nType-2 consistency status : failed<br \/>\nType-2 inconsistency reason : QoSMgr Network QoS configuration incompatible<br \/>\nvPC role : primary<br \/>\nNumber of vPCs configured : 3<br \/>\nPeer Gateway : Disabled<br \/>\nDual-active excluded VLANs : &#8211;<br \/>\nGraceful Consistency Check : Enabled<br \/>\nAuto-recovery status : Disabled<\/p>\n<p>vPC Peer-link status<br \/>\n&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;<br \/>\nid Port Status Active vlans<br \/>\n&#8212; &#8212;- &#8212;&#8212; &#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8211;<br \/>\n1 Po1 up 1,500<\/p>\n<p>vPC status<br \/>\n&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;-<br \/>\nid Port Status Consistency Reason Active vlans<br \/>\n&#8212;&#8212; &#8212;&#8212;&#8212;&#8211; &#8212;&#8212; &#8212;&#8212;&#8212;&#8211; &#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8212;&#8211; &#8212;&#8212;&#8212;&#8211;<br \/>\n10 Po10 up success success 1,100,500<br \/>\n20 Po20 up success success 1,100,500<br \/>\n80 Po80 up success success 1,100,500<\/p>\n<p>See? Nobody wants a type 2 error. Apply settings throughout the data center!<\/p>\n<p>Hope this helps someone struggling with getting jumbo frames on a Nexus 5k.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>&nbsp; I&#8217;ve had the fortunate opportunity to have two Nexus 5548UPs in my lab to help test upgrade problems for one of my customers. \u00a0Its been great to have some gear to play with and really try to understand how it all works together. One of the issues I&#8217;ve run up against in the past&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[990,148],"tags":[224,225],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/posts\/831"}],"collection":[{"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/benincosa.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=831"}],"version-history":[{"count":6,"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/posts\/831\/revisions"}],"predecessor-version":[{"id":2632,"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/posts\/831\/revisions\/2632"}],"wp:attachment":[{"href":"https:\/\/benincosa.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=831"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/benincosa.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=831"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/benincosa.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=831"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}