Lets examine the data by answering one of the most common questions we run across with UCS: \u00a0How many connections do you need from the Fabric Extenders (aka: FEX aka IO Module aka 2104\/2204\/2208) to the Fabric Interconnects. \u00a0Mostly what I see is from 2 to 4 connections per FEX to Fabric Interconnect. \u00a0But it would be great if you could determine how much bandwidth is actually being used to scientifically decide whether you need more or less cables. \u00a0And it turns out you can free of charge with UCS Manager. \u00a0Since we are trying to answer this question, we’ll be focusing on monitoring the network in UCS. \u00a0Keep in mind, however, that you can also monitor the power consumption, temperature, and error statistics of many of the other components.<\/p>\n
To answer this question it takes a little math and a little bit of poking around to figure it out. \u00a0Steve McQuerry presented at Cisco Live session ID BRKCOM-2004 in San Diego earlier this year. \u00a0My blog is based off some of his slides which you can get at http:\/\/ciscolive365.com<\/a> (free login required), but my math is daringly original, so please let me know if I’ve made errors.<\/p>\n Let’s first look and see how UCS collects data. \u00a0On UCS manager navigate to Admin, then filter by Stats Management. \u00a0From here you will see the collection policies.\u00a0 By default each collection policy has a collection interval of 1 minute and a reporting interval of 15 minutes.<\/p>\n So what does that actually mean?<\/p>\n Collection Interval<\/strong>: \u00a0How often the data will be collected. \u00a0We are encouraged to change the collection interval to 30 seconds to get more granulated data. \u00a0This means that every 30 seconds, the device will be queried by the UCSM subprocess responsible for gathering statistics from the underlying NXOS.<\/p>\n Reporting Interval: <\/strong>How often data will be stored to the UCS Manager. \u00a0While we set the collection interval to 30 seconds, the reporting interval is how often it is stored in UCS Manager. \u00a0So we might take our first interval at 9:11AM then the next would be at 9:26, and then every 15 minutes after that. \u00a0UCS can only hold up to 5 of these records. \u00a0That alone should tell you that UCS is not good for long term trend analysis. \u00a0It is recommended that another monitoring solution be used for greater detail.<\/p>\n Cisco recommends that you change the collection interval to 30<\/em><\/strong> seconds for the things you’re interested in. \u00a0The reporting interval doesn’t really matter for what we’re doing here.<\/p>\n I have a first generation IOM so the traffic is not trunked from blade to Fabric Interconnect. \u00a0It follows a defined path based on the number of uplinks. \u00a0(see this great post:\u00a0http:\/\/jeremywaldrop.wordpress.com\/2010\/06\/30\/cisco-ucs-ethernet-frame-flows\/<\/a> for information on how its connected internally)<\/p>\n I have 2 chassis, each connected with 2 ports. \u00a0Ports 1 & 2 connect to chassis 2 and Ports 3 & 4 connect to chassis 1. \u00a0(Yes, this is not good form, but hey, I inherited this lab so that’s just the way it is and I haven’t bothered to fix it). \u00a0To see how your chassis are connected to the Fabric Interconnect, click on the Equipment tab, select the Chassis, and then select Hybrid display from the work pane That should tell you how the connections are made from FEX to Fabric Interconnect.<\/p>\n Now let’s now look at one of the FEX uplinks. \u00a0Navigate to the Equipment tab, filter by Fabric Interconnects and look at the server ports that are connected to Fabric Interconnect A:<\/p>\n Select the first port and lets look at the statistics tab in the work pane:<\/p>\n To measure bandwidth, we are interested in the delta of total bytes received (Rx) and transmitted (Tx) on each of the FEX uplinks.\u00a0This particular uplink for Received and Transmitted Total Bytes shows\u00a0837,101 \u00a0and 691,921 bytes respectively.<\/p>\n We typically measure I\/O in Gbps, Mbps, or Kbps. \u00a0So we need to translate these numbers. \u00a0This is where the math comes in. \u00a0First, remember, that our collection interval is 30 seconds. \u00a0That means, that number reported is x bytes in 30 seconds<\/em>. \u00a0To get bytes per second, just divide that number by 30. \u00a0From there, do the type of multiplication you may have learned in your physics class when converting between different forms of measurements. \u00a0Here’s the formula for Gbps and Mbps:<\/p>\n Bytes to Gbps from 30 second interval collection period<\/strong><\/p>\n = x * 0.000000000248 Gbps<\/p>\n (x bytes \/ 30 seconds) * (8 bits \/ 1 byte ) * ( 1 Gb \/\u00a01,073,741,824 bits )<\/p>\n ** Note: \u00a0You could argue that there are only 1 million bits in a Gigabit, go ahead and use that if it makes you more comfortable.<\/p>\n Bytes to Mbps from 30 second interval collection period<\/strong><\/p>\n Probably easier to do this in Mbps:<\/p>\n = x * 0.000000254 Mbps<\/p>\n (x bytes \/ 30 seconds) * (8 bits \/ 1 byte) * (1 Mb \/ 1,048,576 bits)<\/p>\n Just looking at those formulas (or multipliers as they really are), there are some simple rules we can follow:<\/p>\n Rule 1: \u00a0If the delta is not a 10 digit number or greater then you are not even doing a Gigabit per second on a 10 Gigabit link.<\/strong><\/p>\n Rule 2: \u00a0If the delta is not a 7 digit number or greater then you are not even doing a Megabit per second on a 10 Gigabit link.<\/strong><\/p>\n Armed with this knowledge, we do our math:<\/p>\n Rx: 837,101 * 0.000000254 = .212 Mbps = 212 kbps<\/p>\n Tx: 691,921 * 0.000000254 = \u00a0.1757 Mbps = 175.7kbps<\/p>\n Not a lot going on in this link is there?<\/p>\n After looking at the rest of the links on the system they were all in the same 6 figure range with one exception: \u00a0One link (Fabric B, port 1) had Rx at 13,082,674 and Tx at 3,241,484 which is about 1.5 Mbps and 823 kbps<\/p>\n Now, how can I find out what server is generating all that traffic? \u00a0(Let’s just suppose that 1.5 Mbps is a lot for pedagogical purposes)<\/p>\n Since I have 2 cables per FEX I know that Fabric B uplink 1 is connected to all the B-side uplinks on odd slots. \u00a0(Remember \u00a0\u00a0this post?<\/a>)<\/p>\n All the even slots are connected to the 2nd one. \u00a0So this has to be either blade 1, 3, 5, or 7. \u00a0So what I have to do is check which Service Profiles are in those slots. \u00a0From the equipment tab I determine that I have:<\/p>\n Slot 1: ESXi-1000v-02 -> Slot 1<\/p>\n Slot 3: Empty<\/p>\n Slot 5: CIAC-ESXi4.1-02<\/p>\n Slot 7: Empty<\/p>\n I only have to check 2 servers. \u00a0On each server I have assigned a LAN connectivity connection so I know which vNIC is going out the B side. \u00a0From here its just a matter of finding the chatty one. \u00a0Here’s how I found my Most Chatty Server Port (MCSP):<\/p>\n From the Servers tab, navigate to the service profile of the machine. \u00a0I have 6 vNICs in each one: Since I’ve labeled them, its pretty obvious which ones go out the B side. \u00a0Click on each vNIC and from the work pane, select statistics. \u00a0We expand the statistics and see a familiar screen. \u00a0But this time, we look under vNIC stats:<\/p>\n <\/p>\n![\"\"](\"http:\/\/benincosa.com\/blog\/wp-content\/uploads\/2012\/09\/ucs-mon1.jpg\" "\\"ucs-mon1\\"")
<\/a><\/p>\n![\"\"](\"http:\/\/benincosa.com\/blog\/wp-content\/uploads\/2012\/09\/ucs-mon2.png\" "\\"ucs-mon2\\"")
<\/a><\/p>\nExamining FEX bandwidth<\/h2>\n
\n![\"\"](\"http:\/\/benincosa.com\/blog\/wp-content\/uploads\/2012\/09\/ucs-mon3.png\" "\\"ucs-mon3\\"")
<\/a><\/p>\n![\"\"](\"http:\/\/benincosa.com\/blog\/wp-content\/uploads\/2012\/09\/ucs-mon4.png\" "\\"ucs-mon4\\"")
<\/a><\/p>\n![\"\"](\"http:\/\/benincosa.com\/blog\/wp-content\/uploads\/2012\/09\/ucs-mon5.png\" "\\"ucs-mon5\\"")
<\/a><\/p>\nExamining Server vNIC bandwidth<\/strong><\/h2>\n
\n![\"\"](\"http:\/\/benincosa.com\/blog\/wp-content\/uploads\/2012\/09\/ucs-mon6.png\" "\\"ucs-mon6\\"")
<\/a><\/p>\n
![\"\"](\"http:\/\/benincosa.com\/blog\/wp-content\/uploads\/2012\/09\/ucs-mon7.png\" "\\"ucs-mon7\\"")
<\/a><\/p>\n![\"\"](\"http:\/\/benincosa.com\/blog\/wp-content\/uploads\/2012\/09\/ucs-mon8.png\" "\\"ucs-mon8\\"")
<\/a><\/p>\n