{"id":821,"date":"2014-01-14T12:37:55","date_gmt":"2014-01-14T18:37:55","guid":{"rendered":"http:\/\/benincosa.com\/blog\/?p=821"},"modified":"2014-11-19T11:23:55","modified_gmt":"2014-11-19T17:23:55","slug":"fcoe-with-ucs-c-series","status":"publish","type":"post","link":"https:\/\/benincosa.com\/?p=821","title":{"rendered":"FCoE with UCS C-Series"},"content":{"rendered":"

I have in my lab a C210 that I want to turn into an FCoE target storage. \u00a0I’ll write more on that in another post. \u00a0The first challenge was to get it up with FCoE. \u00a0Its attached to a pair of Nexus 5548s. \u00a0I installed RedHat Linux 6.5 on the C210 and booted up. \u00a0The big issue I had was that even though RedHat Linux 6.5 comes with the fnic and enic drivers, the FCoE never happened. \u00a0It wasn’t until I installed the updated drivers from Cisco that I finally saw a flogi. \u00a0But there were other tricks that you had to do to make the C210 actually work with FCoE.<\/p>\n

C210 CIMC<\/strong><\/p>\n

The first part to start is looking in the CIMC (with the machine powered on) and configure the vHBAs. From the GUI go to:<\/p>\n

Server -> Inventory<\/p>\n

Then on the work pane, the ‘Network Adapters’ tab, then down below select vHBAs. \u00a0Here you will see two vHBAs by default. \u00a0From here you have to set the VLAN that the vHBA will go over. \u00a0Clicking the ‘Properties’ on the interface you have to select the VLAN. \u00a0I set the MAC address to ‘AUTO’ based on a TAC case I looked at, but this never persisted. \u00a0From there I entered the VLAN. \u00a0VLAN 10 for the first interface and VLAN 20 for the second interface. \u00a0This VLAN 10 matches the FCoE VLAN and VSAN that I created on the Nexus 5548. \u00a0On the other Nexus I creed VLAN 20 to match FCoE VLAN 20 and VSAN 20.<\/p>\n

\"\"<\/a><\/p>\n

This then seemed to require a reboot of the Linux Server for the VLANs to take effect. \u00a0In hindsight this is something I probably should have done first.<\/p>\n

RedHat Linux 6.5<\/strong><\/p>\n

This needs to have the Cisco drivers for the fnic. \u00a0You might want to install the enic drivers as well. \u00a0I got these from cisco.com. \u00a0I used the B series drivers and it was a 1.2GB file that I had to download all to get a 656KB driver package. \u00a0I installed the kmod-fnic-1.6.0.6-1 RPM. \u00a0I had a customer who had updated to a later kernel and he had to install the kernel-devel rpm and recompile the driver. \u00a0After it came up, it worked for him.<\/p>\n

With the C210 I wanted to bond the 10Gb NICs into a vPC. \u00a0So I did an LACP bond with Linux. \u00a0This was done as follows:<\/p>\n

Created file: \/etc\/modprobe.d\/bond.conf<\/p>\n

alias bond0 bonding
\noptions bonding mode=4 miimon=100 lacp_rate=1<\/p><\/blockquote>\n

Created file: \/etc\/sysconfig\/network-scripts\/ifcfg-bond0<\/p>\n

DEVICE=bond0
\nIPADDR=172.20.1.1
\nONBOOT=yes
\nNETMASK=255.255.0.0
\nSTARTMODE=onboot
\nMTU=9000<\/p><\/blockquote>\n

Edited the \/etc\/sysconfig\/network-scripts\/ifcfg-eth2<\/p>\n

DEVICE=eth2
\nMASTER=bond0
\nSLAVE=yes
\nHWADDR=58:8D:09:0F:14:BE
\nTYPE=Ethernet
\nUUID=8bde8c1f-926f-4960-87ff-c0973f5ef921
\nONBOOT=yes
\nNM_CONTROLLED=no
\nBOOTPROTO=none<\/p><\/blockquote>\n

Edited the \/etc\/sysconfig\/network-scripts\/ifcfg-eth3<\/p>\n

DEVICE=eth3
\nMASTER=bond0
\nSLAVE=yes
\nHWADDR=58:8D:09:0F:14:BF
\nTYPE=Ethernet
\nUUID=6e2e7493-c1a1-4164-9215-04f0584b338c
\nONBOOT=yes
\nNM_CONTROLLED=no
\nBOOTPROTO=none<\/p><\/blockquote>\n

Next restart the network and you should have a bond. You may need to restart this after you configure the Nexus 5548 side.<\/p>\n

service network restart<\/p><\/blockquote>\n

Nexus 5548 Top<\/strong>
\nLog in and create VPCs and stuff. \u00a0Also don’t forget to do the MTU 9000 system class. \u00a0I use this for jumbo frames in the data center.<\/p>\n

policy-map type network-qos jumbo
\nclass type network-qos class-default
\nmtu 9216
\nmulticast-optimize
\nsystem qos
\nservice-policy type network-qos jumbo<\/p><\/blockquote>\n

One thing that drives me crazy is that you can’t do sh int po 4 to see that the MTU is 9000. From the documents<\/a>, you have to do<\/p>\n

sh queuing int po 4<\/p><\/blockquote>\n

to see that your jumbo frames are enabled.<\/p>\n

The C210 is attached to ethernet port 1 on each of the switches. \u00a0Here’s the Ethernet configuration:<\/p>\n

The ethernet:<\/p>\n

interface Ethernet1\/1
\nswitchport mode trunk
\nswitchport trunk allowed vlan 1,10
\nspanning-tree port type edge trunk
\nchannel-group 4<\/p><\/blockquote>\n

The port channel:<\/p>\n

interface port-channel4
\nswitchport mode trunk
\nswitchport trunk allowed vlan 1,10
\nspeed 10000
\nvpc 4<\/p><\/blockquote>\n

As you can see VLAN 10 is the VSAN. We need to create the VSAN info for that.<\/p>\n

feature fcoe
\nvsan database
\nvsan 10
\nvlan 10
\nfcoe vsan 10<\/p><\/blockquote>\n

Finally, we need to create the vfc for the interface:<\/p>\n

interface vfc1
\nbind interface Ethernet1\/1
\nswitchport description Connection to NFS server FCoE
\nno shutdown
\nvsan database
\nvsan 10 interface vfc1<\/p><\/blockquote>\n

Nexus 5548 Bottom<\/strong>
\nThe other Nexus is similar configuration. \u00a0The difference is that instead of VSAN 10, VLAN 10, we use VSAN20, VLAN 20 and bind the FCoE to VSAN 20. \u00a0In the SAN world, we don’t cross the streams. \u00a0You’ll see that the VLANS are not the same in the two switches.<\/p>\n

Notice that in the below configuration, VLAN 20 nor 10 is defined for through the peer link so you’ll only see VLAN 1 enabled on the vPC:<\/p>\n

N5k-bottom# sh vpc consistency-parameters interface po 4<\/p>\n

Legend:
\nType 1 : vPC will be suspended in case of mismatch<\/p>\n

Name Type Local Value Peer Value
\n————- —- ———————- ———————–
\nShut Lan 1 No No
\nSTP Port Type 1 Default Default
\nSTP Port Guard 1 None None
\nSTP MST Simulate PVST 1 Default Default
\nmode 1 on on
\nSpeed 1 10 Gb\/s 10 Gb\/s
\nDuplex 1 full full
\nPort Mode 1 trunk trunk
\nNative Vlan 1 1 1
\nMTU 1 1500 1500
\nAdmin port mode 1
\nlag-id 1
\nvPC card type 1 Empty Empty
\nAllowed VLANs – 1 1
\nLocal suspended VLANs – – –<\/p><\/blockquote>\n

But on the individual nodes you’ll see that the VLAN is enabled in the VPC. VLAN 10 is carrying storage traffic.<\/p>\n

# sh vpc 4<\/p>\n

vPC status
\n—————————————————————————-
\nid Port Status Consistency Reason Active vlans
\n—— ———– —— ———– ————————– ———–
\n4 Po4 up success success 1,10<\/p><\/blockquote>\n

Success?<\/strong><\/p>\n

How do you know you succeeded?<\/p>\n

N5k-bottom# sh flogi database
\n——————————————————————————–
\nINTERFACE VSAN FCID PORT NAME NODE NAME
\n——————————————————————————–
\nvfc1 10 0x2d0000 20:00:58:8d:09:0f:14:c1 10:00:58:8d:09:0f:14:c1<\/p>\n

Total number of flogi = 1.<\/p><\/blockquote>\n

You’ll see the login. If not, then try restarting the interface on the Linux side. You should see a different WWPN in each Nexus. Another issue you might have is that the VLANS may be mismatched, so make sure you have the right node on the right server.<\/p>\n

Let me know how it worked for you!<\/p>\n","protected":false},"excerpt":{"rendered":"

I have in my lab a C210 that I want to turn into an FCoE target storage. \u00a0I’ll write more on that in another post. \u00a0The first challenge was to get it up with FCoE. \u00a0Its attached to a pair of Nexus 5548s. \u00a0I installed RedHat Linux 6.5 on the C210 and booted up. \u00a0The…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[990,920,148,151,992],"tags":[219,1004,921,217,218],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/posts\/821"}],"collection":[{"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/benincosa.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=821"}],"version-history":[{"count":2,"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/posts\/821\/revisions"}],"predecessor-version":[{"id":2751,"href":"https:\/\/benincosa.com\/index.php?rest_route=\/wp\/v2\/posts\/821\/revisions\/2751"}],"wp:attachment":[{"href":"https:\/\/benincosa.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=821"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/benincosa.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=821"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/benincosa.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=821"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}