I have in my lab a C210 that I want to turn into an FCoE target storage. I’ll write more on that in another post. The first challenge was to get it up with FCoE. Its attached to a pair of Nexus 5548s. I installed RedHat Linux 6.5 on the C210 and booted up. The big issue I had was that even though RedHat Linux 6.5 comes with the fnic and enic drivers, the FCoE never happened. It wasn’t until I installed the updated drivers from Cisco that I finally saw a flogi. But there were other tricks that you had to do to make the C210 actually work with FCoE.
C210 CIMC
The first part to start is looking in the CIMC (with the machine powered on) and configure the vHBAs. From the GUI go to:
Server -> Inventory
Then on the work pane, the ‘Network Adapters’ tab, then down below select vHBAs. Here you will see two vHBAs by default. From here you have to set the VLAN that the vHBA will go over. Clicking the ‘Properties’ on the interface you have to select the VLAN. I set the MAC address to ‘AUTO’ based on a TAC case I looked at, but this never persisted. From there I entered the VLAN. VLAN 10 for the first interface and VLAN 20 for the second interface. This VLAN 10 matches the FCoE VLAN and VSAN that I created on the Nexus 5548. On the other Nexus I creed VLAN 20 to match FCoE VLAN 20 and VSAN 20.
This then seemed to require a reboot of the Linux Server for the VLANs to take effect. In hindsight this is something I probably should have done first.
RedHat Linux 6.5
This needs to have the Cisco drivers for the fnic. You might want to install the enic drivers as well. I got these from cisco.com. I used the B series drivers and it was a 1.2GB file that I had to download all to get a 656KB driver package. I installed the kmod-fnic-1.6.0.6-1 RPM. I had a customer who had updated to a later kernel and he had to install the kernel-devel rpm and recompile the driver. After it came up, it worked for him.
With the C210 I wanted to bond the 10Gb NICs into a vPC. So I did an LACP bond with Linux. This was done as follows:
Created file: /etc/modprobe.d/bond.conf
alias bond0 bonding
options bonding mode=4 miimon=100 lacp_rate=1
Created file: /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
IPADDR=172.20.1.1
ONBOOT=yes
NETMASK=255.255.0.0
STARTMODE=onboot
MTU=9000
Edited the /etc/sysconfig/network-scripts/ifcfg-eth2
DEVICE=eth2
MASTER=bond0
SLAVE=yes
HWADDR=58:8D:09:0F:14:BE
TYPE=Ethernet
UUID=8bde8c1f-926f-4960-87ff-c0973f5ef921
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
Edited the /etc/sysconfig/network-scripts/ifcfg-eth3
DEVICE=eth3
MASTER=bond0
SLAVE=yes
HWADDR=58:8D:09:0F:14:BF
TYPE=Ethernet
UUID=6e2e7493-c1a1-4164-9215-04f0584b338c
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
Next restart the network and you should have a bond. You may need to restart this after you configure the Nexus 5548 side.
service network restart
Nexus 5548 Top
Log in and create VPCs and stuff. Also don’t forget to do the MTU 9000 system class. I use this for jumbo frames in the data center.
policy-map type network-qos jumbo
class type network-qos class-default
mtu 9216
multicast-optimize
system qos
service-policy type network-qos jumbo
One thing that drives me crazy is that you can’t do sh int po 4 to see that the MTU is 9000. From the documents, you have to do
sh queuing int po 4
to see that your jumbo frames are enabled.
The C210 is attached to ethernet port 1 on each of the switches. Here’s the Ethernet configuration:
The ethernet:
interface Ethernet1/1
switchport mode trunk
switchport trunk allowed vlan 1,10
spanning-tree port type edge trunk
channel-group 4
The port channel:
interface port-channel4
switchport mode trunk
switchport trunk allowed vlan 1,10
speed 10000
vpc 4
As you can see VLAN 10 is the VSAN. We need to create the VSAN info for that.
feature fcoe
vsan database
vsan 10
vlan 10
fcoe vsan 10
Finally, we need to create the vfc for the interface:
interface vfc1
bind interface Ethernet1/1
switchport description Connection to NFS server FCoE
no shutdown
vsan database
vsan 10 interface vfc1
Nexus 5548 Bottom
The other Nexus is similar configuration. The difference is that instead of VSAN 10, VLAN 10, we use VSAN20, VLAN 20 and bind the FCoE to VSAN 20. In the SAN world, we don’t cross the streams. You’ll see that the VLANS are not the same in the two switches.
Notice that in the below configuration, VLAN 20 nor 10 is defined for through the peer link so you’ll only see VLAN 1 enabled on the vPC:
N5k-bottom# sh vpc consistency-parameters interface po 4
Legend:
Type 1 : vPC will be suspended in case of mismatchName Type Local Value Peer Value
————- —- ———————- ———————–
Shut Lan 1 No No
STP Port Type 1 Default Default
STP Port Guard 1 None None
STP MST Simulate PVST 1 Default Default
mode 1 on on
Speed 1 10 Gb/s 10 Gb/s
Duplex 1 full full
Port Mode 1 trunk trunk
Native Vlan 1 1 1
MTU 1 1500 1500
Admin port mode 1
lag-id 1
vPC card type 1 Empty Empty
Allowed VLANs – 1 1
Local suspended VLANs – – –
But on the individual nodes you’ll see that the VLAN is enabled in the VPC. VLAN 10 is carrying storage traffic.
# sh vpc 4
vPC status
—————————————————————————-
id Port Status Consistency Reason Active vlans
—— ———– —— ———– ————————– ———–
4 Po4 up success success 1,10
Success?
How do you know you succeeded?
N5k-bottom# sh flogi database
——————————————————————————–
INTERFACE VSAN FCID PORT NAME NODE NAME
——————————————————————————–
vfc1 10 0x2d0000 20:00:58:8d:09:0f:14:c1 10:00:58:8d:09:0f:14:c1Total number of flogi = 1.
You’ll see the login. If not, then try restarting the interface on the Linux side. You should see a different WWPN in each Nexus. Another issue you might have is that the VLANS may be mismatched, so make sure you have the right node on the right server.
Let me know how it worked for you!
Comments are closed.