Category Archives: Storage

FCoE with UCS C-Series

I have in my lab a C210 that I want to turn into an FCoE target storage.  I’ll write more on that in another post.  The first challenge was to get it up with FCoE.  Its attached to a pair of Nexus 5548s.  I installed RedHat Linux 6.5 on the C210 and booted up.  The big issue I had was that even though RedHat Linux 6.5 comes with the fnic and enic drivers, the FCoE never happened.  It wasn’t until I installed the updated drivers from Cisco that I finally saw a flogi.  But there were other tricks that you had to do to make the C210 actually work with FCoE.


The first part to start is looking in the CIMC (with the machine powered on) and configure the vHBAs. From the GUI go to:

Server -> Inventory

Then on the work pane, the ‘Network Adapters’ tab, then down below select vHBAs.  Here you will see two vHBAs by default.  From here you have to set the VLAN that the vHBA will go over.  Clicking the ‘Properties’ on the interface you have to select the VLAN.  I set the MAC address to ‘AUTO’ based on a TAC case I looked at, but this never persisted.  From there I entered the VLAN.  VLAN 10 for the first interface and VLAN 20 for the second interface.  This VLAN 10 matches the FCoE VLAN and VSAN that I created on the Nexus 5548.  On the other Nexus I creed VLAN 20 to match FCoE VLAN 20 and VSAN 20.

This then seemed to require a reboot of the Linux Server for the VLANs to take effect.  In hindsight this is something I probably should have done first.

RedHat Linux 6.5

This needs to have the Cisco drivers for the fnic.  You might want to install the enic drivers as well.  I got these from  I used the B series drivers and it was a 1.2GB file that I had to download all to get a 656KB driver package.  I installed the kmod-fnic- RPM.  I had a customer who had updated to a later kernel and he had to install the kernel-devel rpm and recompile the driver.  After it came up, it worked for him.

With the C210 I wanted to bond the 10Gb NICs into a vPC.  So I did an LACP bond with Linux.  This was done as follows:

Created file: /etc/modprobe.d/bond.conf

alias bond0 bonding
options bonding mode=4 miimon=100 lacp_rate=1

Created file: /etc/sysconfig/network-scripts/ifcfg-bond0


Edited the /etc/sysconfig/network-scripts/ifcfg-eth2


Edited the /etc/sysconfig/network-scripts/ifcfg-eth3


Next restart the network and you should have a bond. You may need to restart this after you configure the Nexus 5548 side.

service network restart

Nexus 5548 Top
Log in and create VPCs and stuff.  Also don’t forget to do the MTU 9000 system class.  I use this for jumbo frames in the data center.

policy-map type network-qos jumbo
class type network-qos class-default
mtu 9216
system qos
service-policy type network-qos jumbo

One thing that drives me crazy is that you can’t do sh int po 4 to see that the MTU is 9000. From the documents, you have to do

sh queuing int po 4

to see that your jumbo frames are enabled.

The C210 is attached to ethernet port 1 on each of the switches.  Here’s the Ethernet configuration:

The ethernet:

interface Ethernet1/1
switchport mode trunk
switchport trunk allowed vlan 1,10
spanning-tree port type edge trunk
channel-group 4

The port channel:

interface port-channel4
switchport mode trunk
switchport trunk allowed vlan 1,10
speed 10000
vpc 4

As you can see VLAN 10 is the VSAN. We need to create the VSAN info for that.

feature fcoe
vsan database
vsan 10
vlan 10
fcoe vsan 10

Finally, we need to create the vfc for the interface:

interface vfc1
bind interface Ethernet1/1
switchport description Connection to NFS server FCoE
no shutdown
vsan database
vsan 10 interface vfc1

Nexus 5548 Bottom
The other Nexus is similar configuration.  The difference is that instead of VSAN 10, VLAN 10, we use VSAN20, VLAN 20 and bind the FCoE to VSAN 20.  In the SAN world, we don’t cross the streams.  You’ll see that the VLANS are not the same in the two switches.

Notice that in the below configuration, VLAN 20 nor 10 is defined for through the peer link so you’ll only see VLAN 1 enabled on the vPC:

N5k-bottom# sh vpc consistency-parameters interface po 4

Type 1 : vPC will be suspended in case of mismatch

Name Type Local Value Peer Value
————- —- ———————- ———————–
Shut Lan 1 No No
STP Port Type 1 Default Default
STP Port Guard 1 None None
STP MST Simulate PVST 1 Default Default
mode 1 on on
Speed 1 10 Gb/s 10 Gb/s
Duplex 1 full full
Port Mode 1 trunk trunk
Native Vlan 1 1 1
MTU 1 1500 1500
Admin port mode 1
lag-id 1
vPC card type 1 Empty Empty
Allowed VLANs – 1 1
Local suspended VLANs – – -

But on the individual nodes you’ll see that the VLAN is enabled in the VPC. VLAN 10 is carrying storage traffic.

# sh vpc 4

vPC status
id Port Status Consistency Reason Active vlans
—— ———– —— ———– ————————– ———–
4 Po4 up success success 1,10


How do you know you succeeded?

N5k-bottom# sh flogi database
vfc1 10 0x2d0000 20:00:58:8d:09:0f:14:c1 10:00:58:8d:09:0f:14:c1

Total number of flogi = 1.

You’ll see the login. If not, then try restarting the interface on the Linux side. You should see a different WWPN in each Nexus. Another issue you might have is that the VLANS may be mismatched, so make sure you have the right node on the right server.

Let me know how it worked for you!

Storage: The new sexy

I was fortunate enough to attend VMworld 2012 in San Francisco this year!  It is indeed a great privilege and I can’t thank Cisco enough for sending me.  There were lots of announcements that were pretty cool from a UCS standpoint like UCS Central and the vCenter plugin. (These were demoed in the Cisco booth) There were cool announcements from VMware about Horizon and the Cloud Suite.  etc.   The sessions were great and the after hours events always entertaining (Although the food at the VMware party sucked compared to 2010).  But among the madness of the myriad of messages there was one that stood out to me more than anything else:  Flash Storage.

Remember when storage was boring?  EMC, NetApp, blah blah blah.  Well those days are gone.  Its not so simple anymore.  The shear number of storage vendors on the showroom floor was a clear illustration that storage is still an open frontier like the wild west, or even like a new season of American Idol… (OK, maybe that last one was a bad analogy)

Sure EMC still leads in market share and NetApp is the fastest growing, but there is plenty of room for disruption.  There were several really good sessions on best practices for storage.  One of my favorite quotes came from Duncan Epping (of YellowBricks and Clustering Deep dive author fame):

We always blame the network for our problems but its usually the storage that is at fault

Indeed we see this in practice quite a lot.  People can’t buy more UCS because they’re constrained by their storage.  The network seems easy enough to blame but as server administrators are getting more comfortable with networking (mostly because they have to since the network access layer is now inside the server at the vSwitch) they’re starting to get that right much more often than the storage.

One of my favorite sessions was a Gartner Storage Best Practices session.  They had the following great messages:

“IOPS resolution is a multi dimensional problem that may not best addressed in the storage array”

This is why putting Fusion-io cards in a server may help with a tiered approach.  (BTW, these were announced to be inside UCS Blades now and should be available before the end of 2012).  It also explains why companies like Atlantis Computing with their ILIO product can offer big performance gains by offloading some of the work the storage array has to do as well as saving space.

This brings up another point from the Gartner session when talking about VDI (aka SHVD: server hosted virtual desktop)

“Storage costs present the #1 barrier to entry”

If you’re wondering when the year of the virtual desktop will come its when the organization has to buy new storage.  Over 40% of data center budget for new equipment goes into storage.  It pretty much pushes server and network equipment to the fringes.  That’s why I don’t ever think there will be a ‘year’ of the virtual desktop.  Instead, we’re in the beginning of the ‘decade of the virtual desktop’

The Tiered approach works as follows:  You have fast disks ( SSDs), slower disks (FC, SAS) and then the slowest disks:  SATA spinning at 7200 RPMS.  Ideally you want the data you use the most (the master copy of a windows VM) sitting on the fast storage, swap files on the mid tier, and lesser used workloads sitting down at the bottom.

The issue with the tiered approach (which NetApp doesn’t really have except for maybe the flash cache which is read only) is that you have to put the workloads where you think they will be.  And there’s a good chance you’ll get it wrong.

And that’s one reason there’s such a huge market out there for storage products that solve the issues of how to store and manage virtual machine files.  It used to be the OS sat on the server and the local disk was the best you could do.  Now with all of those VMs contending for IOPS, the storage is the bottleneck.  The new sexy bottleneck.  Sexy because there’s a lot of money to be made if you can convince people your solution is the best.

Using flash storage seems to be the sexy way to entice your customers.  SSD arrays were all the rage at VMworld.  Tiered solutions that allowed SSDs with FC/SAS/SATA were also quite popular.   I thought I’d go through some of the storage solutions I had a chance to visit with at VMworld.  Most of these are lesser known so it will be interesting to watch and see how the space changes in the next year:

Violin Memory

The Violin 6000 Series flash memory array can do 1 million IOPS with 4Gbps of bandwidth in a 3RU space.  The secret sauce is that they build their own flash memory controllers instead of using the standard SSDs that most flash array vendors use.  This probably isn’t the cheapest but its hard to beat in terms of speed.  This is the storage you buy when money is no object and you just want fast.  Just imagining this connected to some Nexus 5000s and a UCS full of B200 M3s makes me all tingling inside.


I didn’t see Whiptail on the floor but I have helped 2 UCS customers configure this and get it running.  Its cheaper and wicked fast, but offers little intelligence as to what is happening.  Just cram a bunch of SSD drives into the array and lets you go to town.  For some people, speed is all you need and they don’t care about fancy dashboards.


I had a great conversation with a great guy at Tintri at the vFlipCup event on Monday night… Wish I could remember his name.  (BTW, I represented Team JBOD with @CiscoServerGeek and we apparently wrote checks that our team could not cache).   …But I digress… Tintri seemed to be a mix of Atlantis Computing and Whiptail.  Instead of having a VM do the block caching like ILIO does, you instead have that intelligence take place on the controller.  Couple that with an array of SSDs and life looks pretty good.  This seems better than Atlantis in that your chance of VM going down is greater than your chance of the storage array going down.

Pure Storage

Two of the bigger booths of storage companies that I hadn’t heard of were Pure Storage and Tegile.

Pure Storage is a flash array vendor that seemed to have the most sophisticated protocol support including iSCSI (BTW: for the record I really don’t like iSCSI.  When you have UCS, Fibre Channel is so easy and with the UCS Del Mar release later this year you have all the components you need for FCoE without buying any separate equipment)


Tegile is a hybrid array that supports tiered storage.  The value is the deep integration with SSDs.  I would look at this offering as a less confusing EMC offering.  It offers SAN and NAS capabilities as well as data deduplication.  Pretty sweet system.


It seems like all the providers have some great niches and I think most people would be happy with any of these storage solutions.  I’d hate to leave this post without tipping my hat to EMC and NetApp who I work with and do a tremendous job.  There’s a reason they both have so many customers:  They build great products.  I should also call out Hitachi Storage.  Their own team admits they suck at marketing, but in terms of performance and reliability for mission critical apps, its hard to beat their rock solid solutions.  Its truly a company built and run by engineers.  That’s one reason their customers and me like them so much.

So if this post makes you feel all warm inside, that’s because storage is the new sexy and it is good for you to look at on company time.