On reading about choosing between NSX and ACI

I consider myself  very fortunate to work in the IT industry.  Not only do I get to develop and deploy technologies that enhance the world we live in, but I also get more drama from the different companies than a soap opera.  Take for example the story of how Jayshree left Cisco to help build Arista.  There’s also the story of how VMware bought Nicira and caused disruption with the EMC Cisco partnership. None of these stories do I know the full extent of.  I’m just a spectator and focus day to day on my own activities and try to do things that matter to organizations.

But like a spectator watching the Golden Bears win or lose on any given week in college football, I’m entitled to my opinions as well.  In fact, everybody is.  I tell this to my kids all the time.  This quote from Steve Jobs nails it:

Life can be much broader once you discover one simple fact: Everything around you that you call life was made up by people that were no smarter than you and you can change it, you can influence it, you can build your own things that other people can use. “

NSX and ACI were made by very smart people.  But people that have opinions about it and have blogs like the one you’re reading now, aren’t necessarily any smarter than you.  We try to influence opinons, and some have been more successful than others.  Brad has an excellent blog and I’ve learned a lot from it.  But like a U2 album, not every one of their songs is a hit.

My latest opinion on his article about On Choosing VMware NSX or Cisco ACI is that someone is wrong on the Internet.

Duty Calls from xkcd

In a big part of the article, Brad compares a physical network switch to a TV stand and the television to what NSX does.  He then compares ACI to an adjustable TV stand, complete with remote.   He then says:

“You’ll also need to convince people that it makes more sense to buy televisions from an electronics company; and television stands should be bought from a television stand company.”

Umm.  Not quite.  This overlooks all the values ACI brings.

Let’s liken NSX to a network overlay, which is what it is.  Let’s liken the Nexus 9000 in ACI mode to a network switch that has overlay technology built in, which is what it is.  It’s real simple:  With NSX you manage 2 networks.  With ACI you manage one integrated network.

And you manage both with software.   With ACI you put each server into an endpoint group.  They are either physical or virtual.  You can still use the same VMware DVS with ACI.  It then encapsulates that VLAN or VXLAN into an endpoint group and allows those groups to talk to each other in the fabric.

Here’s another analogy.  NSX is like a cute Christmas sweater on a nice day.  Sure, you’ll get a lot of people to look at it.  You’ll get some laughs and some comments that will make you feel good.  But what’s important is the programability of the system.  And on warm days, you really don’t need or use that cute outer sweater.

the Joy of using NSX

I will concede the NSX GUI looks great!  VMware has always done a great job of making things look good and there’s a reason that VMware is the number one hypervisor in the industry.  But companies evolve.    VMware evolves into networking.  Cisco evolves into software.  So does your organization.  Your organization needs solid APIs if you want to program everything.  So if we’re doing it this way, we don’t need a sexy GUI to automate all of this.  I need those solid APIs.  Since Cisco introduced UCS its API business has been serious.  In fact, what other x86 platform has a more solid  API than UCS?   As Cisco continues to invest in software to drive its products, ACI has become that next big thing.  But it’s a whole new paradigm of network.  Gone are VLANs.  All we care about now is how applications connect.  It’s all object oriented now and it’s simple.

A Software Company versus a Hardware Company

This part is great.  Brad then puts 2 quotes from VMware employees about why they think NSX is going to win in the marketplace.  This one from the CEO of Nicira: “Who do you think is going to make better software, a software company or a hardware company?”

Is Apple a hardware company or a software company?  Is Cisco a hardware company or a software company?  You see, only a Sith deals in absolutes.  Cisco is a solutions company.

This is what John Chambers, the Cisco CEO, keeps trying to tell everyone:  It’s the solution that matters.  It’s companies that see the whole vision of the architecture and can make all those pieces work together.  That is who wins.

I don’t think Cisco has that down perfect yet.  I don’t think VMware does either.  But we are working towards it.

The Network Effect

Both Cisco and VMware keep touting how many people are using their SDN technology.  There is a sense of urgency with both companies to make everyone believe that everyone else is jumping on board.  It reminds me of when I was hosting my 20 year high school reunion this past summer.  People would ask me:  “How many people are going?”  And I’d say something like: “Oh, man we have at least 50 tickets sold and tons more who said they’ll come”.  In reality, many of those tickets were given to people on the committee and I had about 2 other people that said they would go.  You see, the network effect is huge and both companies know it.  So they have to make it sound like everyone is doing it.  Then, you are in your IT shop and you’re saying:  How come I’m not doing this?  No one likes to feel like they are missing out.

And for the record:  The 20 year reunion was amazing.  We had well over 150 people there.


Zero Trust micro-segmentation seems is a cool thing.  If you have 10 web servers in the same group then you’d like to keep those secure.  How do we do this with ACI?  We put all the servers in what we call an End Point Group (EPG) which allows ports or IP addresses or other EPGs to talk with it.  This is similar to how with AWS we create Security Groups and can assign them to instances.  Some other cloud providers like Digital Ocean and Softlayer don’t have these features so in Linux instances we use things like iptables or ufw  to secure our instances.

Since we want to secure and automate the entire environment, I’ve been playing with things like Docker and Ansible to create these secure instances and lock them down.  Open source tools to solve problems.  So while it’s a nice feature, it’s not going to apply in every case.  And how long before ACI has it?  Probably before most people adopt ACI or NSX to begin with.

VMware and OpenStack

One last comparison:  VMware is to OpenStack as Microsoft is to Linux.  I’ll just leave it at that.

The Promise Land

The promise land is open.  It’s a place where I can take my applications from my own data centers and migrate them to any cloud provider I want.   This is the vision of Cisco’s Intercloud.  Use the best of public cloud and marry it with the private cloud.  It’s fast and it’s agile and it’s programmable.

I’ll end with this:  Keep in mind that both of these technologies are still pretty fresh.  If I look at my customer set, I have quite a few Nexus 9000s but few ACI customers.  I also have lots of customers that are looking at NSX and ACI, but none of them have deployed it in test let alone production environments.  Now, my market here in the pacific northwest is a micro slice of the picture, and I’m sure Brad sees a lot more from his vantage point.  But if you haven’t jumped on any bandwagon yet (like I’d say 95% or more of IT have not), let me just say this:

You can buy Cisco Nexus 9000s.  They make a great 40Gb switch and have great features including programability RESTful APIs, and python extensions.  It outperforms its competition on Power, Performance, Programmability, and Price.  You can try running NSX over them and you can try running them in ACI mode.  The choice is yours but you lose nothing and gain so much in moving to the Nexus 9k environment.    Its not just an adjustable TV stand.  It’s the whole solution:  The remote, the TV, and the stand, and the room you watch it in.  It’s the whole experience.

You see the winner isn’t who comes up with the best software,  it’s who can produce the best experience.


I’m finally jumping in on the Docker bandwagon and it is pretty exciting.  Here’s how I did a quick trial of it to make it work.

Install OS

I could do this in my AWS account, or I can do it with my local private cloud.  So many options these days.  I installed the latest Ubuntu 14.04 Trusty server on my local cloud.  It probably would have been just as easy to spin it up on AWS.

I had to get my proxy set up correctly before I could get out to the Internet.  This was done by editing /etc/apt/apt.conf.  I just added the one line:

Configure for Docker

I followed the docker documentation on this.  Everything was pretty flawless.  I ran:

That last command had problems. The error I got said:

This is because I need to put a proxy on my docker configuration. A quick google, search lead me to do:

I added my proxy host


And I see a ton of stuff get downloaded. Looks like it works!

Now I have docker on my VM.  But what can we do with it?  Containers are used for applications. Creating a python application would probably be a good start. I searched around and found a good one on Digital Ocean’s site.

I like how it showed how to detach: CTRL-P and CTRL-Q. To reattach:

Get the image ID, then reattach

Docker is one of many interesting projects I’ve been looking at lately. For my own projects, I’ll be using Docker more for easing deployment. If you saw from my article yesterday, I’ve been working on public clouds as well as internal clouds. Connecting those clouds together and letting applications migrate between them is where I’ll be spending a few more hours on this week and next.

IP Masquerading (NAT in Red Hat)

In my lab I have a server that is dual homed.  It is connected to the outside network on one interface (br0) and the internal network (br1) is connected to the rest of my VM cluster.

I want the VMs to be able to get outside.  So the way I did that (on RedHat) was to create a few IP table rules.  I’ve been doing this for 10+ years now, but keep forgetting syntax.

So here it is:

Then, of course, you have do enable forwarding in the /etc/sysctrl.conf

Finally, run

for those changes to take effect.

AWS with python

In my previous post, I set up my Mac to do python development.  Now I’m ready to launch an EC2 image using boto.

Create Virtual Python Environment

I have already logged into AWS and created an account.  I gave it a credit card and now I get a free server for a year that I can do all sorts of fun things.

First, I’m going to create a virtual environment for this:

Connect to AWS

So now we have the base setup.  I checked the documentation for AWS here: http://boto.readthedocs.org/en/latest/ec2_tut.html

It said I needed some security credentials, but I noticed that I don’t have any.  I logged onto AWS and I created a user for my account named tester.  From there, I saw the access key ID and secret access key.  After that, I had to give him administration permissions.

I ran the command as shown in the document:

We are in!

Launch an instance

From the AWS management console when we look at the image store, I found the Ubuntu 14.04 LTS was available in the free tier. Next to the name, I found the instance id: ami-3d50120d. So that’s the one I’m going to try to launch. We also need to set the size. From the console, I found that the t2.micro instance was also free. So I gave that a whirl.  Let’s get the image going and then check how its doing.

I found that I could then see what the status was again by looking a little later:

Notice, I had to refresh the statuses variable.  The next step is to figure out how we can log into the server now that its up.   We could go to the EC2 management portal and create a keypair.  Or perhaps we could try to do this pragmatically?

Create a Boto Configuration File


To start off with, we’d like to put our environment variables in a  config file so we don’t have to enter that into the script.  To do this we create a file:

This file will be referenced by the APIs so we don’t have to provide the information next time.  The format of the file is:

That looks ok. Let’s log in again and terminate the instance:

Create Key Pair for future instances


Create the New Instance with our Keypair

That last command gives us the IP address of the server. So let’s log into him now.

Login to new instance

If we try to ping, we realize that we can’t.  The problem is, we didn’t give it a security group.  The default security group has locked it down from the outside world.  What we need to do is enable some ports on it.  Primarily, we need to enable SSH, port 22.  We’ll do this from the console at this point and leave the API calls to you gentle reader to figure out.  Once that is done, we could do a simple

ssh  -i ~/Desktop/tester-keypair

But that’s kind of a pain.  What we could do instead is create a config file so we only have to run a simpler command.  To do this, we copy the keypair file into the ~/.ssh directory.

Then we create a file called ~/.ssh/config.  We then make it look like the following:

Now we can log right in!


A few things I need to go back and check on:

  1. Creating the default Security group.  This shouldn’t be too hard.
  2. Making the pem file save.  In this article I think there is a bug using python3 and the keypair.save() command as I wasn’t able to write out the file.  I kept getting the exception: TypeError: ‘str’ does not support the buffer interface.

Mac Environment for OS X Yosemite

I updated to OS X Yosemite and as usual a lot of development things stopped working.  I figured it was time to get serious about Python as well, so here’s how I’m setting things up.


I use brew to put packages on Mac OSX.  I like it.  Its easy.  Unfortunately, with the update brew stopped working.  This is because of the new Ruby that is included with OS X.  I found the answer here:



brew packages

That command does it all for me.


This part took the longest.

This is how I access Windows Servers I have to work with.  When I updated rdesktop and I would launch the desktop, I got:

Had to follow instructions here: http://stackoverflow.com/questions/26489928/cant-load-x11-in-r-after-os-x-yosemite-upgrade

Then reboot.  That didn’t work.  Next, I updated XQuartz.  This was done by opening XQuartz and then in the preference menu, check for updates and then update to the latest.  (2.7.7 in my case).  And guess what?  It hung forever.  So I had to go to Apple’s site and download it:


Then I extracted it and it worked fine.  Whew!


All this environment stuff needs some order.  I setup my bash_profile environment similar to this post:


and this post:


The resulting is that I now have Python installed and its coming from /usr/local/bin instead of the one supplied by Apple.

pip install virtualenv

From here I set up a directory called ~/VirtualEnvs as specified by the above referenced setup document.  Now all my projects for python can be done inside of virtual python environments with the command:

virtualenv <myproject>

So we are ready to role with that.

Here is my ~/.bash_profile

and here is the .bashrc file

Hope that helps. And I’m open to nice suggestions.

Ruby on Rails Environment

I noticed my rails projects no longer worked after I opened Xcode and updated some components.  To solve this I found I needed to run:

I also ran

This command took about 45 minutes to run!  That didn’t quite do the trick.  What really did it was my rvm environment got messed up.   We don’t want brew to control ruby versions it turns out.  So I added back the rvm call in the .bash_profile (fixed above).  Then I ran rvm install 2.1 to install the 2.1 version of ruby.  I then did rvm default 2.1 and from now on, that ruby version pops up and I’m good.

Serial Console

With every update I have to update the driver for my USB to Serial cable.  I have an old one.  I found the instructions to upgrade here and it worked perfect.  The one change was the values I had to put in the plist:


Before reading that I kept reinstalling the driver. Turns out just a quick modification of the /System/Library/Extensions/ProlificUsbSerial.kext/Contents/Info.plist was all it took.

Sales Specialists: You’re your own smart guy

This is a little off topic in that it does’t have to do so much with a technology but how systems engineers and sales specialists present their technology as well as themselves in front of customers.  After writing this up, I asked @ciscoservergeek to get his thoughts and he pointed me to @jonisick ‘s article on the Art of Pre-Sales, which may be relevant to this post.   In fact, if you’re a pre-sales engineer, I recommend you read his article first and if you have time, finish this post.

For the two or three people that actually follow my blog, you know that I’m a pre-sales systems engineer for Cisco.  The fun part of my job consists of visiting customers and perspective customers and consulting with them on their data center needs.  This usually involves learning about their environment and what challenges they face.  Its super fun.

A systems engineer is valuable in that they have a deep technical expertise of the customers environment as well as a deep technical understanding of their own product or services set and how they can fit into that organizations environment.  We can also tell people what’s coming next as well as what other people in their similar situations are doing.

Typically, (at least in the two big tech vendors I’ve worked for IBM and Cisco) a systems engineer is paired with an account manager or a product sales specialist and the two of them are assigned to several customers in a vertical or a geography, or both.  This can work very well, especially when both roles are good at what they do.


I’ve been fortunate that I’ve usually always worked with some of the best talent in the industry.  Here at Cisco, I’ve been able to work with some amazing sales specialists and account managers.

So here’s the rant: When a sales person calls their engineer the “smart guy/gal”.  This happens all the time.  I see teams go in front of the customers and the sales specialist or partner sales specialist may say a few words and then introduces the engineer as “the smart guy”.   Or, “We brought the smart guys here to talk with you”

Now don’t get me wrong, I am a smart guy.  Thank you.  But so are you.  And by you saying that the engineer is the smart guy, you’re saying that you’re not as smart.  And if I’m the customer, and I hear that, why do I even want you in the room?  Why do I as the engineer even need you in the room?  If all you are going to do is give a 2 minute introduction and then have the system engineer talk the rest of the meeting while you as the sales person play with your phone, you are not adding value.   I can actually give a pretty good 2 minute introduction of myself.  In fact, since I’m an engineer, I can do it efficiently, and do it in less than 30 seconds, cause I know most people don’t care that I like to take long walks on the beach and watch gladiators wrestle.

When I’m invited to a meeting, and it is introductory, then usually a product sales person will introduce the product and do the value walk through.  For those meetings I consider it my job to be attentive and do what we in the industry call “Add Color”.  That means, as the sales person is talking and I perceive something relevant to their situation, I will speak up and mention it.  If I’m not talking, I’m not contributing.  If I’m not contributing, then why am I there?  (Hopefully, I don’t talk too much and what I say is relevant.  There’s a fine line about adding too much color)

Similarly, if an engineer is talking, a good product sales person interjects comments related to the customers organizational environment and probing questions regarding business cycles and drivers.   Or clarifying things that I should have said.   Asking questions is also very welcomed.  As an engineer, I really don’t mind being put on the spot.  Don’t be afraid that I don’t know the answer to one of the questions you might ask.  If I don’t, I’ll say so and I’ll figure out the answer.  Also, by having others on the sales team ask questions of the engineer, it prompts more discussion from the customer.  Participation is the best thing you can get in any consultative sales meeting.   Good or bad.  Cause the worst sales meetings are the ones you walk out of and you have no idea what the other party thinks about what you just heard/said.

Smart people like being with smart people.  “A” players like being with “A” players. Good sales specialists, the kind that I’ve been fortunate enough to work for, are smart.  These last 3 months have been a blast for me.  Due to some staff shortage I’ve been able to work with 3 very good product sales people in California, Oregon, and Utah who I would call my smart guy/gal counterpart on any given opportunity.

They are total pros and its a total pleasure to work with them.  My thoughts on this came from a meeting I had a few weeks ago.  The sales team (my team) went in and spoke and the conversation shifted from technology, to business processes, to procurement cycles, etc then back to technology.  I realized that this is what makes a formidable team.  I could not have answered the purchasing questions nor could I have understood the cast of characters on the customers side.  I had no relationship with any of them.  Likewise, the sales person didn’t know some of the reasons lossless Ethernet is required for FCoE.  But together, we could really help decision makers get all the information they need to chose a technology.

So next time a process question or ordering, or political issue comes up in a meeting, I’m going to say as I look toward the sales specialist: “Well, we brought the smart guy/gal here to help get those questions answered”.


Installing Cisco DCNM on Red Hat Linux

DCNM is Cisco’s GUI for managing MDS and Nexus products.  It’s pretty great for getting a visual of how things are configured and performing.

I thought I would go into a little more detail than I’ve seen posted online about installing DCNM on RedHat Linux.  In this example we’ll be installing two servers.  One server will be our app server and the other one will be our Postgres database server.  You can do it all in just one server, but where is the fun in that?

1. Download binaries

From Cisco’s homepage, click support.  In the ‘Downloads’ section start typing in ‘Data Center Network’.  (DCNM showed no results when I tried it) You’ll see the first entry is Cisco Prime DCNM as shown below.


We will be using DCNM 6.3.2 since its the latest and works great.  We need to download 2 files.



The installer is really all you need, but its kind of nice to use the silent installer to script the installation process.

2.  Initial VM installation

Using the release notes as our guide as well as other installation instructions we will be creating two VMs with the following characteristics:

Processors 2 x 2GHz cores
Memory 8GB (8096MB)
Storage 64-100GB


For this installation, we’re just doing this as a test, so you may need more space.  Also, notice that in the release notes it states that when doing LAN and SAN monitoring with DCNM you need to use an Oracle Database.  A Postgres Database is supported on just SAN for up to 2000 ports or just LAN for up to 1000 ports.

Create these VMs.  I’m using KVM but you can use vSphere or Hyper-V.

3.  Operating System Installation 

The installation guides show that RHEL 5.4/5.5/5.6/5.7/6.4 (32-bit and 64-bit) are supported.  I’m using RHEL 6.5 x86_64.  It comes by default with PostgreSQL 8.4.  So I might be living on the edge a little bit, but I had 0 problems with the OS.

I installed two machines:


During the installation, I changed 2 things, but other than setting up the network I accepted the defaults with nearly everything.

3.1 dcnm-app

I set up as a Desktop as shown below.



3.2 dcnm-db

Set up as a Database server as shown below


4. Operating System Configuration

There are several quick things to do to get this up and running.  You probably have OS hardening procedures at your organization, but this is howI did it to get up and running.   Do the following on both servers.

4.1 Disable SELinux

Does anybody besides Federal agencies use this?  Edit /etc/sysconfig/selinux.

Change the line to be:

This then requires a reboot.

4.2 Disable iptables

Yeah, I’m just closing the firewall.  There are some ports pointed out in the installation guide you can use to create custom firewalls, but I’m just leaving things wide open.

4.3 Enable YUM

If you set your server up with the RedHat network then you are ready to go.  I’m just going to keep it local bro!  I do this by mounting an expanded RedHat installation media  via NFS.  Here’s how I do it:

If you are cool then you can put it in /etc/fstab so it persists.

I then created the file /etc/yum.repos.d/local.repo.  I edited it to look like the below:

4.4 Install additional RPMs as needed

One that you will need on dcnm-app is glibc.i686

5. Database Installation on dcnm-db

This step is only needed on dcnm-db.  Using the info from the database installation guide we are using Postgres.  If you followed above like I did then you should just be able to see all the postgres RPMs installed.

If not, then you can install them all with

Next, start up the data base:

With the default installation of Postgres on RedHat, a user named postgres is created who pretty much does everything. We use him to configure the database.

5.1 Postgres Config

Postgres on RHEL6.5 doesn’t accept network connections by default.  That makes it more secure.  To enable our App server to connect to it, we need to change two files.


Modify this file by adding the IP address for it to listen on.  By default its set to only listen for connections on ‘localhost’.
Change this line:

To look like this:

Or you can just make it ‘*’ (that says: listen on every interface). In my case this works because my Database servers IP address is, so I’m listening on eth0 and the local interface.


Modify this file by adding in a line for our DCNM user.  At the bottom of the file I added this line:

Once those two files are changed, restart postgres.

Now you should be ready to rock the Database server. We’ll check it in a minute. Now lets go over to the app server.

6.  Configure the App Server

You need to login via either VNC or on the console for XWindows.  VNC is probably the easiest way to see it remote.

Start the VNC server and then you can VNC into it.

You’ll then need to copy the dcmn installer that you downloaded from Cisco in step 1 as well as the properties file that you downloaded.  I put mine in the /tmp directory.  Change this to be an executable by running:

6.1 Modify the installer.properties

The dcnm-silent-installer-properties file is a zip file.  When expanded it has a directory called Postgres+Linux.  In this directory is the file we will use for our installation.  For the most part, I left it alone.  I just changed a few of the entries:


With that, we are ready to run!

7. Install DCNM

On the App server, we finally run:

If all goes well, you should be able to open a browser to dcnm-app and see the Cisco login screen.


The CCIE Data Center Certification Process



On July 9th, 2014 I passed the CCIE Data Center lab exam in San Jose earning me the CCIE certification.  Hurray!  When my team heard that I had done it, their response was:  If Vallard can do it, so can I!  Ha ha.  So needless to say a few more people have started down the path to certification, of which I have no doubt they will surely reach.

I have to say it feels pretty great and the process I went through to get it was very rewarding in that it deepened my understanding of data center architectures as well as the solid hands on skills required to implement these solutions.  With the CCIE certification, its the journey that makes it so worth it.

I thought I would write a bit of my experience of the process and how I approached it.  To summarize, it took me 5 times until I passed the written exam and once I did that I passed the lab exam on my second try.  I’m not saying my approach is the best, but it worked for me and I’m happy with the outcome.  The funny thing is, even though I worked really hard and learned so much to get it, I still feel like there are many things I don’t know about the platforms.  One of the drawbacks of my position is I don’t do a lot of troubleshooting with my customers because most of the solutions Cisco offers work really well.  Take UCS for example:  I spend probably 2 hours a month at the most troubleshooting issues with it – And that’s with the hundreds of UCS systems that my customers have that I support!

In spite of that, I still know this stuff very well now.  When a coworker asked me how to configure VPCs on the Nexus 5548s – to just give him a quick and dirty config – I was able to spit it all out from memory and I knew it was right.  I’ve done it so many times now I can do it in my sleep.

Need an OTV config?  I got you covered there too.  I can do OTV on a stick light speed setting it up with multicast or adjacency servers, I don’t even care.  I can do it all.  Boom.  So yes, passing the CCIE exam gives you confidence because you learn a ton.  That’s kind of how I felt when I graduated with my computer science degree from Berkeley.  Even though the program kicked my trash and made me feel like a sorry sucker most the time, it made me believe that armed with the skills I could do anything… given enough time.

So here’s my experience:

The Written Exam

The CCIE Data Center written exam topics are spelled out pretty clear on the Cisco Learning Network page.    I first took the test, in its beta form and I knew very little about the Nexus product line other than a few switches I had set up before.  I took the test without studying.  Zero prep.  Didn’t even look to see what was on it.  You see, I had to have humility beaten into me.  Anyway, I failed miserably.  Seriously.  I thought I was the man at UCS.  I got less than 20% right on it.  I blamed it on the way  the questions were worded, but in hind site, there were very clear answers that stood out among the wrong ones.  The thing was, it was hard.

After my first failure in August 2012, I gave up for about a year, not thinking it was for me.  Then I learned that a few more friends had already passed the written and were working towards the lab.  My pride made me think the same thing my team mates thought when I passed:  “If they can do it, then I can do it.”  My method, I thought would be a brute force attack on the exam.   So I took the exam again a year later in July 2013 after really working specifically on Nexus and MDS.  I felt that if given the beta exam again I could pass it.  The problem was, the exam was much different than I remembered it and again I did poorly.  When I failed, I rescheduled after realizing a few of my mistakes.  I took it again in August and September each time doing a little better, but each time not quite getting it.  By the time my December test came I was solidly prepared and just before Christmas on the 23rd of December I passed the written.

So what’s my advice on the written:

1.  If you already work in this field and have hands on with Nexus, MDS, & UCS, take the exam to see what’s on it.  CCIE is a total investment and if it takes you a few hundred dollars to pass the written exam, it might be worth it.

2.  If you fail the first time, take it again as soon as you can.  I think there are new rules going into affect that make it so you might not be able to take it as often.  However, once you start down the road to CCIE certification, you can’t stop until you’ve reached the end.  Otherwise you lose it.  That year I spent off was a waste.  I should have kept going.

3.  Once you pass the written exam, schedule the lab exam as soon as possible.  There are several months of waiting time right now and you don’t want this train to stop, so keep working towards it.

The CCIE Data Center Lab Exam

My entire IT career has been spent doing very hands on things.  I’m fortunate in that when I learn how to do something via the command line, my fingers seem to remember how to do it pretty well.  In some ways that’s bad because I have a hard time explaining things (which means maybe I don’t know how it works in the first place?) But I can usually always get things to work.  Being a fast typer helps as well.

As soon as I passed my written exam, I scheduled the lab.  The soonest I could get in was April 15, 2014.  That’s right: 4 months out. I had very little to go by other than the blueprint and Bryan McGahan’s excellent writeup.  I flew in the night before, and went to bed around 10PM, but then at 3AM had trouble going back to sleep.  I tossed and turned until about 5AM and then finally just got up, went for a 3 mile run, ate a good breakfast and showed up at Building C in San Jose 30 minutes early.  I sat in the waiting room with 14 other nervous people.  Man, I was tense.  I hadn’t felt that way since finals in undergrad.  We finally went in and I went to work.

As I was taking the exam, I tried to get that zen experience that Brian talked about in his blog, but it didn’t happen for me at all.  In fact, hardly anything happened for me.  For some reason, though, I thought I had done pretty well.  Wrong.  0% in multiple categories.

But I didn’t go into this thing the first time blindly.  I How did I prepare?  Hands on baby.  Stick time.  Yeah!

I was fortunate enough to have a pretty decent lab.  My equipment was good, but not complete.  I had:

– UCS with the old 6120s (but I’ve worked on plenty of 6248s so I wasn’t worried if that’s what they would have in the lab since I know all about unified ports.).  But 6120 fabric interconnects was all that was available to me.

– One Nexus 7010.  I had 1 Sup1 but I upgraded it to 8GB of RAM so that I could do 4+1 VDCs.  Didn’t matter, 4 would have been fine, since what I really needed was 8 VDCs.  But I made due.  I had one M1 line card and one F1 line card so that I could practice OTV, LISP, FabricPath, FCoE and layer 3 stuff.

– One Nexus 5548.  No line modules but I was fortunate enough to have layer 3 capabilities.  This helped me when I practiced OTV.  I also had several Nexus 2148s hanging around so I could do FEX things, but I could only do so much with a single Nexus 5548.

– One MDS 9148 Fibre Channel switch.  He worked pretty well.

I had a great base to get going on but in the end, I just couldn’t put it all together.  Why did I fail the first time?  Two reasons I think:

1.  Lack of confidence.  This is a big deal.  Nobody expected me to pass.  I’ve only been at Cisco for 3 years and I know people  who have been here a long time and haven’t earned the CCIE certification.  The second time I went in, I told my manager that I was getting it.  I was solidly prepared.

2.  Lack of equipment.  This was the biggest reason in my mind.  I’m cocky (conceited? immature? ignorant?) enough to think I can do these things.  I have young 4 children, and I’ve watched them all alone for 4 days straight, so I’ve already faced huge challenges!  I can do this!  If you look at the lab information and the equipment they use, you can see that I’m somewhat lacking.  For example, I had no director class fibre channel switch and not enough equipment to fully test things out.  This is one of the biggest barriers to passing the CCIE data center exam:  Having the equipment.  You are at least looking at several million dollars here and that’s probably why renting is such a good option and makes a ton of sense!

Anyway, here are my tips for the lab, when I passed, as well as for life in general:

Tip 1:  Higher is lower/ lower is higher?

I was also informed of a very cool trick.  When you think about priorities of different protocols or features, there’s an easy way to remember it.  This was taught to me by Ryan Boyd, a great guy I work with:  If its a layer 2 protocol (LACP, fibre channel stuff, VCP, spanning tree) the lower the number means higher the priority.  If its a layer 3 protocol (OSPF, EIGRP, OTV, VRRP, etc) higher the number higher the priority.  Fabric path is tricky, because its supposedly layer 2, but when you realize that its running IS-IS as the control plane then it makes more sense that it falls under the layer 3 rule: The higher the number, the higher the priority.  Why didn’t anyone tell me this before?

Tip 2:  copy & paste

I had several people tell me they use notepad, copy the command line stuff into it and then just put it in.  One of my friends told me he did that and blew away his switch and had to start from scratch.  This takes away far too many precious minutes from your lab time.  Lab day is one of the fastest days ever.  I spent a lot of time trying to debug something in the lab the day I passed.  When I looked at the clock, I realized that I had just spent 45 minutes burning away lab time.  Bad form!  (Fortunately, I had everything else done) So I don’t copy and paste.  I just type it out on the command line.  I have really good typing skills.  Its the one thing in high school that I did on a typewriter that really helped and has stuck with me.  Plus, writing all that code in college got me pretty good as well.  So for me it was type away.  Even if I’m doing the same thing on multiple switches.

As an aside:  The other funny thing I noticed is that people that do Cisco switches don’t type in all the words.  They do things like

sh run or sh int brie

Since I have big Linux roots, I do a lot of tabbing.  So maybe I add one extra keystroke, but this works for me.

Tip 3:

Draw it out.  In Brian’s blog he shows how he spent the first hour drawing it out.  I didn’t do quite that much the second time when I passed, but I did read through each section before I started working on that section.  This helped me when I had to remember which interfaces were connected to where.  You get as much scratch paper as you want.  I used more than the average.

After I failed the first test, I scheduled the second lab attempt as soon as I could.  The problem was:  The next available time was in September!!  Wow.  So I checked every day, several times a day for an opening.  After 3 days of this, I got July 9th.  So my lesson of not getting off the train helped out.  I thought:  Let’s keep going.

My friends had recommended INE labs and those things are *really* good.  I read through some of my friends labs, but didn’t use any of them.  Instead, a colleague of mine was building a lab out of spare parts and I joined forces and we built it together.  I like this approach a lot because I like touching hardware.  I like knowing how to set it up from scratch.  I’ve always done this.  We got a study group together of people that were going to take the lab exam and we hammered through all kinds of scenarios, really making sure we knew how to do it.  I’ll never forget watching the USA play in the world cup trying to get all our components working.

I tore the lab up several times and the week before the test, I really went to town.  (UCS, N1kv, MDS, N7k, N5k on the 4th of July is super patriotic, so that’s how I celebrated!)  I was continuing to go through it all the way up until 11PM the night before the test.  By that point, I had had enough.  I felt super ready.  I slept all the way until 6AM, extremely thankful I didn’t wake up at 3AM again.  I was still really nervous.  I got to building C early.

10518018_830156400328249_982627123_nThis time I had experience and I blew through all the questions keeping track of points feeling like I got nearly everything.  By lunch time I felt really good.  By 2PM I was sure I was passing… if only I could get this one thing working… I got it working by 3PM by being calm and retracing my steps.  I spent the remaining time going through the questions and making sure I had answered them right, tweaking things here and there and finding some things I had forgot. I counted the points and even though there were some things I never got working, I felt pretty sure I had enough to make it happen.

I left San Jose and went to the airport.  I called my wife and told her I felt good, but still wasn’t sure.  What if I missed something?  What if I didn’t save something?  (But I remember saving at least 3 times on every item before I left, so I was pretty sure about that)  Before I boarded the plan an email came.  I opened it up.  Put my hands in the air and jumped for joy.  The people in the airport probably thought I had just won the lottery.  But this wasn’t luck my friends, this was being prepared.  I had passed.  I texted my manager a few good friends and thanked them for their support.  It was a good day.


Beyond Virtual Desktop Infrastructure

I wrote a blog a few days ago that I wanted to modify because I didn’t get it right.  First of all, please note that everything I write here are my own thoughts and not those of my employer.

This article is about Virtual Desktop Infrastructure (VDI), end user computing, Desktop As a Service (DaaS), or whatever you want to call it.  Its very relevant to many organizations today and there are a lot of great solutions and people very vested in it.  Is this the year of the virtual desktop?  It is to some people!  To other people, it was 4 years ago and what’s the big deal?  But to some organizations, its not going to happen ever because there’s no use case.

What problems VDI solves

Let’s think about the problems VDI solves.  It gives us our enterprise environment remotely and allows Desktop support to control the image that workers get.  That’s what it does, not the problem it solves.  The problem it solves is giving us our enterprise applications anywhere.  You see, many of us could care less about having our mandated enterprise environment.  When I worked for my formal employer, the first thing I did when I got my corporate issued laptop was to promptly erase their blessed image install the whole thing from scratch.  Wipe it out, get rid of employer stuff and put Linux on it.  Then I was in control.  Then I’d worry about getting the apps on that I needed and used and not everything else that I didn’t need.

Desktop support probably didn’t like that, but I never called or used them and they never called or used me.  I got the apps I wanted and whenever I gave a presentation, I never had a little window at the bottom prompt me that my computer needed to reboot in 10 minutes to install some extremely important updates.  We lived separate happy lives.

Desktop support is not evil.  They need to control the operating system image to ensure the applications could run and run securely.  Plus they aren’t catering to people like me.  They’re catering to people who just want to get things done and not mess with things like I do.   So when you look at what VDI is today, its extending Desktop Support’s control into a virtual image.  I think this is great!  Then I can run my own image and whenever I need my corporate apps, I can log into a VDI image.  Perfect.

Why VDI is temporary for most Enterprises

But, VDI in most cases is a patch, or a temporary solution to getting today’s legacy applications to enterprise users.  Here’s where it works very well:  If you have an application that was written for Windows XP or Windows 7, then creating a virtual desktop to serve those apps can be very effective.  But applications have changed.  Most of my applications I use are web based.  I still use Excel,  and PowerPoint, but I store those now in Box that my company provided me as a secure place to put them.  (Think: DropBox for corporations)

My Desktop support is now application support.  They make applications available to me and I can use whatever device I want to access them.   They now have even greater control:  When my corporate support team updates our configuration tool that I use to create build of materials (BoM) for my customers, they control upgrades and revisions.  I never have to do it on my laptop.  Even if I liked the old way better, I have no control.  Application support now has more control than ever and ensures no one is running older apps.  Its great!  (You may have seen people complain against the new Facebook layout in the past.  Nothing they can do, because its not an app they run on their desktop)

Applications continue to migrate this way.  No one is continuing to build the next great desktop application.  They’re looking to make applications that run anywhere on anything.  Even Microsoft Office runs on my iPad!

In fact, if you look at it:  Desktop support (newly rechristened as : Application support) is actually getting more control while I feel like I’m getting more control!  What a great arrangement for two type-A personalities.

Skipping VDI 

I thought about this a little bit over the last few years but it wasn’t made super clear to me until about 2 weeks ago.  I happened upon a visit to a little known school district in the mountains of Utah.  Davis County school district  is the most advanced public school district I have ever seen.  I was blown away.  We started out talking about their applications and data center plans.  Mark Reid, the IT director, and several of his coworkers have been at the school district for the last 30 years.  Its a testament to see what the power of vision and long standing partnerships can achieve.  From the very beginning they’ve been writing their own applications to deliver IT services to the district.

Unlike many of the IT shops that I work with, Davis County employees a staff of developers that churn out their own applications for the school district.  From payroll, to financial, to grades they are doing it.  In fact, they even have an application myDSD that allows you to log in from the web or even on your iPhone or Android to check grades, notify if your child will be absent from school, and pretty much anything else you might need from a school district.  Wow.   I bet your school district doesn’t have anything like that.

The applications speak to each other through different software layers and protocols but they all come back to an Oracle RAC cluster.  This is where all the data is consolidated and backed up. They’ve already got Office 365 for the students out there.

During our meeting, one of the people in the room asked if Davis County School District was thinking about VDI.  Before Mark could answer, I already knew:  They didn’t have legacy apps.  There was no reason to deliver a virtual desktop.  All the applications could be accessed from the web or iOS/Android clients.  You see, if you already have apps that can live anywhere, you don’t need to serve a special desktop image.

The real problem they need to solve is a way to stitch together distributed data centers and develop a plan to source workloads to different clouds.

Where VDI will always be important

VDI is still and will be important to many organizations.  After all, it sure is better than installing and managing a bunch of desktops in a computer lab.  People still have legacy apps and there are license restrictions that may make you have to do it on a blessed desktop image.

But one area that is really growing the use of VDI to share powerful GPUs for heavy graphics applications.  As data continues to explode, visualizing it will be ever more important.  That is why I don’t see an obvious replacement or better way to do this.


How do you think this transition of applications being centered on a desktop to being cloud enabled will effect the future? Back around 2006 when I was a remote worker at IBM they announced we could no longer expense our internet service.  They reasoned that most homes had this anyway and besides it was a great way for IBM to cut cost.   HP followed and so did others.   Soon all the tech companies started to do it.  Most companies don’t pay for remote access even though a significant amount of employees work from home.  (source:  my friends)

Today most companies will issue laptops to their knowledge workers.  Its great and they refresh every few years.  But could there be a time when employers say:  You already have a device (computer,  iPad, etc) we don’t need to pay for that anymore.  Just use our VPN service to get your applications and you are good.   Perhaps instead what they would do is give us an allowance of money that could be spent on a machine.

I don’t think this will happen at my employer soon because a nice laptop machine is a nice perk that makes employees happy.  But what about the universities and schools?  Would they eventually just shut down the computing labs and mandate all students bring their own?  Probably not for the engineering/art ones as I discussed above where they need GPUs.  But My friend’s kid 4 years ago went to a private school.  It was mandated  that  every student get a Mac Book.    The days may not be far off.

So next time you are evaluating whether or not this is the year of the virtual desktop, first look at your strategy for delivering applications anywhere.  Perhaps resources should be diverted towards new applications or BYOD initiatives to get off the legacy applications that are tying you down.  Remember:  Nobody wants or cares about your enterprise desktop image (nobody in their right mind).  They just want applications that work and allow them to get things done.


Distributed Data Centers

My thoughts on what cloud computing and the future of the data center has changed a bit in the last 3 years.  When I first started working on a cloud computing project for a large bank in America back in 2008 I was convinced that soon every enterprise would create their own private cloud and use xCAT (or something).  Then I thought they would instead all use OpenStack.  But I figured every organization would indeed build its own private cloud.  This has not panned out.  Not even close and its 6 years later.

Eventually, I thought, all enterprises would migrate to one public cloud provider, and it never occurred to me that people would see fit to use more than one public cloud provider.   I did form a concept of the InterCloud back then so I’m not too far off the mark.  But my vision is evolving and becoming more clear.  I finally see where IT is going.  (Or at least I think I do)

In my small sector of the world hardly anybody has a private cloud.  And when I say private cloud, I mean self service portals with completely automated provisioning.  Yeah, that’s just not happening.    The truth is, I don’t think it will for most organizations.  There’s not enough need there.  The only people that need VMs in a self service portal for most organizations are the VMware admins themselves and they are savvy enough to right click and make that happen without all your bloated self provisioning tools, thank you very much.

What I am seeing is that more and more are going to the public cloud.  This started out more as a shadow IT initiative, but more of the people I work with have in fact embraced it at central IT.   But its managed as a one off and people are still trying to figure it out.  People aren’t ditching their own data centers, and just like they’re not ditching their mainframes, in large enterprises there will always be some footprint on premise for IT services.

The other thing that seems completely obvious now is that people will want to use more than one public cloud provider.  The reason being some public clouds specialize in different things.  For example:  I might run Exchange/Office 365 on Azure, but I might run some development applications on AWS.  Similarly, I might have a backup as a service contract with SunGuard.  But I may not trust my data to anyone but my own 6 node Oracle RAC cluster that’s sitting in my very own datacenter.  Can you see where this leads us?

Central IT is now responsible for sourcing workloads.  The data center is distributed.  My organization’s data is all over the place.  My problem now is managing the sprawl.  Getting visibility to where the sprawl is and making sure I’m using it most effectively.

Another misconception I see is that people think using two or more public clouds  means VMs move between data centers.  Today, that’s pretty impractical.  Migrating VMs between data centers takes too long, even if the network problems weren’t a problem.  And besides, when you think that way, you are thinking more about pets in your data center instead of cattle like the future of applications is.  So forget about that right now.

Instead, focus on the real issue that needs to be solved.  And this is where I think Cisco can make big things happen.  That is:  How do you connect distributed data centers?

The Nexus 1000v InterCloud, or InterCloud Fabric I think is what Cisco is calling it now starts down this road.   It allows us to communicate with VMs in a public cloud with our own cloud using our same layer 2 address schema.  This is pretty cool, and a good start, but we’ll need more.  For example:  We might have our data base servers reside in our own data center.  (No self service portal here). Then we’ll develop apps that will be hosted in public clouds.  The application servers will need to communicate with each other and with the database.  The different applications may be in different clouds.  The real issue is how do they talk and communicate effectively, securely, and seamlessly.  That is the big issue that needs to be solved with distributed data centers.

Is this where you think we’re headed?  I feel like for the first time in five years I finally get what’s happening to IT.  So I’ll take comfort in that for now, until things change next month.