Deploying Instances on COPC (metacloud) with Ansible

I wanted to show a quick example of how to deploy an instance on Cisco OpenStack Private Cloud (COPC or Cisco OPC or MetaCloud) with Ansible.  Since COPC is just a fully engineered and operated distribution of OpenStack from Cisco, this blog is also applicable to normal OpenStack environments.

I’m a big fan of Ansible because everything is agentless.  I also think the team has done a phenomenal job on the docs.  We’ll be using the nova compute docs here.  I don’t have to install anything on the instances to be able to do it and I can just run it from my laptop with minimal dependencies.  Here’s how I do it with CoreOS.

1.  Get Credentials

On COPC, you can navigate to your project and download the OpenStack RC File.  This is done from the ACCESS & SECURITY tab and then clicking on the API Access tab on the right.

COPC-Access&SecurityOnce you download this file, you put it in your ~/ directory.  I use a Mac so I just added the contents to my ~/ file. It looks like this:

Now, we’re ready to role.

2. Ansible Setup

I covered Ansible in previous posts.  So I’m going to assume you already have it.  Let’s create a few directories and files.  I put all my stuff in the ~/Code directory and then under the projects directory.  I then make sure everything in this directory belongs to some sort of git repo.  Some of those are on github (like this one) and others are in a private gitlab, or a private github repository.


This file will have our info for where our inventory is.

This will be global settings for our environment.  We tell it not to use cowsay, but you can if you want.  Its kind of cute.  You may not have it installed.  We also tell it to use the contents of the inventory directory (which we’re going to create) to go to our hosts.

host_key_checking tells it that when we access a new server, not to worry if we’ve never seen the host before and attach to it anyway.  Finally, our remote user is core as this is the default user for the coreos instance that I’m using.


We create a directory called inventory and add the file hosts.  We then add our one machine (our localhost!)  The contents looks like this:

You’ll notice here I also added which python I wanted to use, just in case I had other versions on the system.  This might be good too if you were using virtual environments.


This is where we put the specifics of what we want deployed.  In our case we need to define the following:

The security group ‘default’ in my project, as seen from the dashboard actually includes port 22.  This is important so that I can ssh into it after its provisioned and do more things.

I imported my coreos image from the CoreOS OpenStack image website.  After importing it in from the dashboard, I clicked on the image to see the image ID:

Screen Shot 2015-05-01 at 2.50.42 PM   The floating IP pool is nova, I got that from looking at the dashboard as well.

Finally, the keypair is one I generated beforehand and downloaded into my server so I can log into it afterwards.


This file is our playbook.  It will provision a server.  Let’s look at the contents:

The great thing about this script is that none of the secrets are put into it.  Using the environment variables that we did by sourcing the ~/ file we are able to run the code perfectly.

Everything here is pretty self explanatory in that we are just passing in variables to the nova_compute task to bring up a new instance.  The name will be demo-server and everything else we’ve defined.  If the instance is already up, Ansible won’t go and try to provision a new one.  Its looking for demo-server, if he’s there, he won’t touch him.

3. Run the Playbook

We’re now watch the output on the dashboard and you can see it will spawn up.

Screen Shot 2015-05-01 at 3.19.13 PM


The next step is to make it so we can run Ansible playbooks on this host.  The problem right now is that coreos is just a stripped down barebones OS.  So there is no Python!  We’ll have to add a cloud init script or do something else to make this work.  I’ll save that for another post.  But if you were using Ubuntu or RedHat, you’d be good to go at this point.


All the code in this is available at github here.

Cisco OpenStack Private Cloud Developer Demo

In my new role here at Cisco I show people how powerful the Cisco OpenStack Private Cloud solution can be for developers.  I made the below video last night to demonstrate continuous integration.

The video is a scenario for a new company called  The company provides lawn gnomes as a service (LGaaS).  Under the cover of darkness LG technicians will place 100 gnomes on the front yard of an address you specify.  The person living in the address you specified will wake up to find 100 gnomes on their front yard with a special greeting.  LG wants to add a new feature to their ordering site to allow for a personalized greeting.

The flow goes as follows:

  • LG decides they want to try out a new feature
  • Developers implement feature and perform unit tests to add the feature.
  • Developers check in the feature to a private repository.  In this case they are using Gitlab, an open source version of the service that Github offers.
  • Gitlab has a trigger connected to Jenkins, the continuous integration server.  When Jenkins sees a code checkin from the project it performs integration tests on the merged code.
  • Jenkins integration tests that pass then are pushed into a new Docker Container that is uploaded to a locally hosted docker registry.
  • Jenkins has a production job that monitors the Docker registry.  When new containers are pushed up, a job is kicked off to go through the containers, take them off line and put up the new container.  The load balancer (NGINX) handles the load.

This demo will be posted on Github with all the devOps code as I continue to refine it.  Also, any suggestions are more than welcomed!  Perhaps I’m doing it wrong?  I will post my solutions and then let Cunningham’s law dictate how accurate I am.


Subview Rotation on iOS 8 with Swift while keeping static main view

That title is a mouthful.  Basically, I just want to know how to imitate the native iPhone camera app.  The camera button stays locked where the home screen is but the icons rotate with the device as well as the capture area.  Should be pretty simple right?

tl;dr:  See the code here on github that I did.


Well with iOS 8 Apple introduced the concept of adaptive layouts.  See WWDC session 216.  This introduced size classes and traits which I think is fantastic.  Except for when you want to imitate the iOS camera application.  Then you don’t know what to think.

There were two main ideas I could have used that I came across:

1.  Using two windows.  This was brilliant and I think would work.  This even came with sample code to show how to keep the rotations separate.  I had read other people saying it was a bad idea to have two UIWindows.  I played with this a little bit but it seemed too much for what I needed.  Plus, I had the UI Tab Bar controller as the root image so it was somewhat complicated.

2.  UIInterfaceOrientation.  These methods all seem to be depreciated in iOS8 and may or may not work.  The problem with these methods is that the root method gets the notification and then signals to every body else.  I may have been able to work with this but I didn’t want to go through all the way down the hierarchy and implement all these methods for those that should be static and those that shouldn’t be.

I went with UIDevice.orientation.

Here’s the steps:

1. Subclass UITabBarController

Since my main project has a tab bar as the root interface I started here.  This way I want all the views to be able to rotate and use auto layout to do this.  There’s just one subview that needs to stay the same.  This was accomplished by adding the following method to the view controller:

This makes it so that this view 1 in the tab bar won’t rotate.

2.  Subscribe for notifications in the App Delegate

I may have been able to do this in the main class, but I did it in the app delegate in case it didn’t get alerts.  Then I had that propagate another notification.  This may be a redundant step, but figured I’d try it and was too lazy to change it back.

 3.  Subscribe to notifications in the non rotating view controller.

Now to react to these we’re going to rotate the subviews that need to be rotated.  This is done in 3 methods:

Maybe you have a better way?  I’d love to know!

There is one problem with this method:  If the application launches in landscape mode then you’ll have to rotate it a few times to actually work in the right mode.

See the full code here.

Corkscrew: SSH over HTTP Proxy

I found myself behind a firewall that didn’t allow SSH to the outside world.  They did allow an HTTP proxy though.  So I thought:  Let’s make sure we can get out!  After all, I needed to download some of my code from Github, make changes, and then upload again.

Here’s how I did it.

1.  Install Corkscrew


2.  SSH Config

Now let’s make github work.  We edit ~/.ssh/config

Now we can test by running: ssh github

This gives us the familiar response:

I won’t worry about the channel 0 failure.

3. Git Config

Inside our git repository, we can update .git/config to point to the alias of github.

Now we can do git push without any issues!

Secrets with Ansible: Ansible Vault and GPG

I was blown away last night at our Ansible PDX meetup by a great presentation by Andrew Lorente about how to track secrets with your applications.  Andrew gave a method of how to do this that I wanted to write down so I know how to do it.  Andrew has his own blog here where he wrote about the solution.  I wanted to go over it a little more in details cause I want to make sure it sticks in my head!  (bonus: I also learned about the pbcopy command on Mac last night!)  The other thing is since I didn’t have any of this on my machine it helps someone get started who hasn’t done anything with GPG yet.

His technique involves some pretty simple tools:

1. Generate an Ansible Vault Password

On my mac I run:

This gets me my tools!  Ok, so now I need to generate my pgp key.

Doing this I just accepted all the defaults.

Now I generate a password for the vault.

Now that I have that I follow Andrew’s instructions and create a file called with the contents being:

Then make sure I can run this file as an executable

Add this to my ansible.cfg file

 2.  Setup the GPG Agent

If you now run the command ./ you’ll find that it says: “Hey, there’s no agent running!”.  There are a few ways we can start the agent.  You can create a LaunchAgent as shown here, or you can just configure something in your own shell.  I went with my own shell method and basically followed this post.

Create ~/.bash_gpg

Append to ~/.bashrc

Create ~/.gnupg/gpg-agent.conf

Opening up a new shell, I should now be able to run  the ./ command.  It will ask me for my password the first time, but if I run it again, it won’t ask me again.  Right now the default-cache-ttl is set to 600 or 10 minutes.  This can be increased if I want it open longer.

3. Encrypt the file

The file I will be encrypting is a main.yml file that contains all my variables.  Since it already exists I run the command

Now, if I look at this file roles/openstack-controller/vars/main.yml you’ll see its just a bunch of random encypted numbers!  Awesome!  Now my environment variables and my password file can all be committed with git.

Now obviously, if you look at the history of this project I’m working on, you’ll see the old unencrypted file, but that’s ok, I’ve changed the passwords now so its super secure!  From now on though, no more simple passwords and I’ll be using these methods to encrypt.

4.  Sharing Keys

Obviously this solution works great for one developer, but what if we have more developers?  They will also need to be able to run the key.  To do this, we just need to encrypt the file with all of our users.  We now decrypt the vault_passphrase.gpg with our command.  We then get the output of our passphrase.

Now, we encrypt it again with all of our users.  The new user will need to share his key with you so that you can encrypt it.

Test that it works by trying to edit the file


VMware is the AOL of the private cloud

This week at VMware Partner Exchange there was a nice announcement of VMware Integrated OpenStack or VIO.  VIO attempts to do what most of the developers of OpenStack have failed to do:  Make OpenStack easy to deploy and manage.  (It’s not, anybody who tells you different is trying to sell you something).  The sound bite is this:  You can leverage your investment in both skill and knowledge in existing VMware products and transition to OpenStack.  Or this:  VMware is the easy button for OpenStack.

The first statement (paraphrased by me) is worthless and the second statement (also paraphrased by me) is just a lie.  Putting OpenStack on top of ESX is actually more difficult to manage and troubleshoot than if you were to run it on native KVM.  When you use VIO you are dealing with an appliance that has its own layers of complexity as well as more dependencies that don’t really need to be there.   I’m convinced that if you want to learn open stack, you can’t use these magical tools to install them, but that you first have to go through and manually install it.  It took me a week the first time and every time I’ve done it since I run into other issues.  Each time, however, I come away with a better understanding of the core components and have even made my own automated installer for learning purposes only.  So in that sense, you are not at all leveraging  your investment in VMware and it is instead just holding you back.

I caveat that above paragraph by also saying I’m not new to Linux administration nor managing large scale systems. So I shudder to think how the person with all their Microsoft and VMware certs with little Linux experience will do with this.

The analogy is AOL.  Remember in their sunsetting days when they started offering AOL broadband?  By that time we were all too advanced and realized we didn’t need all that bloatware, nor portal in our environment.  So we cast off the shackles of AOL and started making art with their DVDs.

This is how I view VIO.  However, for those of you who have not worked with OpenStack and want to just try it out, I say go for it!  Just like I would to those who had never been on the Internet before.  But there will come a time, where if you want to move forward with it, you’ll have to learn some new skills.  This is IT after all, and if you’re not learning something new every year, you quickly become irrelevant.

There are still workloads that don’t go well with an OpenStack nor AWS model.  Exchange and SharePoint in my opinion still do great on VMware or Hyper-V.  Any time you are treating applications like pets instead of cattle, then VMware vSphere, is a great solution for you and there really isn’t a need for a self service portal.  Certainly VMware software will continue to evolve, but there are few products from them (or even other traditional enterprises and even my own company Cisco that will help you)

The State of Things

Everybody is failing at making a successful private cloud.  I used to think it was 80% and that the other 19% were just in denial, but it turns out its 95% if you believe Gartner (which I usually don’t).  Why all the failures? Is it management that doesn’t get behind it and has no vision?  Is it that the teams are too siloed? Do the engineers just lack the skill set, or are too stuck with their own pet technologies?  Probably all of these.

And it seems to be getting worse for central IT.  We had one customer (Central IT in famous University) where we asked them what projects they were working on this year.  “What’s in your budget?  What do you guys want to do this year?”  He responded by telling us that his budget was cut to basically 0 and they are in maintenance mode.  Instead the lines of business have all gone to public clouds.  Our customer in Central IT is now just supporting file shares and legacy systems.  This is the future for most of central IT unless they evolve.

You’ll notice that there isn’t “The Private Cloud solution” for those that wish to serve their constituents like AWS can serve their customers.  Every legacy IT shop offers one, but its not selling off the shelves.  Cisco has UCS Director that tries to do it, HP has their tool, VMware has vCAC or whatever its called now… But OpenStack is the only one that is universally both hailed and derided.  OpenStack is the solution central IT would love to love, but can’t because its too X, where X = (immature, difficult, geeky, esoteric, complicated,…)

OpenStack is the solution central IT would love to love, but can’t because its too X, where X = (immature, difficult, geeky, esoteric, complicated,…)


The economics still show (too lazy to find link, exercise left to the user) that hosting your own private cloud is cheaper if you can do it.  There was one person I met a the last Ansible meetup who told us his start up company that had no shipping product, nor users was running a $20,000 AWS bill every month!  The case for moving to a public cloud isn’t to save cost, as my good friend tells me, its to become more competitive.  Its to move faster and get products out and make more money.  Plus, nobody trusts that their central IT can even deliver it and keep it running.

Well I hate to be all negative and doom and gloom.  So I’d like to propose a possible solution.  Its called MetaCloud, but was recently rebranded Cisco OpenStack Private Cloud.  I think we’ll be hearing a lot more about this offering this year as more enterprises embrace it.  Check it out and see the advantages.  Its the product at Cisco I am most excited about.  MetaCloud allows central IT to focus on services to their customers:  Database as a service, Load Balancing as a service, common app as a service, etc.  That is the new role of Central IT.  Providing higher level services instead of just infrastructure.

To conclude, I’d like to encourage central IT people to try out VIO as a way to test OpenStack, much like I would have encouraged people who wanted to get on the Internet for the first time to do a 60 day trial with their AOL disk they got in the mail.  But keep in mind that just having OpenStack deployed is not going to keep your customers from fleeing your firewall and going into public cloud services.  If you want to keep users, the real value IT needs to do is deliver higher level services.  Which services?  Take a look at the things you can get from AWS for examples, because that’s who you’re competing against.

iOS: ManagedObject doesn’t trigger NSFetchResultsController Reload

One of the problems I have been working on is to update a UITableViewCell with updates after the parent view controller reappears.  In this case the user clicks on a ‘comments’ button which segues to a new view controller where the user enters comments for the post.  Once the user enters these comments, they click the back button to return to the previous view controller.  That view controller should have one of its table cells updated with the new comment.

But it doesn’t work!

I tried several solutions and spent a few hours on this.  There were some helpful posts explaining what happened such as this one and this one.   None of those seemed to work for me.

I started out creating an unwind segue on the main view controller:

And as you can see from all the commented out garbage above, I tried just about everything to get that table cell to reload.  The curious thing was that the size did change, but the contents did not.

For completeness, on the exit for the comment view controller I had the code call this segue:

(Note: In storyboard you have to control drag from this view controller to the exit and select the “unwindComment” segue for this to be hooked up)

Well, none of that worked.  What I found was that the new comment wasn’t being saved in time for when the segue unwind was called.  I think if I put that segue in viewDidDisappear it might work as that is called later but I’m happy with my solution so won’t change it.

Instead the way I solved it was by putting in a notification in my Data Model when the comment was saved.

Here when its saved, we post a notification saying we saved something.

Now, back on the main view controller (where there is no editing) we listen for this notification:

Then we update the tableview when this is fired:

What a relief it was to see this work!

Now when the user clicks the back button after posting a comment, the main tableview is updated with that comment.

Git proxy

To install Ansible on my RH machine I had to get out from a proxy.

First, I modify .gitconfig

Mine looks like this behind the corporate firewall:


It does the same thing.

I also had to add to my .bashrc file:

Strange that I had to add it to two places.

I also found I couldn’t do it like the Ansible documentation said.  I had to do:

That got me a good environment.  Next I created /etc/profile.d/ansible:

That seemed to work for me!

New Job

For the past 3 years I’ve been working at Cisco in what we call the SLED vertical.  SLED = State Government, Local Government, K-12, and Higher Education.  I have nothing but positive things to say about the entire experience.  Its been very different than what I’ve been doing for the previous 10 years.  The people were awesome and I am so happy I was able to meet and have such good experiences with them.

Previous to taking this job at Cisco my day to day activities for the last 10 years was more on the bleeding edge: Scientific applications running on Linux, Open Source, KVM & large scale out distributed systems.  These last 3 years gave me exposure to VMware & Microsoft in the Enterprise.  While they are not necessarily my favorite platforms to work on, it has really helped me understand them a lot better.  I’ve also been able to learn a ton about Cisco enterprise products:  Specifically UCS and Nexus.  This has been great to be known as an expert in these fields. And those products I do like working on simply because: “They work”.  What you don’t know I’m an expert?  Ok, well, I can at least open a TAC case for you.

Like Nancy, I am “Overjoyed”

Starting in the next few weeks I’ll be transitioning to be working with the Cisco Cloud & Managed Services Organization.  I am beyond thrilled to be working in this space.  Or in the words of Fancy Nancy:  Overjoyed.  (Yeah, I have a kindergartner that we’re reading these now).  My goal is to help transform Cisco into the type of company that people look to when they are trying accelerate application delivery.  Specifically:  How to do better Hybrid IT.

I’m looking forward to doing a lot more DevOps and get back to my programming roots.  So you’ll see a lot more of that on this blog.

Currently, I’m reading a book called Lean Enterprise.  I think it should be required reading for those that work in a large organization.

Buy this book now and read it. Then start questioning everything your boss tells you to do.


One of the things that struck home was this quote:

“Whenever you hear of a new IT project starting up with a large budget, teams of tens or hundreds of people, and a timeline of many months before something actually gets shipped, you can expect the project will go over time and budget and not deliver the expected value”

This quote struck me especially because when I read it I started thinking about this announcement.  So that’s the warning.  Its not as if Cisco is alone in trying to to transition to the next big IT movement.  HP, Dell, and IBM are all trying to do it.  I think Cisco can be successful in this if they can do one simple thing: Deliver outcomes that customers want.

I’ve taken my mission to do this with two easy mantras:

1.  Develop solutions that customers want.

Lean Enterprise talks about how decisions are largely made by the HiPPO (Highest Paid Persons Opinion).  I’m a fan a analytics and trying things out.  See how they work and moving on quickly.  I am hopeful Cisco can continue to develop this practice.  I’ve seen it already with the Project Squared initiative that was received rather well.  I’m a fan!  I hope to see more like this from our cloud organization.

I have been very vocal about certain products and asking for use cases to make sure they are the kind of things I would want if I were trying to run a business.  I hope to continue that.

2.  Develop solutions in the open.

Instead of keeping all this back behind the Cisco curtain I want to push our company to develop everything they do in the open where people can see.  Docker networking sucks?  Security is immature?  Perhaps we should create a framework that maturates it.  Get a Mean Viable Product out there with as least amount of work as possible.  If it works, expand it: See if we can support Rocket too.  Make this open source, let our customers use it.  If it provides value, move forward, if not scrap it.  Facebook’s ‘break things’ and move fast mentality applies here.  But unlike Facebook, I’m hoping our outcomes actually do better things than waste people’s time with clickbait with captions like “You won’t believe what happens next”.

I’ve been influenced a lot by the talks and discussions I heard at Dockercon.  One that resonated is that middleware should be open and free.  We server a greater purpose, get bigger buy-in, and move faster if people can try it for free.  That’s not something a ‘software only’ company would like to hear.  But if your company is focused on outcomes then it makes perfect sense.  An outcomes company not only makes its money with support and services but with a platform people can build upon.

Looking forward to some fun times ahead!

Oh, PS, I had to share the best UCS Presentation I’ve ever seen.  You won’t believe what happens next.


iOS: bounds vs. frames

I got totally hosed this week trying to add a subview to a UITableViewCell.  The cool thing is that I learned a lot.

I created a custom UITableViewCell that called FeedCommentTableViewCell class.  Below is the working code:

A few lessons learned:

1.  initWithStyle is called only when the cell is created and thus will be used multiple times.  So I set the frame to zero realizing that this cell will dynamically change in size based on a function I wrote in the UITableViewController subclass.

2.  Based on experiments suggested by this post, I learned that tableview:CellForRowAtIndexPath will get called first and at the very last layoutSubviews will be called as the cell is displayed.  This is the place to set the size of the subviews based on the size of the cell.

3.  I was setting my (commentLabel in the example above) subview’s frame based on the cell’s frame (self.frame).  This was a no-no.

Expanding on number 3.

Let’s say my layoutSubviews method wanted the commentLabel to be the same size as the cell itself.  I originally did something like the following:

This would render the first cell perfect, but as I scrolled and scrolled back up I saw problems.  You see, self.contentView.frame is the coordinates of the cell in the parent view’s coordinate system.  We want to assign the frame of the commentLabel to be set to the coordinates based cell’s subview itself.  Simply moving this to something like:

Adds this subview into the coordinate system of the contentView, which is what we want!

Now I’m sure Autolayout and using Storyboards make this easier, but I’ve found that for more complex UITableViewCells, laying it out in code seems to be a lot easier to debug.

Moving on to the rest of the project now that that bug is under my belt!