KUBAM update

While working at Cisco I developed a project called KUBAM:  A simple bare metal provision tool for UCS.  I updated the kubam.io page and am now offering support for $500 / node per year.  I couldn’t do this at Cisco because I could never get it officially recognized by the powers that be.  That is nothing against them, they have their own agenda and objectives they are trying to achieve.  I’m excited to work on a few new features that will help improve usability. KUBAM will continue to support UCS only as I have no other plans to support any other servers.  This focus helps me to get the most out of the UCS APIs and bring greater value to the offering.  I will probably do little advertising to start out only a mention here on this blog as well as the updated home page.

Since leaving Cisco last February I’ve oscillated between amazing happiness to be creative and setting my own path as well as longing for the old days when I could be with so many great people and bask in the safety of a giant warm corporation.  However, the freedom and ability to work on projects I’m passionate about have above all given me greater sense of purpose and drive to improve my craft.  I’m fortunate enough that money is still coming in and we have a long runway.  I’m more confident now that I’ve made the right decision.

Over these next few months my goals for KUBAM are to do the following:

  • Improve documentation.  Right now the docs sit at an old Cisco page, but they need to be retired and updated with more streamlined information.
  • Improve OS installation methods and add more samples.  We were able to add Windows last year (I remember sitting on a beach in Castelldefels writing the Windows code) and Ubuntu in January (Frantically trying to finish before I left Cisco for a customer to use), but those methods can be improved with a little research.
  • Improve the GUI.  I’ll probably rewrite it.  I appreciate all the work that Lara and Michael did for it while I was at Cisco but I think we need to rethink the workflow and probably add a little more back in.

I’ve appreciated all the people who have reached out to me about KUBAM and given constructive feedback and encouragement.  It’s a fun project and I enjoy working on it!

As far as what else I’ve done these last 6 months I can briefly write them here:

  • Recorded a Kubernetes on Bare Metal class for Pearson.  This took a ton of work and we’re still going through the editing process.  It’s pretty painful for me to watch myself in front of a camera but I’m told I don’t come off too bad and I hope others will gain from it.
  • Created a new service called Zenabi Data Services.  (ZDS).  The aim of the service is to provide a streamlined way for people to upload data and automatically get data science results.  I’ll update more on that as time goes on. The MVP of ZDS is that you can enter your Google Ads credentials and ZDS will automatically run your ad campaigns using AI.  So far we’ve seen incredible return on ad spend (ROAS) on our customers and we’re just getting started!

All in all it’s been great stuff!  I hope whatever your professional goals and projects you are working on you are enjoying and if I’ve met you in the past I hope our paths cross soon again!

Kubernetes Cron Job vs. AWS Glue

As I’ve been dealing with streaming data one of the architectural decisions I’ve had to make is how to run periodic batch jobs on the data as it comes in.  In the case of web traffic, it is logged into a database.  What my batch jobs do is take the data from the MariaDB MySQL database, convert the data to Parquet format and then store the data in AWS S3.  Once it’s in S3 I want to update RedShift Spectrum to be aware of the new data, then run a query on RedShift spectrum that I can then feed into a Redis database which is used by an application to give pretty close to real time results.  Whew!  That is a mouthful.  Perhaps a diagram would be helpful:

The blue box represents the first job that has to run.  As I did this work in python I thought at first I could use AWS Glue to run the job.  After all, it’s a simple query and store operation.  But as I’ve been debugging Glue I found that it was actually easier to just put this python script into a Kubernetes Cron Job.  This gives me the same functionality and since I’m already paying for EKS it gives me more utilization out of it.  Since a lot of the other infrastructure runs in EKS, there’s no reason not to use it.  I’m familiar with both and this was a quick win.

The Glue job is the orange box.  In this job it crawls the S3 directories that I setup and then creates the format.  This is simply configured from the AWS Glue console with mostly default parameters. I’ll need to figure out how to make this part automated soon, but for now it seems to do the job.  The part I keep having issues here is that some data when it comes in may not be formatted correctly and it crashes my queries.  To get around this I keep having to change the job in the blue box.  Kubernetes makes this pretty easy, but still not my favorite.

The green box represents the kubernetes cron job that runs queries in RedShift that our data scientist Min gave me.  The query results are than placed into Redis for processing.  Again I could have put this in Glue, but Glue I don’t think saves me much time that Kubernetes already gives me with cron jobs.

That therein is perhaps the part of Glue I’ve learned:  It saves time on S3 crawls, but doesn’t save you much time in doing some basic other ETL jobs, especially when you have a Kubernetes cluster.  I’m a big fan of keeping things serverless and using Kubernetes in this way still feels serverless to me.

One part we need to look into in the future is making sure all of our jobs are processing without errors, finding problems in the flows, and working on visibility for our end users.  Pretty fun stuff!

Golang: AWS SDK, Cognito and API Gateway

The situation is as follows:

  1. Create an application with the serverless framework.  This uses API Gateway, Lambda, and all kinds of cool stuff.
  2. Authenticate on the application using Cognito.
  3. Write a client that can call the API created by API gateway in Go.

Steps 1-2 are covered everywhere on the internet.  My favorite reference is this serverless stack tutorial.  It is gold and covers so much.

But step 3 is pretty poorly documented.  What I found are several old libraries that do v4 signing but are no longer maintained and shouldn’t be used.  Below I document a solution that works using the AWS Go SDK.

1. Cognito Setup

User Pool:  I have a user pool and a corresponding App Client.  The App Client for the command line Go code that I’m writing is separate from the App Client that is used by the web interface.  You’ll need to note both of these:

  • UserPoolId: us-east-1_123456789
  • AppClientId: 123456789abcdefghijklmnopq

Federated Identity: In addition an identity pool which uses the user pool ID should be created.  This will also give you an identification

  • IdentityPoolId: us-east-1:abc13813-4444-4444-4444-123456789abc

Make sure you can login and out and that this cognito stuff works.  I used the serverless stack tutorial above and I could log in and out with the web interface.

2. API Gateway Setup

I’m using the serverless framework to create my stack. A few notes about it:

  1. The Method Request should specify that it requires Auth: AWS_IAM
  2. The Cognito Identity pool has an authenticated and unathenticated role.  Make sure that role has the proper permissions to call the lambda functions.

As an example, my cognito identity pool authenticated role has the following properties:

You would make sure the execute-api is set to your correct API Gateway ID.  You can find this in the stages portion where you invoke the API:

  • InvokeURL: https://123456789.execute-api.us-east-1.amazonaws.com/dev

3. Golang

By leaving out the error checking and structure I’ll make this as simple as possible to do.  You’ll probably want to do some caching of some sorts and of course check for errors.  In the steps below we will turn our Cognito username and password into IAM credentials that assume the role of executing the API, after which we will use them invoke the API.

3.1 Create aws session

3.2 Authenticate user from Cognito Identity Provider

Your cognito user has a username and password.  (I’m using an email).  Authenticate this:

The AuthResp will contain the IdToken, AccessToken, RefreshToken, etc.  What you need is an IAM user.  Notice that you’re just using the AppClientID and not the UserPoolId.  I thought this was a little strange but since the AppClientId belongs to the UserPoolId I guess it works.

3.3 Get ID from Cognito Identity

This section follows (to some extent) the documentation in the Overview section of the cognito Identity API Reference.  (Seriously, Amazon, a few examples would be nice) Now that we have the IdToken we can use that to get the ID of the user:

This result gives us a user id which we can now get credentials:

Cool, now if you look at credRes you have IAM AccessKeyId, SecretKey, SessionToken!  Everything we need to now call our API.

3.4 Invoke the API Gateway with a Signed Request

To invoke the API we create a new request like we would if it were unauthenticated:

In this case I’m calling the “/list” resource and using the GET method.

Now the trick here is that we need to sign the request.  With the SDK we now have a library that does that for us:

Notice that we pass in the req variable.  This step will add headers to the request that will authorize our request.  Finally do the request:

Check out the response:

If all went well you have just sent an authenticated call to your API using the AWS SDK for Go.

4. Parting Thoughts

This is very complicated, but hopefully very secure!  I’m putting this here as I saw very little documentation.  One example goes a long way to drive home understanding and I’m hoping I can save someone else some time.  Having never done this it took me 2 days to figure this out! There seems to be other methods of accomplishing this as well.  For example, in API Gateway you can configure an authorizer that can accept just the IdToken from the Cognito User.  Using that method you could skip 3.3 and just add a header instead of using the v4 library in 3.4.  However, I already had this API setup for the web interface and didn’t want to change what it had.

 

 

Ingress on AWS

While looking for different options on how to run an ingress controller on AWS I found that EKS recently announced support for an open source project that creates an ALB ingress controller.  One of the advantages of an ingress rule is that I can have many services be reachable via one external load balancer.  In the research I found that the EKS ALB ingress controller does not have that feature.  Instead for each ingress rule it creates a new ALB!  That sort of defeats the purpose of what I’m looking for.  As such, I’ll be going back to good old trusted NGINX.  NGINX is great in that it supports TLS and I can attach it to one NLB.

There may be features NLB lacks that ALB has, but for now this is the option I will role with.  I just wish someone would have told me that as it cost me about 2 hours of research.

Book Review: Sapiens A Brief History of Humankind

Sapiens: A Brief History of Humankind

If you ever want to get a real perspective on the destruction our species has done to the planet and most of the other species on this earth, this is a great read.  I was fascinated by the accounts of our ancestors spreading from Africa to the rest of the world.  I read in horror that after several thousand years of Sapiens arriving in a new place, like Australia or North and South America how the native fauna was wiped out – and evidence that this most the cause was our own species

large mega fauna from north america which evidence suggests was the result of the actions of humans.

I loved learning about the giant Megatherium – A giant sloth

Or the large Marsupial mega fauna as well as the Thylacine.  Equally fascinating to me was to realize how Sapiens lived contemporary with a number of other human species like the Neanderthals and others.

These other species Harari gives evidence that they were probably also casualties of the expansion of Sapiens.

The last part that made me completely sad was the chapter on “Life on the conveyer belt” or the life of domesticated animals such as cows, pigs, and chickens that we have mass produced and consumed making these animals completely miserable in the process.  This is another good reason to make me want to be a vegan.  While everyone has a problem with how the sausage gets made people still eat the sausage.  (I had some tonight in fact!)  However, this part has influenced me to want to live a more plant based diet.

By midway through the book I was quite sad about the human race.  It seemed very tragic and unnecessary.  But then I felt there was a period in the book, after all the wars to the present time to where I started feeling good.  Harari mentioned how low the chance of war is right now in many parts of the world.  Comparing life now and 500 years in the past things are quite good.  I started realizing that many humans are realizing the extent of the damage and are hoping to take responsibility.  For the first time in all these millennia it looked as if humans were starting to try to do something good.

But then just as you were starting to feel good about all the great things that are happening because of the age of science and admitting we don’t have all the answers you get lambasted with all the crazy things that are about to happen:  How our species will eventually be extinct because of our own doing.  Not because we’ll blow ourselves up (although that is a possibility he doesn’t deny), but because we will transform our own species through scientific discovers.  These bring up new issues:

  1. If we as a species are able to become a mortals by curing nearly all diseases then who gets to decide who gets to live forever?  Harari mentions that for years the consolation of the poor has been that the rich have the same 24 hours every day and their time is limited the same as any man.  How unfair it is that they would now get to live longer.  (As if anything is fair in life or even should be)
  2. If we are a species are able to enhance our minds to become super humans, who gets to have this procedure done?  Do we start making different classes of humans with the un-enhanced humans eventually being made extinct over time?
  3. What about when we are able to read each others mind and share a common brain, and download the memories of one person into our own brain?  Is that person us or are we still an individual?

The future from this perspective looks wild and perhaps a bit scary to live in.

Finally the message I get from the book is that everything is temporary for us humans and that meditation and trying to free ourselves from wants seems to be the only way to be happy.  The author goes on to explain how Buddhism practices may be the closest ways people can be truly happy.  While I appreciate the scientific evidence and thoughts he brought up, I can’t help but thinking there was a bit of a slant to it.  But of course there was!  History is always slanted with some bias.  However, I will say, no matter what your beliefs, you will learn a lot and this book will really cause you to think about your own place in the cosmos.  Two thumbs up!

 

 

Last Day at Cisco

My desk in the Cisco Lake Oswego office from May 31, 2011 to February 22, 2019. This photo taken in 2012. Doesn’t look too different 🙂

On Wednesday I attended the funeral of Brad Jackson, a beloved former Cisco colleague.  His story is not one for me to tell, but suffice to say we will all greatly miss him.  The story I can tell is the outpouring of love and support shared by the greater Cisco community and how much that support has impacted me these last 7.5 years.  When I first joined Cisco in 2011, Brad was the systems engineering manager for the local team of engineers based in the pacific northwest focused on the public sector: higher ed, school districts, and local state governments.  I worked very closely with that team during my first few years here at the company. He and his wife used to host the Christmas parties at their house.  One great tribute they said about him was: “Brad wasn’t just the life of the party, he started the party”.  I loved that quote about him.  People from Cisco flew in from Arizona, Utah, and other places to show support.  It is a testament to not only the great person Brad was but also to the great culture he helped build in the northwest.  That time period for me from 2011 to 2014 was very special.

My life and the life of my family has been greatly enriched by the people I’ve met while working at Cisco. Everyone of my managers, team mates, and extended team members down in the trenches have been unbelievable.  Cisco is the greatest company I have ever worked at.  The work/life balance, opportunities, excitement are unrivaled.

Our greatest talk we ever did!

Where am I going?

I’m now an employee at https://zenabidata.com. (And yes, one of my todos will be to fix the website).  Today the company does data analytics consulting jobs for large and small companies.  We help people turn their data into dollars.  There are about a dozen data scientists and I’ll be joining the handful of engineers working on our automated ingestion platform as well as the future products we will release.  I will be writing code full time and architecting our solutions together with some really smart people.  While I’m very sad to leave Cisco I am excited to be back to working on software design and I’m excited to be tackling AI and ML problems.

This kid brought me into this whole mess. Maybe I’ll follow him to future places? Hopefully that place is early retirement 🙂
Tom’s Captain Cloud days were perhaps some of his darkest. But we, as good friends stood by him 🙂
Talented Cloud Guru and Cartoonist
Wes and I worked closely at IBM for many years. I followed him to Cisco.
Tom and I became great friends. He now runs the whole DevNet Sandbox team and is doing amazing things!
Legends I get to work with
I had two interns Sana and Joud. Joud left last year but Sana is still at Cisco. She does great work and handled everything I threw her way!
Gabi and I went to two FC Barcelona games while we attended Cisco Live. Gabi did much better speaking than I did and got higher scores 🙂
I had some pretty amazing meals with some amazing people: Julio, Lara, Michael M., Chris, and Michael D.
Roger Dickinson is so talented! I got to work on him with tech, presentations, and strategy. Such a great guy

Book Review: Bad Blood

 

Bad Blood – Finished this pretty fast.

I finished this book in about 1.5 weeks.  I had some airplane travel time but I was riveted.  I was appalled, saddened, and shocked at the same time.  The book is a true story about a company named Theranos founded by Stanford dropout Elizabeth Holmes that appears to have falsified and misled investors, partners, and even worse its end customers:  People who were hoping to get better health care.

What made me the saddest was that Elizabeth Holmes could have been what many people, including my own three daughters, could have looked up to.  A brilliant charismatic leader on the forefront of the world of tech and health care, breaking glass, and making huge improvements in society.  We’ve seen the tech mogul story repeated with Gates, Jobs, Musk, Ellison, Bezos, etc.  But they’re all white dudes.  As a culture in tech we are craving some serious diversity and we get more lip service than real results.  Perhaps the reason Holmes was able to fool so many people was that she filled that void.  Would she had had as much success if she were a dude?  What sort of different treatment would she have received?

The book shows that her one great skill was to share a vision and get powerful people to buy into that vision.  The other thing it proved to me was how important transparency is.  I’ve seen it so many places where people don’t want transparency because it exposes people.  I first felt this when we did peer programming with my startup company Sumavi.  I was exposed for what I knew and didn’t know.  It was just one other coworker but I remember there was no time to goof around or slack off.  There was no time to pretend I knew something when I didn’t. However, transparency makes you better and makes you perform better.  What if everyone knew what you were doing all throughout the day?  Would it shame you or make you feel proud?  With Holmes we learn that with that lack of transparency nobody in the company really understood what was happening.

Ironically, what aggravated me in the book was the focus on time at the desk.  Sunny and Elizabeth both kept a tally of who worked long hours.  I’ve been a huge proponent of life balance, and Cisco has been a great company to work for on that front.  But I’ve seen so many times where time at the office does not translate to productivity.  In fact, I am betting that deep focus for 4 hours trumps any of those 12-16 hour days.

Lastly, what shocked me in the book (but I guess it shouldn’t anymore) is guess who went to jail?  Nobody.  Nobody admitted any wrongdoing, and yet Sunny and Elizabeth are as free as you and me right now.  I don’t understand this mentality.  This is the same mentality our current president of the United States has:  Never admit you did anything wrong.  I hope this trend goes away.  Elizabeth idolized Steve Jobs and tried to imitate some of his mannerisms.  Having read this book and the account of Steve Jobs in Walter Isaacson’s biography, the one thing Holmes didn’t seem to get was the passion that Jobs had.  Isaacson talks again and again on how Jobs would get emotional and cry in front of different people cause he felt so strongly about certain things.  It’s hard to know from this book what Holmes really cared about.  And to me this is one of the saddest parts of the story:  Holmes never provided any information to Carreyrou, but instead tried to destroy and threaten him.  I would really like to have more of her side of the details, but I don’t think we’ll get that.  For now, we can just take the lesson that being honest is really a good idea.

Kubernetes contexts

Today we had to work on switching between two kubernetes clusters.  The commands of the day are:

It’s surprisingly not so simple.  Perhaps there are tools out there that people know of to make it easier?  What I’ve been doing is taking to ~/.kube/config files and concatenating them together.  As an example, my config file looks as follows:

Notice how we combine them together.  Then I can switch contexts using the commands above.  I’ll have ccp or bm.

2018 – What did I do this year?

Big things of course!

  • Took the whole family to Europe for 2.5 months.  This was the greatest.  Kids got such an education on this.  So happy we got to do this.  I worked with a lot of customers and internal Cisco people while over in Europe.  It was nice to finally be on their timezone to do this!
  • Rewrote the https://kubam.io project including back end and front end.  We used this at a few banks here and there worldwide.  I did not do this alone and had lots of help.  I got to spend one week this year in Spain at the offices of one of the banks writing the code and making it work for them.  We added CentOS 7.5, RHV, ESX 6.7 and other operating system support.  I created some Ansible playbooks for this as well that we showed to a few customers.
  • Mentored several Cisco employees.  We worked in Prague for a week on building Windows support for KUBAM as well as all the backend fixes.  Really glad I got to be a part of that.
  • Kept up with our blockchain project!  I worked on this with Sana Abo helal the whole year.  We are just about finished.  Basically, its an IoT device system that allows you to subscribe to streaming feeds of other IoT devices using Ethereum.  I believe what blockchain is lacking is NOT another proprietary blockchain, but instead applications that can run on the blockchain.  Our Ashya project creates no new tokens and uses Ethereum as the way to subscribe.  We also make use of YOLO to send object recognition data to the subscribers!
  • DEVNETCoin.  This was a smart contract that I made for the Blockchain talks I do with Tom Davies.  We did this talk at Cisco Live EUR (Barcelona, Feb 2018), CodeMotion (Amsterdam, May 2018), Cisco Live US (Orlando, Jun 2018), and Cisco Live LATAM (Cancún, Dec 2018).  We are scheduled to do it again in Barcelona Jan 2019.  We are always adding things to it.  The last iteration we added the Metamask Web3JS libraries so people could buy the coins from their browsers if they had Metamask.
  • I spent a lot of time doing Kubernetes work for customers.  Mainly with the Cisco Container Platform.  Setting up a Hyperflex lab with ACI has been a lot of fun.  It’s still not 100% but there is lots to be proud of.  I wrote several articles on CCP including persistent volumes and integration with App Dynamics.  Trying to push the messaging on CCP has been a big push for me since September when I’ve really embraced it.  I’ll be delivering a 4 hour technical seminar on writing applications for on premise Kubernetes clusters in Jan 2019.  Yes of course you can come!
  • Helped Pete Johnson with the Fonk Apps Project.  This project may still be ahead of its time but I think if we work on it we’ll be able to have some interesting things for on-prem serverless activities.  I’m really pumped about writing applications this way.
  • Created a DEVNET sandbox lab on Tensorflow.  I guess we didn’t quite launch it, but we are very close!  I’ll be making sure it is up for the new year.
  • Education: Finished the Deep Learning specialization on Coursera!  Lots of work for this! Recertified the CCIE, and of course felt like I was learning something new like every day!  I worked on Minio, Kubernetes, Kubeless, Web3JS, Metamask, Keras, Tensorflow, Kafka,… feels like everyday I’m trying to figure out something new.  My ReactJS skills are better as are my python.  I didn’t do a lot of Golang this year unfortunately.  While Python is not my favorite it is still quite useful since UCS, ACI, and deep learning libs are all written in this language.

Blockchain, Kubernetes, and Tensorflow, oh my!  It’s not fair that I get to play with all the cool stuff.  I look forward to a very exciting 2019 and wish all you the best!

 

 

ESXi Kickstart and automated vCenter registration

I haven’t worked on VMware for a while but needed to work on a project to automatically install ESXi on a few servers.  I invented a tool called KUBAM that was originally for deploying Kubernetes on UCS Bare Metal (KUBAM) but have realized there are a lot of people that can benefit from the use of vMedia policy installations.  I’ve written a few articles on this method here and there.

When looking to deploy ESXi we had made the kickstart portion work perfectly and even upgraded it to 6.7.  However, when looking for information on how to automatically register the ESXi servers to vCenter after the installation was concluded the best we found was information from the legendary WIlliam Lam at VMware from a post in 2011.  Here we are 7 almost 8 years later still trying to accomplish the same thing.  The problem is the post was written in 2011 and ESXi updated the version from Python 2 to Python 3 and so parts of the script don’t work.  I’ve updated it in a dirty way to make it work and checked the code into the KUBAM project.  It could use some cleaning up to make it nice like William’s python.  I may do that as time goes on.  For now here is the code:

A few notes here:

  • urllib2 was split into different urllib packages so that is no longer included
  • The top line sets the default context to not do cert checks.  Usually I find in enterprise companies there are no certs so people just accept the cert even though there is no root authority.
  • I only have IP addresses for hostnames, but if you have DNS then you will probably want to add contents from William’s script.
  • I’m pretty lazy with this and not updating the logs.  I’ll probably go back and spend some time doing that in the future if I need to.

The code is found in the KUBAM project here.

To use this code, you can include it in your ESXi Kickstart file.  An example in the same directory is here.  Notice the key are the last lines:

We put the script (renamed vcenter.py) in the ~/kubam directory of the installer.  Then as the machine boots up it grabs the file and runs the script registering itself.

The install is nice and without glamour.  It simply adds a new server to the cluster:

With my example I didn’t add another user account but I recommend it.  I also didn’t base encode the passwords but that is something you could do as well.