Vallard's Blog | The Full Stack: Apps, AI, DevOps, Infrastructure

Golang: AWS SDK, Cognito and API Gateway

The situation is as follows:

Create an application with the serverless framework. This uses API Gateway, Lambda, and all kinds of cool stuff.
Authenticate on the application using Cognito.
Write a client that can call the API created by API gateway in Go.

Steps 1-2 are covered everywhere on the internet. My favorite reference is this serverless stack tutorial. It is gold and covers so much.

But step 3 is pretty poorly documented. What I found are several old libraries that do v4 signing but are no longer maintained and shouldn’t be used. Below I document a solution that works using the AWS Go SDK.

1. Cognito Setup

User Pool: I have a user pool and a corresponding App Client. The App Client for the command line Go code that I’m writing is separate from the App Client that is used by the web interface. You’ll need to note both of these:

UserPoolId: us-east-1_123456789
AppClientId: 123456789abcdefghijklmnopq

Federated Identity: In addition an identity pool which uses the user pool ID should be created. This will also give you an identification

IdentityPoolId: us-east-1:abc13813-4444-4444-4444-123456789abc

Make sure you can login and out and that this cognito stuff works. I used the serverless stack tutorial above and I could log in and out with the web interface.

2. API Gateway Setup

I’m using the serverless framework to create my stack. A few notes about it:

The Method Request should specify that it requires Auth: AWS_IAM
The Cognito Identity pool has an authenticated and unathenticated role. Make sure that role has the proper permissions to call the lambda functions.

As an example, my cognito identity pool authenticated role has the following properties:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "mobileanalytics:PutEvents",
                "cognito-sync:*",
                "cognito-identity:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "execute-api:Invoke"
            ],
            "Resource": [
                "arn:aws:execute-api:us-east-1:*:123456789/*/*/*"
            ]
        }
    ]
}

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Action": [

"mobileanalytics:PutEvents",

"cognito-sync:*",

"cognito-identity:*"

"Resource": [

"*"

]

{

"Effect": "Allow",

"Action": [

"execute-api:Invoke"

"Resource": [

"arn:aws:execute-api:us-east-1:*:123456789/*/*/*"

]

}

]

}

You would make sure the execute-api is set to your correct API Gateway ID. You can find this in the stages portion where you invoke the API:

InvokeURL: https://123456789.execute-api.us-east-1.amazonaws.com/dev

3. Golang

By leaving out the error checking and structure I’ll make this as simple as possible to do. You’ll probably want to do some caching of some sorts and of course check for errors. In the steps below we will turn our Cognito username and password into IAM credentials that assume the role of executing the API, after which we will use them invoke the API.

3.1 Create aws session

ses, _ := session.NewSession(&aws.Config{Region: aws.String("us-east-1")})

1	ses, _ := session.NewSession(&aws.Config{Region: aws.String("us-east-1")})

3.2 Authenticate user from Cognito Identity Provider

Your cognito user has a username and password. (I’m using an email). Authenticate this:

params := &cognitoidentityprovider.InitiateAuthInput{
    AuthFlow: aws.String("USER_PASSWORD_AUTH"),
    AuthParameters: map[string]*string{
      "USERNAME": aws.String("maria@vontropps.com"),
      "PASSWORD": aws.String("doremefasolatido"),
    },
    ClientId: aws.String("123456789abcdefghijklmnopq"), // this is the app client ID
  }
cip := cognitoidentityprovider.New(ses)
authResp, _ := cip.InitiateAuth(params)

params := &cognitoidentityprovider.InitiateAuthInput{

AuthFlow: aws.String("USER_PASSWORD_AUTH"),

AuthParameters: map[string]*string{

"USERNAME": aws.String("maria@vontropps.com"),

"PASSWORD": aws.String("doremefasolatido"),

ClientId: aws.String("123456789abcdefghijklmnopq"), // this is the app client ID

}

cip := cognitoidentityprovider.New(ses)

authResp, _ := cip.InitiateAuth(params)

The AuthResp will contain the IdToken, AccessToken, RefreshToken, etc. What you need is an IAM user. Notice that you’re just using the AppClientID and not the UserPoolId. I thought this was a little strange but since the AppClientId belongs to the UserPoolId I guess it works.

3.3 Get ID from Cognito Identity

This section follows (to some extent) the documentation in the Overview section of the cognito Identity API Reference. (Seriously, Amazon, a few examples would be nice) Now that we have the IdToken we can use that to get the ID of the user:

svc := cognitoidentity.New(ses)
idRes, _ := svc.GetId(&cognitoidentity.GetIdInput{
    IdentityPoolId: aws.String("us-east-1:123456789-444-4444-123456789abc"),
    Logins: map[string]*string{
      "cognito-idp.us-east-1.amazonaws.com/us-east-1_123456789": authResp.AuthenticationResult.IdToken,
    },
  })

svc := cognitoidentity.New(ses)

idRes, _ := svc.GetId(&cognitoidentity.GetIdInput{

IdentityPoolId: aws.String("us-east-1:123456789-444-4444-123456789abc"),

Logins: map[string]*string{

"cognito-idp.us-east-1.amazonaws.com/us-east-1_123456789": authResp.AuthenticationResult.IdToken,

})

This result gives us a user id which we can now get credentials:

credRes, _ := svc.GetCredentialsForIdentity(&cognitoidentity.GetCredentialsForIdentityInput{
    IdentityId: idRes.IdentityId,
    Logins: map[string]*string{
      "cognito-idp.us-east-1.amazonaws.com/us-east-1_123456789": authResp.AuthenticationResult.IdToken,
    },
  })

credRes, _ := svc.GetCredentialsForIdentity(&cognitoidentity.GetCredentialsForIdentityInput{

IdentityId: idRes.IdentityId,

Logins: map[string]*string{

"cognito-idp.us-east-1.amazonaws.com/us-east-1_123456789": authResp.AuthenticationResult.IdToken,

})

Cool, now if you look at credRes you have IAM AccessKeyId, SecretKey, SessionToken! Everything we need to now call our API.

3.4 Invoke the API Gateway with a Signed Request

To invoke the API we create a new request like we would if it were unauthenticated:

url := "https://123456789.execute-api.us-east-1.amazonaws.com/dev/list"
client := new(http.Client)
req, _ := http.NewRequest("GET", url, nil)

url := "https://123456789.execute-api.us-east-1.amazonaws.com/dev/list"

client := new(http.Client)

req, _ := http.NewRequest("GET", url, nil)

In this case I’m calling the “/list” resource and using the GET method.

Now the trick here is that we need to sign the request. With the SDK we now have a library that does that for us:

v := v4.NewSigner(credentials.NewStaticCredentials(
    *credRes.Credentials.AccessKeyId,
    *credRes.Credentials.SecretKey,
    *credRes.Credentials.SessionToken,
  ))

v.Sign(req, nil, "execute-api", "us-east-1", time.Now())

v := v4.NewSigner(credentials.NewStaticCredentials(

*credRes.Credentials.AccessKeyId,

*credRes.Credentials.SecretKey,

*credRes.Credentials.SessionToken,

))

v.Sign(req, nil, "execute-api", "us-east-1", time.Now())

Notice that we pass in the req variable. This step will add headers to the request that will authorize our request. Finally do the request:

resp, _ := client.Do(req)

1	resp, _ := client.Do(req)

Check out the response:

b, _ := ioutil.ReadAll(resp.Body)
resp.Body.Close()
fmt.Printf("%s\n", b)

b, _ := ioutil.ReadAll(resp.Body)

resp.Body.Close()

fmt.Printf("%s\n", b)

If all went well you have just sent an authenticated call to your API using the AWS SDK for Go.

4. Parting Thoughts

This is very complicated, but hopefully very secure! I’m putting this here as I saw very little documentation. One example goes a long way to drive home understanding and I’m hoping I can save someone else some time. Having never done this it took me 2 days to figure this out! There seems to be other methods of accomplishing this as well. For example, in API Gateway you can configure an authorizer that can accept just the IdToken from the Cognito User. Using that method you could skip 3.3 and just add a header instead of using the v4 library in 3.4. However, I already had this API setup for the web interface and didn’t want to change what it had.

Ingress on AWS

While looking for different options on how to run an ingress controller on AWS I found that EKS recently announced support for an open source project that creates an ALB ingress controller. One of the advantages of an ingress rule is that I can have many services be reachable via one external load balancer. In the research I found that the EKS ALB ingress controller does not have that feature. Instead for each ingress rule it creates a new ALB! That sort of defeats the purpose of what I’m looking for. As such, I’ll be going back to good old trusted NGINX. NGINX is great in that it supports TLS and I can attach it to one NLB.

There may be features NLB lacks that ALB has, but for now this is the option I will role with. I just wish someone would have told me that as it cost me about 2 hours of research.

Book Review: Sapiens A Brief History of Humankind

If you ever want to get a real perspective on the destruction our species has done to the planet and most of the other species on this earth, this is a great read. I was fascinated by the accounts of our ancestors spreading from Africa to the rest of the world. I read in horror that after several thousand years of Sapiens arriving in a new place, like Australia or North and South America how the native fauna was wiped out – and evidence that this most the cause was our own species

large mega fauna from north america which evidence suggests was the result of the actions of humans.

I loved learning about the giant Megatherium – A giant sloth

Or the large Marsupial mega fauna as well as the Thylacine. Equally fascinating to me was to realize how Sapiens lived contemporary with a number of other human species like the Neanderthals and others.

These other species Harari gives evidence that they were probably also casualties of the expansion of Sapiens.

The last part that made me completely sad was the chapter on “Life on the conveyer belt” or the life of domesticated animals such as cows, pigs, and chickens that we have mass produced and consumed making these animals completely miserable in the process. This is another good reason to make me want to be a vegan. While everyone has a problem with how the sausage gets made people still eat the sausage. (I had some tonight in fact!) However, this part has influenced me to want to live a more plant based diet.

By midway through the book I was quite sad about the human race. It seemed very tragic and unnecessary. But then I felt there was a period in the book, after all the wars to the present time to where I started feeling good. Harari mentioned how low the chance of war is right now in many parts of the world. Comparing life now and 500 years in the past things are quite good. I started realizing that many humans are realizing the extent of the damage and are hoping to take responsibility. For the first time in all these millennia it looked as if humans were starting to try to do something good.

But then just as you were starting to feel good about all the great things that are happening because of the age of science and admitting we don’t have all the answers you get lambasted with all the crazy things that are about to happen: How our species will eventually be extinct because of our own doing. Not because we’ll blow ourselves up (although that is a possibility he doesn’t deny), but because we will transform our own species through scientific discovers. These bring up new issues:

If we as a species are able to become a mortals by curing nearly all diseases then who gets to decide who gets to live forever? Harari mentions that for years the consolation of the poor has been that the rich have the same 24 hours every day and their time is limited the same as any man. How unfair it is that they would now get to live longer. (As if anything is fair in life or even should be)
If we are a species are able to enhance our minds to become super humans, who gets to have this procedure done? Do we start making different classes of humans with the un-enhanced humans eventually being made extinct over time?
What about when we are able to read each others mind and share a common brain, and download the memories of one person into our own brain? Is that person us or are we still an individual?

The future from this perspective looks wild and perhaps a bit scary to live in.

Finally the message I get from the book is that everything is temporary for us humans and that meditation and trying to free ourselves from wants seems to be the only way to be happy. The author goes on to explain how Buddhism practices may be the closest ways people can be truly happy. While I appreciate the scientific evidence and thoughts he brought up, I can’t help but thinking there was a bit of a slant to it. But of course there was! History is always slanted with some bias. However, I will say, no matter what your beliefs, you will learn a lot and this book will really cause you to think about your own place in the cosmos. Two thumbs up!

Last Day at Cisco

On Wednesday I attended the funeral of Brad Jackson, a beloved former Cisco colleague. His story is not one for me to tell, but suffice to say we will all greatly miss him. The story I can tell is the outpouring of love and support shared by the greater Cisco community and how much that support has impacted me these last 7.5 years. When I first joined Cisco in 2011, Brad was the systems engineering manager for the local team of engineers based in the pacific northwest focused on the public sector: higher ed, school districts, and local state governments. I worked very closely with that team during my first few years here at the company. He and his wife used to host the Christmas parties at their house. One great tribute they said about him was: “Brad wasn’t just the life of the party, he started the party”. I loved that quote about him. People from Cisco flew in from Arizona, Utah, and other places to show support. It is a testament to not only the great person Brad was but also to the great culture he helped build in the northwest. That time period for me from 2011 to 2014 was very special.

My life and the life of my family has been greatly enriched by the people I’ve met while working at Cisco. Everyone of my managers, team mates, and extended team members down in the trenches have been unbelievable. Cisco is the greatest company I have ever worked at. The work/life balance, opportunities, excitement are unrivaled.

Where am I going?

I’m now an employee at https://zenabidata.com. (And yes, one of my todos will be to fix the website). Today the company does data analytics consulting jobs for large and small companies. We help people turn their data into dollars. There are about a dozen data scientists and I’ll be joining the handful of engineers working on our automated ingestion platform as well as the future products we will release. I will be writing code full time and architecting our solutions together with some really smart people. While I’m very sad to leave Cisco I am excited to be back to working on software design and I’m excited to be tackling AI and ML problems.

Tom and I became great friends. He now runs the whole DevNet Sandbox team and is doing amazing things!

Book Review: Bad Blood

I finished this book in about 1.5 weeks. I had some airplane travel time but I was riveted. I was appalled, saddened, and shocked at the same time. The book is a true story about a company named Theranos founded by Stanford dropout Elizabeth Holmes that appears to have falsified and misled investors, partners, and even worse its end customers: People who were hoping to get better health care.

What made me the saddest was that Elizabeth Holmes could have been what many people, including my own three daughters, could have looked up to. A brilliant charismatic leader on the forefront of the world of tech and health care, breaking glass, and making huge improvements in society. We’ve seen the tech mogul story repeated with Gates, Jobs, Musk, Ellison, Bezos, etc. But they’re all white dudes. As a culture in tech we are craving some serious diversity and we get more lip service than real results. Perhaps the reason Holmes was able to fool so many people was that she filled that void. Would she had had as much success if she were a dude? What sort of different treatment would she have received?

The book shows that her one great skill was to share a vision and get powerful people to buy into that vision. The other thing it proved to me was how important transparency is. I’ve seen it so many places where people don’t want transparency because it exposes people. I first felt this when we did peer programming with my startup company Sumavi. I was exposed for what I knew and didn’t know. It was just one other coworker but I remember there was no time to goof around or slack off. There was no time to pretend I knew something when I didn’t. However, transparency makes you better and makes you perform better. What if everyone knew what you were doing all throughout the day? Would it shame you or make you feel proud? With Holmes we learn that with that lack of transparency nobody in the company really understood what was happening.

Ironically, what aggravated me in the book was the focus on time at the desk. Sunny and Elizabeth both kept a tally of who worked long hours. I’ve been a huge proponent of life balance, and Cisco has been a great company to work for on that front. But I’ve seen so many times where time at the office does not translate to productivity. In fact, I am betting that deep focus for 4 hours trumps any of those 12-16 hour days.

Lastly, what shocked me in the book (but I guess it shouldn’t anymore) is guess who went to jail? Nobody. Nobody admitted any wrongdoing, and yet Sunny and Elizabeth are as free as you and me right now. I don’t understand this mentality. This is the same mentality our current president of the United States has: Never admit you did anything wrong. I hope this trend goes away. Elizabeth idolized Steve Jobs and tried to imitate some of his mannerisms. Having read this book and the account of Steve Jobs in Walter Isaacson’s biography, the one thing Holmes didn’t seem to get was the passion that Jobs had. Isaacson talks again and again on how Jobs would get emotional and cry in front of different people cause he felt so strongly about certain things. It’s hard to know from this book what Holmes really cared about. And to me this is one of the saddest parts of the story: Holmes never provided any information to Carreyrou, but instead tried to destroy and threaten him. I would really like to have more of her side of the details, but I don’t think we’ll get that. For now, we can just take the lesson that being honest is really a good idea.

Kubernetes contexts

Today we had to work on switching between two kubernetes clusters. The commands of the day are:

kubectl config get-contexts
kubectl config use-context <context>

1 2	kubectl config get-contexts kubectl config use-context <context>

It’s surprisingly not so simple. Perhaps there are tools out there that people know of to make it easier? What I’ve been doing is taking to ~/.kube/config files and concatenating them together. As an example, my config file looks as follows:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: blahblah... 
    server: https://172.28.225.133:6443
  name: bm
- cluster:
    certificate-authority-data: blahblah... 
    server: https://10.93.140.127:6443
  name: ccp

contexts:
- context:
    cluster: bm
    user: kubernetes-admin
  name: bm
- context:
    cluster: ccp
    user: kubernetes-admin1
  name: ccp
current-context: ccp
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: blahblah...
    client-key-data: blahblah...
    token: blahblah...
- name: kubernetes-admin1
  user:
    client-certificate-data: blahblah...
    client-key-data: blahblah...
    token: blahblah...

apiVersion: v1

clusters:

- cluster:

certificate-authority-data: blahblah...

server: https://172.28.225.133:6443

- cluster:

certificate-authority-data: blahblah...

server: https://10.93.140.127:6443

contexts:

- context:

cluster: bm

user: kubernetes-admin

- context:

cluster: ccp

user: kubernetes-admin1

current-context: ccp

kind: Config

preferences: {}

users:

- name: kubernetes-admin

user:

client-certificate-data: blahblah...

client-key-data: blahblah...

token: blahblah...

- name: kubernetes-admin1

user:

client-certificate-data: blahblah...

client-key-data: blahblah...

token: blahblah...

Notice how we combine them together. Then I can switch contexts using the commands above. I’ll have ccp or bm.

2018 – What did I do this year?

Big things of course!

Took the whole family to Europe for 2.5 months. This was the greatest. Kids got such an education on this. So happy we got to do this. I worked with a lot of customers and internal Cisco people while over in Europe. It was nice to finally be on their timezone to do this!
Rewrote the https://kubam.io project including back end and front end. We used this at a few banks here and there worldwide. I did not do this alone and had lots of help. I got to spend one week this year in Spain at the offices of one of the banks writing the code and making it work for them. We added CentOS 7.5, RHV, ESX 6.7 and other operating system support. I created some Ansible playbooks for this as well that we showed to a few customers.
Mentored several Cisco employees. We worked in Prague for a week on building Windows support for KUBAM as well as all the backend fixes. Really glad I got to be a part of that.
Kept up with our blockchain project! I worked on this with Sana Abo helal the whole year. We are just about finished. Basically, its an IoT device system that allows you to subscribe to streaming feeds of other IoT devices using Ethereum. I believe what blockchain is lacking is NOT another proprietary blockchain, but instead applications that can run on the blockchain. Our Ashya project creates no new tokens and uses Ethereum as the way to subscribe. We also make use of YOLO to send object recognition data to the subscribers!
DEVNETCoin. This was a smart contract that I made for the Blockchain talks I do with Tom Davies. We did this talk at Cisco Live EUR (Barcelona, Feb 2018), CodeMotion (Amsterdam, May 2018), Cisco Live US (Orlando, Jun 2018), and Cisco Live LATAM (Cancún, Dec 2018). We are scheduled to do it again in Barcelona Jan 2019. We are always adding things to it. The last iteration we added the Metamask Web3JS libraries so people could buy the coins from their browsers if they had Metamask.
I spent a lot of time doing Kubernetes work for customers. Mainly with the Cisco Container Platform. Setting up a Hyperflex lab with ACI has been a lot of fun. It’s still not 100% but there is lots to be proud of. I wrote several articles on CCP including persistent volumes and integration with App Dynamics. Trying to push the messaging on CCP has been a big push for me since September when I’ve really embraced it. I’ll be delivering a 4 hour technical seminar on writing applications for on premise Kubernetes clusters in Jan 2019. Yes of course you can come!
Helped Pete Johnson with the Fonk Apps Project. This project may still be ahead of its time but I think if we work on it we’ll be able to have some interesting things for on-prem serverless activities. I’m really pumped about writing applications this way.
Created a DEVNET sandbox lab on Tensorflow. I guess we didn’t quite launch it, but we are very close! I’ll be making sure it is up for the new year.
Education: Finished the Deep Learning specialization on Coursera! Lots of work for this! Recertified the CCIE, and of course felt like I was learning something new like every day! I worked on Minio, Kubernetes, Kubeless, Web3JS, Metamask, Keras, Tensorflow, Kafka,… feels like everyday I’m trying to figure out something new. My ReactJS skills are better as are my python. I didn’t do a lot of Golang this year unfortunately. While Python is not my favorite it is still quite useful since UCS, ACI, and deep learning libs are all written in this language.

Blockchain, Kubernetes, and Tensorflow, oh my! It’s not fair that I get to play with all the cool stuff. I look forward to a very exciting 2019 and wish all you the best!

ESXi Kickstart and automated vCenter registration

I haven’t worked on VMware for a while but needed to work on a project to automatically install ESXi on a few servers. I invented a tool called KUBAM that was originally for deploying Kubernetes on UCS Bare Metal (KUBAM) but have realized there are a lot of people that can benefit from the use of vMedia policy installations. I’ve written a few articles on this method here and there.

When looking to deploy ESXi we had made the kickstart portion work perfectly and even upgraded it to 6.7. However, when looking for information on how to automatically register the ESXi servers to vCenter after the installation was concluded the best we found was information from the legendary WIlliam Lam at VMware from a post in 2011. Here we are 7 almost 8 years later still trying to accomplish the same thing. The problem is the post was written in 2011 and ESXi updated the version from Python 2 to Python 3 and so parts of the script don’t work. I’ve updated it in a dirty way to make it work and checked the code into the KUBAM project. It could use some cleaning up to make it nice like William’s python. I may do that as time goes on. For now here is the code:

import sys,re,os,urllib,base64,syslog,socket
import urllib.request as request
import ssl

# used for unverified SSL/TLS certs. Insecure, not good for public.
ssl._create_default_https_context = ssl._create_unverified_context

## Variables that need to be filled out for different environments ##
# URL of vcenter, just the IP address
vcenter = "10.93.140.100"

# Data center
#cluster = "<datacenter>/host/<cluster>"
cluster = "Lucky Lab (LK02)/host/SandyBridge"

# vCenter credentials
user = "administrator@vsphere.local"
password = "Cisco.123"

# host ESXi credentials
host_username = "root"
host_password = "Cisco.123"

url = "https://" + vcenter + "/mob/?moid=SearchIndex&method=findByInventoryPath"


passman = request.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None,url, user, password)
authhandler = request.HTTPBasicAuthHandler(passman)
opener = request.build_opener(authhandler)
request.install_opener(opener)
#cont = ssl._create_unverified_context()
#ssl._create_default_https_context = ssl._create_unverified_context()
req = request.Request(url)
#cont = ssl._create_unverified_context()
#page = request.urlopen(req, context=context)
page = request.urlopen(req)
page_content = page.read().decode('utf-8')
#print(page_content)

reg = re.compile('name="vmware-session-nonce" type="hidden" value="?([^\s^"]+)"')
nonce = reg.search(page_content).group(1)

headers = page.info()
cookie = headers.get("Set-Cookie")

params = {'vmware-session-nonce':nonce,'inventoryPath':cluster}
e_params = urllib.parse.urlencode(params)
#print(e_params)
bin_data = e_params.encode('utf8')
req = request.Request(url, bin_data, headers={"Cookie":cookie})
page = request.urlopen(req)
page_content = page.read().decode('utf-8')
#print(page_content)

clusterMoRef = re.search('domain-c[0-9]*',page_content)
if clusterMoRef:
	print("Found cluster: " + cluster)
else:
	opener.close()
	sys.exit(1)

# cert stuff
cmd = "openssl x509 -sha1 -in /etc/vmware/ssl/rui.crt -noout -fingerprint"
tmp = os.popen(cmd)
tmp_sha1 = tmp.readline()
tmp.close()
s1 = re.split('=',tmp_sha1)
s2 = s1[1]
s3 = re.split('\n', s2)
sha1 = s3[0]

if sha1:
	print("Hash: ", sha1)
else:
	opener.close()
	sys.exit(1)

xml = '<spec xsi:type="HostConnectSpec"><hostName>%hostname</hostName><sslThumbprint>%sha</sslThumbprint><userName>%user</userName><password>%pass</password><force>1</force></spec>'
 
# Code to extract IP Address to perform DNS lookup to add FQDN to vCenter
hostip = socket.gethostbyname(socket.gethostname())

if hostip:
	print("IP address of host: ", hostip.strip())
else:
	opener.close()
	sys.exit(1)

# could add logic to do DNS lookup, but no dns in our environment. 

xml = xml.replace("%hostname",hostip)
xml = xml.replace("%sha",sha1)
xml = xml.replace("%user",host_username)
xml = xml.replace("%pass",host_password)
print(xml)
# now join to vcenter cluster
try: 
	url = "https://" + vcenter + "/mob/?moid=" + clusterMoRef.group() + "&method=addHost"
	params = {'vmware-session-nonce':nonce,'spec':xml,'asConnected':'1','resourcePool':'','license':''}
	e_params = urllib.parse.urlencode(params)
	bin_data = e_params.encode('utf8')
	req = request.Request(url, bin_data, headers={"Cookie":cookie})
	page = request.urlopen(req).read()
except IOError as e:
	opener.close()
	print("Couldn't join cluster", e)
	sys.exit(1)
else:
	print("Joined vcenter cluster!")
	url = "https://" + vcenter + "/mob/?moid=SessionManager&method=logout"
	params = {'vmware-session-nonce':nonce}
	e_params = urllib.parse.urlencode(params)
	bin_data = e_params.encode('utf8')
	req = request.Request(url, bin_data, headers={"Cookie":cookie})
	page = request.urlopen(req).read()
	sys.exit(0)

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

import sys,re,os,urllib,base64,syslog,socket

import urllib.request as request

import ssl

# used for unverified SSL/TLS certs. Insecure, not good for public.

ssl._create_default_https_context = ssl._create_unverified_context

## Variables that need to be filled out for different environments ##

# URL of vcenter, just the IP address

vcenter = "10.93.140.100"

# Data center

#cluster = "<datacenter>/host/<cluster>"

cluster = "Lucky Lab (LK02)/host/SandyBridge"

# vCenter credentials

user = "administrator@vsphere.local"

password = "Cisco.123"

# host ESXi credentials

host_username = "root"

host_password = "Cisco.123"

url = "https://" + vcenter + "/mob/?moid=SearchIndex&method=findByInventoryPath"

passman = request.HTTPPasswordMgrWithDefaultRealm()

passman.add_password(None,url, user, password)

authhandler = request.HTTPBasicAuthHandler(passman)

opener = request.build_opener(authhandler)

request.install_opener(opener)

#cont = ssl._create_unverified_context()

#ssl._create_default_https_context = ssl._create_unverified_context()

req = request.Request(url)

#cont = ssl._create_unverified_context()

#page = request.urlopen(req, context=context)

page = request.urlopen(req)

page_content = page.read().decode('utf-8')

#print(page_content)

reg = re.compile('name="vmware-session-nonce" type="hidden" value="?([^\s^"]+)"')

nonce = reg.search(page_content).group(1)

headers = page.info()

cookie = headers.get("Set-Cookie")

params = {'vmware-session-nonce':nonce,'inventoryPath':cluster}

e_params = urllib.parse.urlencode(params)

#print(e_params)

bin_data = e_params.encode('utf8')

req = request.Request(url, bin_data, headers={"Cookie":cookie})

page = request.urlopen(req)

page_content = page.read().decode('utf-8')

#print(page_content)

clusterMoRef = re.search('domain-c[0-9]*',page_content)

if clusterMoRef:

print("Found cluster: " + cluster)

else:

opener.close()

sys.exit(1)

# cert stuff

cmd = "openssl x509 -sha1 -in /etc/vmware/ssl/rui.crt -noout -fingerprint"

tmp = os.popen(cmd)

tmp_sha1 = tmp.readline()

tmp.close()

s1 = re.split('=',tmp_sha1)

s2 = s1[1]

s3 = re.split('\n', s2)

sha1 = s3[0]

if sha1:

print("Hash: ", sha1)

else:

opener.close()

sys.exit(1)

xml = '<spec xsi:type="HostConnectSpec"><hostName>%hostname</hostName><sslThumbprint>%sha</sslThumbprint><userName>%user</userName><password>%pass</password><force>1</force></spec>'

# Code to extract IP Address to perform DNS lookup to add FQDN to vCenter

hostip = socket.gethostbyname(socket.gethostname())

if hostip:

print("IP address of host: ", hostip.strip())

else:

opener.close()

sys.exit(1)

# could add logic to do DNS lookup, but no dns in our environment.

xml = xml.replace("%hostname",hostip)

xml = xml.replace("%sha",sha1)

xml = xml.replace("%user",host_username)

xml = xml.replace("%pass",host_password)

print(xml)

# now join to vcenter cluster

try:

url = "https://" + vcenter + "/mob/?moid=" + clusterMoRef.group() + "&method=addHost"

params = {'vmware-session-nonce':nonce,'spec':xml,'asConnected':'1','resourcePool':'','license':''}

e_params = urllib.parse.urlencode(params)

bin_data = e_params.encode('utf8')

req = request.Request(url, bin_data, headers={"Cookie":cookie})

page = request.urlopen(req).read()

except IOError as e:

opener.close()

print("Couldn't join cluster", e)

sys.exit(1)

else:

print("Joined vcenter cluster!")

url = "https://" + vcenter + "/mob/?moid=SessionManager&method=logout"

params = {'vmware-session-nonce':nonce}

e_params = urllib.parse.urlencode(params)

bin_data = e_params.encode('utf8')

req = request.Request(url, bin_data, headers={"Cookie":cookie})

page = request.urlopen(req).read()

sys.exit(0)

A few notes here:

urllib2 was split into different urllib packages so that is no longer included
The top line sets the default context to not do cert checks. Usually I find in enterprise companies there are no certs so people just accept the cert even though there is no root authority.
I only have IP addresses for hostnames, but if you have DNS then you will probably want to add contents from William’s script.
I’m pretty lazy with this and not updating the logs. I’ll probably go back and spend some time doing that in the future if I need to.

The code is found in the KUBAM project here.

To use this code, you can include it in your ESXi Kickstart file. An example in the same directory is here. Notice the key are the last lines:

# register with vcenter
esxcli network firewall ruleset set -e true -r httpClient
wget -O vcenter.py http://10.93.140.118/kubam/vcenter.py 
/bin/python vcenter.py

# register with vcenter

esxcli network firewall ruleset set -e true -r httpClient

wget -O vcenter.py http://10.93.140.118/kubam/vcenter.py

/bin/python vcenter.py

We put the script (renamed vcenter.py) in the ~/kubam directory of the installer. Then as the machine boots up it grabs the file and runs the script registering itself.

The install is nice and without glamour. It simply adds a new server to the cluster:

With my example I didn’t add another user account but I recommend it. I also didn’t base encode the passwords but that is something you could do as well.

A brief history of on-prem serverless development

My colleague Pete Johnson, at Cisco has released a blog about a project called FONK. I wanted to talk a little bit about why its important.

iOS Developers

Let’s first start back with the dawn of mobile application development. Pretend you are an iPhone developer back in 2008. You know Objective-C and you know how to make killer user interfaces and you make a fun game about dwarfs hunting butterflies. You release the game, the app is good, and you are happy.

As the game gets bigger you realize you’d like to add features to make it so certain parts of the game are stored in the cloud. For example, you want to keep the all time highest score for all players who have ever played the game with their avatar name. Bragging rights are cool, and you want your game to be cool.

The problem is, you don’t have any expertise in this. You know how to write Objective-C but you don’t know how to manage servers. In fact, you have probably never even installed an operating system in your life. So even though someone may be really good at spinning up VMs and putting NGINX on it and running some back end ruby on rails code, that is more hassle than you want. All you want is a backend that you don’t have to manage where the game can upload and retrieve all time highest player scores and display it.

To do this, you know you want a place in the cloud that accepts a JSON POST request to send the highest score and you want to be able to put a GET request to get the highest scores. If you don’t know what POST or GET are, they are basic HTTP request methods. Read more here. You don’t want to manage servers, virtual machines, or even containers. (Also, containers don’t really get popular until 2014 so we are still 6 years away from that).

Backend-as-a-Service

What you want is a service. And in 2008 if we wait a little bit until 2011 then we start to see two cool solutions emerge for us: Firebase and Parse. Firebase was bought by Google and still lives on but Parse was bought by Facebook and closed down. Both of these two companies offered a Backend-As-A-Service for your applications. Really cool. Now you could just use their GUI and put in database calls and then call it with your mobile app. You didn’t have to learn about managing VMs and all that. It was great!

The other thing that people start to realize is that it is good to accompany a mobile app with a web page. To make it consistent, it would be great if the webpage would call the same APIs as the mobile app. That way you only need to maintain one backend that both the web and mobile app can call. Great! How to do that? Remember we still don’t want to manage VMs, OSes, or even the stacks that run that. Well it turns out that AWS has offered us static web page hosting on S3.

Static web page hosting on S3 doesn’t sound like something that would call APIs. But let’s understand what it is. Our dwarf game is now hiring a web developer who knows Javascript. When you visit a website that has javascript, the javascript doesn’t run on that site. Javascript runs on your browser. What happens is your browser goes to the URL and the URL gives you static HTML files. Some of those files are javascript, css, and html that execute on your browser. So we can actually host this in a public S3 bucket, point people to go there and they download the code. You get replication, durability, high availability and uptime for free! Still no VMs, no containers, no operating systems.

Enter AWS

It is now 2014 and you are a smart person at AWS. You see this Backend as a service trend start to come and realize you could make some more money. You also notice that some of these backend as a service companies also call your own products like your databases. You’re also worried about the threat of Google and Facebook entering the market and taking more of your customers. What do you do? You build a backend as a service by stitching together your own services. The advantage you have at AWS: You already have some well known database services: RDS, DynamoDB, etc. You also own S3 and your customers are already using it for static pages. You also own API Gateway which allows you to call S3 services on the backend but it needs something a little more powerful. So how can you complete the picture? You introduce AWS Lambda.

In December 2014 AWS introduced AWS Lambda and many people were puzzled by the idea of function as a service. What good was it for? Why would you want to call a function based on an event? If you keep in mind the entire architecture then you realize that AWS Lambda just completes the picture. No VMs, no containers, no operating systems. No expertise needed in running this. Just put code in their GUI and off you go. Since you already use S3 you can easily put your functions in AWS and you are golden.

Let’s realize that this way of running applications is not the right way for every application. Remember our dwarf game is pretty simple. We just want to store the high score and get and receive it. We may chose to add user identities and logins later as well, and our app can handle that. But at some point if the complexity gets too much on the back end we’ll have to hire a backend engineer. But for now we can still run it all with javascript and put our backend on AWS lambda, API Gateway, and DynamoDB.

Kubernetes

In 2014 something happened that probably wasn’t meant to. Google introduced Kubernetes to the world and that was intentional. After all, nobody was using their cloud and they wanted to make people realize that they could use containers on their cloud better than anyone else. By open sourcing Kubernetes Google could make a splash and assert some dominance. That was intentional. But what wasn’t intentional was that when 2015 hit and Kubernetes 1.0 was released it actually started adding some parity between public clouds and private clouds.

Let me explain. To have an AWS experience on Prem the best thing you could do was OpenStack or some other do it yourself project. OpenStack was pretty complicated, to put it nicely. There were very few orgs that successfully implemented OpenStack on prem and so many people kept migrating to the public cloud. Kubernetes has been so impactful that companies that have tried to resist or compete against it have thrown in the towel and embraced it. Check out the list:

Google: Offered GKE, the initial cloud based kubernetes platform
RedHat (now IBM): Completely threw away OpenShift PaaS code and built entirely on Kubernetes
Pivotal: Had their own solution but now pushes PKS
Azure: AKS Kubernetes offering
Amazon: AWS who resisted and offered several different ways to manage containers (ECS, faregate, etc) finally offered native Kubernetes with EKS.

Many of these organizations probably would have rather liked to own the solution but everyone is now a Kubernetes provider! The way of the PaaS is dying. They are morphing into opinionated Kubernetes services.

But the biggest impact of all, to me, was that it started giving parity between public clouds and private clouds. The same Kubernetes platform that runs on AWS, GCP, Azure, PKS, could be run on your own data center on your own bare metal servers. You just leveled the playing field and made it easier for me to bring my apps back on prem. (If I wanted to, but of course why would I?) Well for the dwarf game developer, I have no need to go back on prem. But if I’m a big enterprise with lots of space in my datacenter, this might look interesting.

Kubernetes is simple to install, and much easier to manage than OpenStack, but still has its challenges. However, as enterprises mature, perhaps it won’t be so bad? After all there are solutions now from vendors including Cisco that offer Kubernetes with enterprise support.

FaaS, Object Storage, NoSQL, Kubernetes

Back to the mobile app world. When AWS lambda came out some people made a framework called serverless (check out serverless.com). People thought this whole function as a service thing was pretty rad. But now what makes it radder is we have a solution to run it on our own data centers using just Kubernetes.

I worked with Pete a little on his FONK project by submitting some code samples for the Guestbook app running on Kubeless. For a developer to not have to worry about creating Dockerfiles, Kubernetes YAML files, and being able to just write code and get it working it is very appealing. This serverless business is still pretty cutting edge and even though we are 3 years from when AWS came out with Lambda, there is still a lot of buzz about it and people jumping into the space. What I like about FONK is that it levels the playing field between what I can get on public clouds and private clouds. Certainly many people would argue that using a private cloud is the only way to go, but I see great dangers in this.

I am not comfortable with a company as big as Amazon making even more money off of me and holding me hostage. I know they are good. I’ve used their stuff. I just don’t
At a certain scale it is more cost effective to run kubernetes on prem than running in any public cloud. Granted you must have the expertise
I can get better performance on my own datacenter. Sure if I have to burst, public cloud is great. But for chugging apps, I like my on prem stuff.

You can entirely disagree with me and you could be right. But just remember, in engineering it is always about tradeoffs. It’s not wrong or right. It’s what tradeoffs do you want.

Technical Details on Gas on Ethereum

I’ve been working over the last year off and on on different smart contracts. What follows are some technical details I sent to some of my colleagues here at Cisco regarding gas price and gas limits. Two important parts of Ethereum that are not understood very well: Units and Gas.

Units

1 Ether in Ethereum is divisible up to 18 decimal units. The smallest unit is called a wei. Thus, 1 wei followed by 18 zeros equals 1 ether. So:

1 ether = 1,000,000,000,000,000,000 wei.

1	1 ether = 1,000,000,000,000,000,000 wei.

The other important unit to know is a gwei. A gwei is halfway between 1 ether and 1 wei.

1 gwei = 1,000,000,000 wei
1 ether = 1,000,000,000 gwei

1 2	1 gwei = 1,000,000,000 wei 1 ether = 1,000,000,000 gwei

In solidity everything is stored in wei values. There are no fractions nor decimals, we just work with integers. So by having this long zero values provide a way to have decimals without having decimals.

There are several functions I’ve used in some recent test cases that use this conversion as it is much easier. Here are some examples:

result = await devnetcoin.buyDEV({from: hank,
                        value: web3.toWei(1, "ether"),
                        gas: 508460,
                        gasPrice: web3.toWei('20', "gwei")})

result = await devnetcoin.buyDEV({from: hank,

value: web3.toWei(1, "ether"),

gas: 508460,

gasPrice: web3.toWei('20', "gwei")})

web3.fromWei(web3.eth.getBalance(val).toString(), 'ether').toString()

1	web3.fromWei(web3.eth.getBalance(val).toString(), 'ether').toString()

Essentially, the fromWei function lets you format a value from wei to something else, like ether. The toWei let’s you convert from ether to wei without having to write out all the zeros. In Ethereum we have this whole other BigNumber class to deal with. BigNumber is the class that allows us to store these giant 18 digit numbers as well as manipulate them. It’s important to understand the units before understanding how gas works.

Gas

Gas is a unit or number of steps to be execute for your contract. Gas has fixed number of units for a specific operation/computation, this is fixed by Ethereum community. For example to add two numbers EVM consumes it may consumes 3 gas units. You set a Gas Limit in your transaction to tell how many steps you are willing to take. The unit of gas is not in Ether, Gwei, or Wei. It’s just an integer representing a number of steps.

To get basic fees to get some idea of how much gas, note the following:

32k gas to create a contract.
21k gas for normal transaction
200 gas * 1 byte of byte code.
Transaction data costs 64 for non-zero bytes and 4 for zero bytes
Constructor cost

See this response on Ethereum Stack Exchange.

The limit of computation per block is not constant. Miners can set this.

VALUE field – The amount of wei to transfer from the sender to the recipient, this is the value we are going to send to either another user or to a contract.

GASPRICE value, representing the fee the sender is willing to pay for gas. One unit of gas corresponds to the execution of one atomic instruction, Gasprice is in wei. Wei is a smallest unit. (1 wei = 10^-18 ether or 10^18 wei = 1 ether. Because it is quite high, we use gwei. 1 gwei = 1,000,000,000 wei.

Total cost of transaction is Gas Limit * Gas Price

Gas price fluctuates. During normal times:

40 GWEI will always get you in next block
20 GWEI will get you in the next few blocks
2 GWEI will get you in the next few minutes.

Depends on what minors are willing to take.

1 ether = 1,000,000,000 gwei

1	1 ether = 1,000,000,000 gwei

To see the current price for a unit of gas you can run:

web3.eth.gasPrice

1	web3.eth.gasPrice

The new function in web3.js 1.0.0 (still beta as of this writing) requires a callback:

web3.eth.getGasPrice(console.log)

1	web3.eth.getGasPrice(console.log)

(the callbacks are always error, response in 1.0.0 so in the new stuff you’ll get two values printed in the console log)

This will be about 20 Gwei. Or you can look to see what the Gas price is at the ETH Gas Station. The current price shown at the ETH Gas station is 11 Gwei (std) and 9 Gwei (safe low). Creating contracts seem to cost a little more.

In practice when we’ve created contracts we’ve seen that we can use the estimateGas function to tell how much for storing the bytes:

this.state.provider.eth.estimateGas({ data: contract.bytecode }, somecallbackfunction)

1	this.state.provider.eth.estimateGas({ data: contract.bytecode }, somecallbackfunction)

The callback function for the contracts we’ve been working on shows the gas limit (or amount of gas) is about 645234. But this is only the cost of storing the contract code on the blockchain. You need to add more gas limit for creating the contract, for passing parameters, etc. We’ve found that by adding 80000 more to our gas Limit the contract goes through.