Engineering @ WeaveGrid

For a little over two years now I’ve been pretty busy working for a growing company founded in San Francisco by Apoorv Bhargava and John Taggart. I was hired in December 2020 during the pandemic and one of the first remote employees outside of the bay area. We’re around 70 people now as I write…

2023 – The Road Ahead

Well, 2022 is a wrap and it was wonderful! I took a pretty big chance in 2019 when I left Cisco and joined Zenabi. Little did I know what would happen since then: * Working with some incredible people and really getting back into full time software development. We launched a few projects and serviced…

Increasing the size of persistent volumes

My coworker and I needed to increase the amount of days that Prometheus keeps logs from 15 days to around 90 days. This meant that we wanted to increase our storage, or our persistent volume. We use the kube-prometheus-stack helm chart for installing helm and have customized it to have persistent storage. It turned out…

Kubernetes Pod Anti Affinity

We have a couple of pods that get pretty intense memory use. It’s possible there is a memory leak because it OOM’d a node due to a spike in usage. One of our temporary solutions while we investigate is to put a pod anti-affinity rule in the deployment. What this means is that we don’t…

Kubernetes Cron Jobs vs. AWS Lambdas

I’ve always stressed that it should never be Kubernetes or Serverless, but rather: What is the right tool for the job. Both technologies are great and both technologies carry their own baggage. This week as I was working on some design and thought about its implementation I had to decide whether I should add a…

EKS Additional users

In our EKS Kubernetes cluster we want multiple AWS users to be able to use the kubectl command to examine resources and for now, they can even have admin access to a few select groups. The way I’ve always done this in the past is I create new stanza in the aws-auth configMap in the…

Prometheus Service Monitors

Prometheus is confusing. It’s such a great project and there is all kinds of information out there, but it’s taken me a bit of legwork to understand it. The first issue is: How are you going to install it? Since I’m running this on Kubernetes it makes sense to use whatever most people are doing….

AWS Client VPN SAML authentication with Google G-Suite

Note: Video for this Blog Post is Here. When dealing with cloud resources the two opposing needs are security and accessibility. When we often deploy resources in a private network inside of an AWS VPC that are not accessible directly from the outside. To access these resources, we can use a bastion server or VPN….

Lens and EKS

Lens is a really cool front end dashboard for Kubernetes clusters. One issue I ran into from connecting to the cluster was that I typically use different AWS Profiles for different clusters. Typically I’ll do something like: To set my user. To put this in Lens so that it can connect, we just add this…