Allowing Applications in AWS EKS to access other AWS services

I have an application that runs in a container. The application is pretty simple in that all it does is list the contents of a DynamoDB database I created. The code is here, as well as all the YAML files for deploying to EKS. The problem is, when we pull this up on a standard EKS cluster we get the following issue:

I’ve purposely allowed my application to display the AWS errors to the main screen so I could see that there is a permission issue. The issue here is that the container is not authorized to access DynamoDB. So we get IAM permission errors.

Quick and Easy Fix

The easy fix is to allow all the nodes in the cluster to access the DynamoDB resource. When eksctl creates the managed nodes it also created an IAM role that is displayed in the error message:

User: arn:aws:sts::188966951897:assumed-role/eksctl-dec08-nodegroup-standard-w-NodeInstanceRole-1P18ZVF8HUQKY/i-0d8b78cc00172b721 is not authorized to perform: dynamodb:Scan on resource: arn:aws:dynamodb:us-west-2:188966951897:table/dynamoUsers

So we can go into IAM and add a new policy and then apply it to the eksctl-dec08-nodegroup-standard-w-NodeInstanceRole-... role. This looks as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "dynamodb:Scan",
            "Resource": "arn:aws:dynamodb:us-west-2:188966951897:table/dynamoUsers"
        }
    ]
}

The problem with this approach is it now means that any container in the network can now access this database. We may decide that we only want certain pods to have the ability to access the database, and not leave it wide open.

More Granularity: Pod Service Accounts

Let’s remove that policy from the node group and try a more secure way. Amazon supports IAM roles for service accounts that can help us with this.

1. Add OpenID permissions to your EKS user

Back in IAM add the permission to the user (or group) to iam:CreateOpenIDConnectProvider. This can be done as an inline policy or you could create a new policy. You may also want to give them the permission to delete as well with iam:DeleteOpenIDConnectProvider. I’ve added it into an existing IAM policy, but at minimum it should look as follows:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [

        "iam:CreateOpenIDConnectProvider",
  
      ],
      "Resources" [
         "arn:aws:iam::<your account>:oidc-provider/oidc.eks.<yourregion>.amazonaws.com",
         "arn:aws:iam::<your account>:oidc-provider/oidc.eks.<yourregion>.amazonaws.com/*"
    }
  ]
}

In other words, the user that created the EKS cluster should also have permissions to create an OpenIDConnectProvider on the cluster. We accomplish this by adding the policy above.

2. Create OIDC Provider for the Cluster

Once the user has permission to do this run the following eksctl command to create the provider. You only need to do this once on the cluster.

$ eksctl utils associate-iam-oidc-provider --cluster dec08 --approve
[ℹ]  eksctl version 0.32.0
[ℹ]  using region us-west-2
[ℹ]  will create IAM Open ID Connect provider for cluster "dec08" in "us-west-2"
[✔]  created IAM Open ID Connect provider for cluster "dec08" in "us-west-2"

3. Create Policy for Application

Next, decide what AWS resources your application will require access to and what the least amount of permissions they need. In my case, my application only needs to Scan a particular DynamoDB table. So I lock it down very tight and create a policy that allows that. You can call the policy something like the applications name. In mine, since its EKS accessing the DynamoDB Users table I called it EKSDynamoUsers:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "dynamodb:Scan",
            "Resource": "arn:aws:dynamodb:us-west-2:188966951897:table/dynamoUsers"
        }
    ]
}

4. Create Service Account

Now that the OpenID is created and we have a policy, let’s create an IAM Service Account for EKS:

eksctl create iamserviceaccount --name dynamo-users --namespace default --cluster dec08 --attach-policy-arn arn:aws:iam::188966951897:policy/EKSDynamoUsers --approve --override-existing-serviceaccounts

The above command creates the Service Account named dynamo-users or something that should be unique for your application that the pod uses. Once this is completed we should be able to see the service account in Kubernetes:

kubectl get serviceaccount

5. Modify Pod To Use Service Account

Going back to our YAML file, we need to specify that the pod run with the service account we just created. To do this, we simply add the service account name to the YAML file. This works well if the container is running as the root user:

...
apiVersion: apps/v1
kind: Deployment
metadata:
  name: eks-dynamo
spec:
  replicas: 1
  selector:
    matchLabels:
      run: eks-dynamo
  template:
    metadata:
      labels:
        run: eks-dynamo
    spec:
      serviceAccountName: dynamo-users
      containers:
      - name: eks-dynamo
        image: 188966951897.dkr.ecr.us-west-2.amazonaws.com/vallard/eks-dynamo:latest
...

Notice that we add the serviceAccount to the container in the above code. This will delete the current pod once we put kubectl apply app.yaml and the new one will run with the permissions.

Now we can refresh and see that our application has access to perform the DynamoDB operation on that one table

Hopefully that will help you if you have problems. Check out my twitter account if you have any questions.