Handling aws-auth in EKS Clusters

If you use EKS then you have found yourself in a situation where a user can't access the cluster despite having all the IAM permissions and gets an Unauthorized message.  AWS EKS uses IAM credentials for authentication and Kubernetes RBAC for authorization. As per EKS docs:

EKS uses IAM permissions for authentication of valid entities such IAM users or roles. All the permissions for interacting with the EKS cluster is managed through Kubernetes RBAC

or simply put, EKS doesn't work the same way as other services such as S3 where if you have AmazonS3FullAccess, you can access any S3 bucket and create or delete files/folders. In EKS, IAM permissions are only used to check if the user has valid IAM credentials and permissions to run any command using kubectl such as kubectl get pods is managed by Kubernetes API that uses RBAC to control the access.

By default, the IAM Role or IAM User that was used to create the cluster, is added to the system:masters group and gets cluster-wide admin permission with cluster-admin ClusterRole.

As per Kubernetes documentation system:masters group is one of the default ClusterRoleBindings available in the Kubernetes cluster, it's attached to the cluster-admin ClusterRole that gives the user admin permissions in the cluster.

Note: This mapping of creator IAM User or Role to system:masters group is not visible in any configuration such as aws-auth configmap.

EKS allows giving access to other users by adding them in a configmap aws-auth in kube-system namespace. By default, this configmap is empty. However, If you are using eksctl to create the cluster, this config map will have the role created by eksctl for the node group and this role is attached to the system:bootstrappers and system:nodes groups.

aws-auth configmap is based on aws-iam-authenticator and has several configuration options:

1. mapRoles
2. mapUsers
3. mapAccounts

Using mapRoles to Map an IAM Role to the Cluster

mapRoles allows mapping an IAM role in the cluster to allow any entity or user assuming that role to access the cluster. After mapping an IAM role with mapRoles, any user or entity assuming this role is allowed to access the cluster, However, the level of access is defined by the groups attribute.

mapRoles has three attributes:

1. rolearn - IAM Role ARN to map to EKS cluster.
2. username - Username for the IAM Role to map in Kubernetes, this could be a static value like eks-developer or ci-account or a templated variable like {{AccountID}}/{{SessionName}}/{{EC2PrivateDNSName}} or both. This value would be printed in the aws-authenticator Cloudwatch logs if logging is enabled.
3. groups - List of Kubernetes groups that are defined in ClusterRoleBinding/RoleBinding. Example

  subjects:
  - kind: Group
    name: "my-group"
    apiGroup: ""

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: iam-auth-cluster
  region: us-east-1
  version: "1.21"

availabilityZones: 
  - us-east-1a
  - us-east-1b
  - us-east-1c

cloudWatch:
  clusterLogging:
    enableTypes: ["authenticator"]

managedNodeGroups:
  - name: managed-ng-1
    instanceType: t2.micro
    minSize: 1
    maxSize: 4
    desiredCapacity: 4

kubectl get configmap aws-auth -n kube-system -oyaml

apiVersion: v1
data:
  mapRoles: |
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/eksctl-iam-auth-cluster-nodegroup-NodeInstanceRole-1RNKIEA50ZD0B
      username: system:node:{{EC2PrivateDNSName}}
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system

kubectl get pods

error: You must be logged in to the server (Unauthorized)

eksctl create iamidentitymapping \
  --cluster iam-auth-cluster \
  --region us-east-1 \
  --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-developer" \
  --username "eks-developer"

apiVersion: v1
data:
  mapRoles: |
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/eksctl-iam-auth-cluster-nodegroup-NodeInstanceRole-1RNKIEA50ZD0B
      username: system:node:{{EC2PrivateDNSName}}

    - rolearn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-developer
      username: eks-developer
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system

kubectl get pods

Error from server (Forbidden): pods is forbidden: User "eks-developer" cannot list resource "pods" in API group "" at the cluster scope

1. RBAC permissions with Kubernetes User
2. RBAC permissions with Kubernetes Groups

RBAC permissions with Kubernetes User

We can assign RBAC permissions to an IAM role by binding mapped Kubernetes User in aws-auth i.e eks-developer to a ClusterRole/Role.

1. Create a ClusterRole eks-developer-cluster-role with permissions to get, list or watch the pods resources :

   apiVersion: rbac.authorization.k8s.io/v1beta1
   kind: ClusterRole
   metadata:
     name: eks-developer-cluster-role
   rules:
     - apiGroups: [""] # Pod is part of Core API Group and "" indicates the core API group
       resources: ["pods"] # pods resource
       verbs: ["get", "list", "watch"] # Allow user to get, list of watch the pods.
   

2. We have mapped IAM role arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-developer to Kubernetes user eks-developer in aws-auth, now create a ClusterRoleBinding to bind developer-cluster-role to the Kubernetes user eks-developer.

   apiVersion: rbac.authorization.k8s.io/v1beta1
   kind: ClusterRoleBinding
   metadata:
     name: eks-developer-user-cluster-role-binding
   subjects:
     - kind: User
       name: eks-developer # Kubernetes User mapped to the IAM role in aws-auth configmap.
       apiGroup: ""
   roleRef:
     kind: ClusterRole
     name: eks-developer-cluster-role
     apiGroup: ""
   

3. Access the cluster again by assuming the IAM role eks-developer

   kubectl get pods -A
   
   NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
   kube-system   aws-node-f584p             1/1     Running   0          79m
   kube-system   coredns-66cb55d4f4-8hjj2   1/1     Running   0          91m
   kube-system   coredns-66cb55d4f4-vtf6j   1/1     Running   0          91m
   kube-system   kube-proxy-psjk5           1/1     Running   0          79m
   

and this time it works!!!

RBAC permissions with Kubernetes Groups

While assigning permissions directly to the Kubernetes User works just fine for most of the use-cases however this approach is not so great if you want to audit who is assuming the IAM role and accessing the cluster and would like additional information captured in the audit logs.

One such use-case is AWS SSO, where many users are assigned to a permission set and whenever these users log in using their credentials, they assume the same IAM role.

We can assign IAM permissions to an IAM role by creating a Kubernetes Group and add it to the mapRoles.groups field of IAM Role mapping in aws-auth.

Let's take the earlier example of the eks-developer IAM role and create a ClusterRoleBinding with Kubernetes Group developer bound to eks-developer-cluster-role and add Kubernetes Group developer in the mapping of this IAM role in aws-auth.

1. First delete the earlier ClusterRoleBinding eks-developer-user-cluster-role-binding :

   kubectl delete clusterrolebindings eks-developer-user-cluster-role-binding
   clusterrolebinding.rbac.authorization.k8s.io "eks-developer-user-cluster-role-binding" deleted
   

   As soon as we delete the ClusterRolebinding, eks-developer IAM role won't be able to list the pods, let's check the access by assuming the eks-developer IAM role :

   kubectl get pods -A
   Error from server (Forbidden): pods is forbidden: User "eks-developer" cannot list resource "pods" in API group "" in the namespace "default"
   

2. Delete IAM Mapping from aws-auth :

   eksctl delete iamidentitymapping \
         --cluster iam-auth-cluster \
         --region us-east-1 \
         --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-developer"
   

3. Create a ClusterRoleBinding to bind Kubernetes Group developer to cluster role eks-developer-cluster-role:

   apiVersion: rbac.authorization.k8s.io/v1beta1
   kind: ClusterRoleBinding
   metadata:
     name: eks-developer-group-cluster-role-binding
   subjects:
     - kind: Group
       name: developer
       apiGroup: ""
   roleRef:
     kind: ClusterRole
     name: eks-developer-cluster-role
     apiGroup: ""
   

4. Add Kubernetes Group developer to IAM role mapping of eks-developer in aws-auth and include the session name in username using templated variable {{SessionName}}:

   eksctl create iamidentitymapping \
     --cluster iam-auth-cluster \
     --region us-east-1 \
     --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-developer" \
     --username "eks-developer:{{SessionName}}" \
     --group "developer"
   

   this would create an entry under mapRoles section in aws-auth configmap as:

   mapRoles: |
     - groups:
       - developer
       rolearn: arn:aws:iam::017558828988:role/eks-developer
       username: eks-developer:{{SessionName}}
   

Check the Cloudwatch authenticator logs for the authenticated user assuming eks-developer IAM role and we can see that this time session name is appended to the username in logs:

time="2021-09-13T17:57:46Z" level=info msg="access granted" arn="arn:aws:iam::0175XXXXXXXX:role/eks-developer" client="127.0.0.1:48520" groups="[developer]" method=POST path=/authenticate sts=sts.us-east-1.amazonaws.com uid="heptio-authenticator-aws:0175XXXXXXXX:AROAQIFUWO66PDOXKSLMQ" username="eks-developer:eks-developer-session"

If the session name consists of @, it would be replaced with -. Let's assume the IAM role eks-developer with session name containing @ :

aws sts assume-role \
    --role-arn "<IAM_ROLE_ARN>" \
    --role-session-name "my-develper-session@123456789" \
    --duration-seconds 3600

Now Cloudwatch logs would have session name printed as eks-developer:my-developer-session-123456789.

time="2021-09-14T17:50:25Z" level=info msg="access granted" arn="arn:aws:iam::017558828988:role/eks-developer" client="127.0.0.1:57794" groups="[developer]" method=POST path=/authenticate sts=sts.us-east-1.amazonaws.com uid="heptio-authenticator-aws:017558828988:AROAQIFUWO66PDOXKSLMQ" username="eks-developer:my-develper-session-123456789"

There is one problem here, if your EKS cluster is being accessed from multiple AWS accounts, it would not be possible to track the AWS account of the user who accessed the EKS cluster just by session name.

{{AccountID}} comes to the rescue, we can use this templated variable to get the AWS account ID of the user who is assuming the role, so we can set the username to :

aws:{{AccountID}}:eks-developer:{{SessionName}}

eksctl delete iamidentitymapping \
      --cluster "iam-auth-cluster" \
      --region "us-east-1" \
      --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-developer"

eksctl create iamidentitymapping \
      --cluster "iam-auth-cluster" \
      --region "us-east-1" \
      --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-developer" \
      --username "aws:{{AccountID}}:eks-developer:{{SessionName}}" \
      --group "developer"

time="2021-09-14T18:26:33Z" level=info msg="access granted" arn="arn:aws:iam::017558828988:role/eks-developer" client="127.0.0.1:39752" groups="[developer]" method=POST path=/authenticate sts=sts.us-east-1.amazonaws.com uid="heptio-authenticator-aws:0175XXXXXXXX:AROAQIFUWO66PDOXKSLMQ" username="aws:0175XXXXXXXX:eks-developer:my-develper-session-123456789"

Note: If you want session name in raw format, you can use templated variable {{SessionNameRaw}} instead. However as of EKS 1.21, these two variables {{AccessKeyID}} and {{SessionNameRaw}} don't work.

Using mapUser to Map an IAM User to the Cluster

mapUsers allows mapping an IAM User to the cluster and add the user to one or more Kubernetes Groups. It has 3 attributes :

1. userarn - ARN of IAM User to map to EKS cluster. This could be an IAM user from the same AWS account or another account.
2. username - Static username to map this IAM User to, in Kubernetes.
3. groups - List of Kubernetes groups that are defined in ClusterRoleBinding/RoleBinding.

Note: Templated variables are not supported in username field with mapUser.

To add an IAM user with ARN arn:aws:iam::<AWS_ACCOUNT_ID>:user/dev-user in aws-auth configmap, we can run the below command:

eksctl create iamidentitymapping \
  --cluster iam-auth-cluster \
  --region us-east-1 \
  --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:user/dev-user" \
  --username "dev-user"

this command would add these lines in aws-auth configMap:

  mapUsers: |
    - userarn: arn:aws:iam::<AWS_ACCOUNT_ID>:user/dev-user
      username: dev-user

For users mapped using mapUsers, RBAC permission can be given in two ways :

1. RBAC permissions with Kubernetes User
2. RBAC permissions with Kubernetes Groups

RBAC permissions with Kubernetes User

We can assign RBAC permissions to an IAM user by binding mapped Kubernetes User in aws-auth i.e dev-user to a ClusterRole/Role.

1. Create a ClusterRoleBinding to bind Kubernetes user dev-user to the developer-cluster-role:

   apiVersion: rbac.authorization.k8s.io/v1beta1
   kind: ClusterRoleBinding
   metadata:
     name: dev-user-cluster-role-binding
   subjects:
     - kind: User
       name: dev-user # Kubernetes User mapped to the IAM user in aws-auth configmap.
       apiGroup: ""
   roleRef:
     kind: ClusterRole
     name: eks-developer-cluster-role
     apiGroup: ""
   

2. Map IAM user arn:aws:iam::<AWS_ACCOUNT_ID>:user/dev-user to Kubernetes user dev-user in aws-auth configMap:
   
   

  eksctl create iamidentitymapping \
    --cluster iam-auth-cluster \
    --region us-east-1 \
    --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:user/dev-user" \
    --username "dev-user"

RBAC permissions with Kubernetes Groups

If we need to give the same set of permissions to multiple users, then instead of creating multiple ClusterRoleBindings, we can use Kubernetes Groups and attach that group to the users for whom those permissions are required.

1. Create a ClusterRoleBinding to bind Kubernetes Group developer to cluster role eks-developer-cluster-role:

   apiVersion: rbac.authorization.k8s.io/v1beta1
   kind: ClusterRoleBinding
   metadata:
     name: dev-user-group-cluster-role-binding
   subjects:
     - kind: Group
       name: dev
       apiGroup: ""
   roleRef:
     kind: ClusterRole
     name: eks-developer-cluster-role
     apiGroup: ""
   

2. Map IAM user arn:aws:iam::<AWS_ACCOUNT_ID>:user/dev-user to Kubernetes user dev-user with dev group in aws-auth configMap:

   eksctl create iamidentitymapping \
     --cluster iam-auth-cluster \
     --region us-east-1 \
     --arn "arn:aws:iam::<AWS_ACCOUNT_ID>:role/dev-user" \
     --username "dev-user" \
     --group "dev"
   

   this would create an entry under mapUsers section in aws-auth configmap as:

   mapUsers: |
     - groups:
       - dev
       userarn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/dev-user
       username: dev-user
   

Using mapAccounts to Map IAM ARN in an AWS Account to the Cluster

mapAccounts allows mapping all the IAM Users or IAM Roles of an AWS account to the cluster. It accepts the list of AWS Account IDs:

mapAccounts: |
  - "<AWS_ACCOUNT_ID_1>"
  - "<AWS_ACCOUNT_ID_2>"