OIDC

GSK supports OIDC identity protocol, which allows you to connect third-party auth providers to your cluster.

Overview

There is a good in depth explanation on the Kuberetes docs how the auth flow works: OpenID Connect Tokens

What you will need to do in a broad overview:

Configure your OIDC provider like Microsoft Entra ID (formerly Microsoft Azure AD)
Setup GSK with OIDC parameters
Generate a client secret on your Identity Provider
Distribute the client secret to all people who need kubectl access
Create a RoleBinding or ClusterRoleBinding for you users to Roles or ClusterRoles
(optional) Setup kubelogin to have a better user experience and automatic auth flow via the browser
Use kubectl with the correct context in order to use the OIDC users

IDP Setup

This highly depends on your OIDC provider. You need to get at least these things:

Issuer URL (Is needed for GSK Setup)
Client ID (Is needed for GSK Setup)
Client Secret (Is needed for client Setup)

Lets imagine you got get these values from the IDP:

oidc-issuer-url https://sts.windows.net/1234-1234-1234-1234-1234/ \
oidc-client-id=5678-5678-5678-5678-5678 \
oidc-client-secret="some-client-secret" \

GSK Setup

The minimum needed config for OIDC to work are these settings:

k8s_oidc_enabled
k8s_oidc_issuer_url
k8s_oidc_client_id

This will basically allow your azure users to authorize against the cluster and does not make much sense. You most likely want to use groups and specific users to authorize against the cluster. Therefore you will follow this example with groups and users:

{
  "name": "oidc-cluster",
  "paas_service_template_uuid": "<UUID of GSK template>",
	"parameters": {
		"k8s_worker_node_count": 1,
		"k8s_worker_node_cores": 2,
		"k8s_worker_node_ram": 8,
		"k8s_worker_node_storage_type": "storage_insane",
		"k8s_worker_node_storage": 30,
    "k8s_oidc_enabled": true,
    "k8s_oidc_issuer_url": "https://sts.windows.net/1234-1234-1234-1234-1234/",
    "k8s_oidc_client_id": "5678-5678-5678-5678-5678",
    "k8s_oidc_groups_claim": "groups",
    "k8s_oidc_username_claim": "email"
	}
}

This will setup GSK cluster with the JWT claims email and groups in the JWT payload.

This will allow all users in the IDP to authenticate against the k8s cluster. The authorization (what a user can do) is still missing. For this the users need to be mapped to a Roles or ClusterRoles.

Since the users from your IDP are not to allowed to interact with the cluster yet, an admin needs to setup a ClusterRoleBinding or RoleBinding for authorization. That can easily be done with the generated kubeconfig. The kubeconfig can be obtained by downloading in the panel or using gscloud.

This example will map the user surname@example.com and the groups admins to the cluster-admin role.

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-admins-from-azure
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: "surname@example.com"
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: "admins"
EOF

Now users from the IDP are allowed to interact with the cluster.

Client Setup

kubectl does allow to use plugins. These plugins can provide benefits like making the auth flow easier. Since the tokens from IDP are short lived you want tooling which will take care of the renewal of tokens automatically. You are using kubelogin for this.

Install kubelogin according to their docs.

Test if you get a token:

kubectl-oidc_login get-token \
  --oidc-issuer-url https://sts.windows.net/1234-1234-1234-1234-1234/ \
  --oidc-client-id=5678-5678-5678-5678-5678 \
  --oidc-client-secret="some-client-secret" \
  --oidc-extra-scope="email groups"

If this worked you can continue with creating the “azure” user:

kubectl config set-credentials azure \
  --exec-api-version=client.authentication.k8s.io/v1beta1 \
  --exec-command=kubectl \
  --exec-arg=oidc-login \
  --exec-arg=get-token \
  --exec-arg=--oidc-issuer-url=https://sts.windows.net/1234-1234-1234-1234-1234/ \
  --exec-arg=--oidc-client-id=5678-5678-5678-5678-5678 \
  --exec-arg=--oidc-client-secret="some-client-secret" \
  --exec-arg=--oidc-extra-scope="email groups"

Now you need to create a context for the user to use the created cluster. Get a list of your clusters:

kubectl config get-clusters
NAME
oidc-cluster

Create a new context for the new user:

kubectl config set-context oidc-cluster --cluster=oidc-cluster --user=azure

Switch the context to the new user:

kubectl config use-context azure@oidc-cluster

Now you should be able to access the cluster:

kubectl get pods -A

Available Parameters

All available parameters for OIDC. Changing these parameters will restart the kube-apiserver which leads a to a small Kubernetes API downtime for a few seconds.

k8s_oidc_enabled

Enable or disable oidc.

Available values: true or false

k8s_oidc_issuer_url

URL of the provider that allows the API server to discover public signing keys. Only URLs that use the https:// scheme are accepted. This is typically the provider’s discovery URL, changed to have an empty path.

k8s_oidc_client_id

A client id that all tokens must be issued for.

k8s_oidc_username_claim

JWT claim to use as the user name. By default sub, which is expected to be a unique identifier of the end user. Admins can choose other claims, such as email or name, depending on their provider. However, claims other than email will be prefixed with the issuer URL to prevent naming clashes with other plugins.

k8s_oidc_username_prefix

Prefix prepended to username claims to prevent clashes with existing names (such as system: users).

For example, the value oidc: will create usernames like oidc:jane.doe. If this flag isn’t provided and k8s_oidc_username_claim is a value other than email the prefix defaults to ( Issuer URL )# where ( Issuer URL ) is the value of k8s_oidc_issuer_url. The value - can be used to disable all prefixing.

k8s_oidc_groups_claim

JWT claim to use as the user’s group.

k8s_oidc_groups_prefix

Prefix prepended to group claims to prevent clashes with existing names (such as system: groups). For example, the value oidc: will create group names like oidc:engineering and oidc:infra.

k8s_oidc_required_claim

A key=value pair that describes a required claim in the ID Token. If set, the claim is verified to be present in the ID Token with a matching value.

Add multiple key values pairs separated by comma like so: key1=value1,key2=value2

k8s_oidc_ca_pem

The certificate for the CA that signed your identity provider’s web certificate. Defaults to the host’s root CAs.

If you want to use a ca.pem file in a json payload the payload needs to be escaped. Here an example how to escape the ca.pem with the Lets Encrypt Root CA:

awk '{printf "%s\\n", $0}' isrgrootx1.pem

k8s_oidc_signing_algs

The signing algorithms accepted. Default is “RS256”. Another option is “R512”.

Debugging

In order to properly debug the OIDC configuration you will need the kube-apiserver logs. Enable log shipping in order to see those logs.

Tips

kubelogin will store all tokens in ~/.kube/cache/oidc-login/.

You can get a token from there and ues https://jwt.ms/ in order to see the payload and all its claims with explanation for debugging purpose.

OIDC

Overview #

IDP Setup #

GSK Setup #

Client Setup #

Available Parameters #

k8s_oidc_enabled #

k8s_oidc_issuer_url #

k8s_oidc_client_id #

k8s_oidc_username_claim #

k8s_oidc_username_prefix #

k8s_oidc_groups_claim #

k8s_oidc_groups_prefix #

k8s_oidc_required_claim #

k8s_oidc_ca_pem #

k8s_oidc_signing_algs #

Debugging #

Tips #