Kubeconfig files for Multiple Kubernetes Clusters

Nov 4, 2022 · 4 min read · Kubernetes ·

Overview

Background

If you're working with Kubernetes at all, you may be needing to interact with more than one cluster - maybe different clusters for development and production, maybe different applications, or whatever.

Each cluster probably has a kubeconfig file that you can use to interact with that cluster, but changing context based on different files is a bit cumbersome. Options include changing environment variable for each context change or using the --kubeconfig command line option with most commands.

With a little extra one-time work for each cluster you have, changing context can be a snap.

Anatomy of a Kubeconfig File

Each cluster you have will often be able to provide you with a kubeconfig file, and they look something like this:

 1---
 2apiVersion: "v1"
 3kind: "Config"
 4
 5clusters:
 6- name: "erik-test"
 7  cluster:
 8    server: "https://some-url:6443"
 9    certificate-authority-data: "LS0tLS1CRUdJTiBDRVJUSU...."
10
11contexts:
12- name: "erik-test-admin@erik-test"
13  context:
14    cluster: "erik-test"
15    user: "erik-test-admin"
16
17users:
18- name: "erik-test-admin"
19  user:
20    client-certificate-data: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS...."
21    client-key-data: "LS0tLS1CRUdJTiBSU0EgUFJJVkFU...."
22
23current-context: "erik-test-admin@erik-test"

The kubeconfig file above represents a single cluster, but the same overall structure can be used by a single kubeconfig file to store context for multiple clusters.

clusters: contains one or more cluster item
contexts: contains one or more context item
users: contains one or more user item
current-context: contains exactly one reference to the name property of a context

Modifying the Default Kubeconfig File

The default kubeconfig file is ~/.kube/config - and that's what should contain your collection of clusters. Modifying that is pretty easy - do that once for each cluster you have, and switching between contexts is pretty straight-forward.

In the above example, you would add this content to the clusters node:

1- name: "erik-test"
2  cluster:
3    server: "https://some-url:6443"
4    certificate-authority-data: "LS0tLS1CRUdJTiBDRVJUSU...."

Then add this to the contexts node:

1- name: "erik-test-admin@erik-test"
2  context:
3    cluster: "erik-test"
4    user: "erik-test-admin"

Then add this to the users node:

1- name: "erik-test-admin"
2  user:
3    client-certificate-data: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS...."
4    client-key-data: "LS0tLS1CRUdJTiBSU0EgUFJJVkFU...."

Give a Friendly Name to your Context

As a bonus note, the name of each context is whatever is easy for you to understand, so examples of name values you might use instead of erik-test-admin@erik-test are:

dev-local-identity
prod-aws-identity
dev-identity@admin

Including some info about the application, location, envrironment, and possibly user are possible suggestions - just use a name value that is meaningful to you.

Changing Contexts

Once you've got the changes made to your default kubeconfig file, you can use the following command to change the context:

1kubectl config use-context <name-of-context>

So in the example from the kubeconfig content above, the following command would work:

1kubectl config use-context erik-test-admin@erik-test

The last argument is the name of the context you are switching to, and that's why choosing a value that's meaningful to you is important.

To get a list of the contexts you have available, use the following:

1kubectl config get-contexts

Include Cluster Name in Prompt

As a further aid in keeping track of your current context, you may want to include the current context (maybe even with the namespace!!) in your prompt if you are using something like Oh-My-Posh.

I've written a post covering how to do that.

In case you're curious, though, this is the current theme I'm using for Oh-My-Posh:

The json is here:

 1{
 2  "$schema": "https://raw.githubusercontent.com/JanDeDobbeleer/oh-my-posh/main/themes/schema.json",
 3  "blocks": [
 4    {
 5      "alignment": "left",
 6      "segments": [
 7        {
 8          "background": "#6272a4",
 9          "foreground": "#ffffff",
10          "leading_diamond": "\ue0b6",
11		  "trailing_diamond": "\ue0b0", 
12          "style": "diamond",          
13          "type": "os"          
14        },
15        {
16          "background": "#bd93f9",
17          "foreground": "#ffffff",
18          "powerline_symbol": "\ue0b0",    
19          "style": "powerline",          
20		  "properties": {
21			  "style": "folder"		  
22		  },
23          "type": "path"          
24        },
25        {
26          "background": "#ffb86c",
27          "foreground": "#ffffff",
28          "powerline_symbol": "\ue0b0",
29          "properties": {
30            "branch_icon": "",
31            "fetch_stash_count": true,
32            "fetch_status": false,
33            "fetch_upstream_icon": true
34          },
35          "style": "powerline",
36          "template": " \u279c ({{ .UpstreamIcon }}{{ .HEAD }}{{ if gt .StashCount 0 }} \uf692 {{ .StashCount }}{{ end }}) ",
37          "type": "git"
38        },        
39        {
40          "background": "#ff79c6",
41          "foreground": "#ffffff",
42		  "powerline_symbol": "\ue0b0",    
43          "style": "powerline",
44          
45          "type": "kubectl",
46          "properties": {				
47            "prefix": " \uF1D1 "
48          }
49        }
50      ],
51      "type": "prompt"
52    }
53  ],
54  "final_space": true,
55  "version": 2
56}