Kubeconfig files for Multiple Kubernetes Clusters
Overview
Background
If you're working with Kubernetes at all, you may be needing to interact with more than one cluster - maybe different clusters for development and production, maybe different applications, or whatever.
Each cluster probably has a kubeconfig
file that you
can use to interact with that cluster, but changing context
based on different files is a bit cumbersome. Options include
changing environment variable for each context change or
using the --kubeconfig
command line option with most
commands.
With a little extra one-time work for each cluster you have, changing context can be a snap.
Anatomy of a Kubeconfig File
Each cluster you have will often be able to provide you
with a kubeconfig
file, and they look something like this:
1---
2apiVersion: "v1"
3kind: "Config"
4
5clusters:
6- name: "erik-test"
7 cluster:
8 server: "https://some-url:6443"
9 certificate-authority-data: "LS0tLS1CRUdJTiBDRVJUSU...."
10
11contexts:
12- name: "erik-test-admin@erik-test"
13 context:
14 cluster: "erik-test"
15 user: "erik-test-admin"
16
17users:
18- name: "erik-test-admin"
19 user:
20 client-certificate-data: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS...."
21 client-key-data: "LS0tLS1CRUdJTiBSU0EgUFJJVkFU...."
22
23current-context: "erik-test-admin@erik-test"
The kubeconfig
file above represents a single cluster,
but the same overall structure can be used by a single kubeconfig
file
to store context for multiple clusters.
- clusters: contains one or more
cluster
item - contexts: contains one or more
context
item - users: contains one or more
user
item - current-context: contains exactly one reference to the
name
property of acontext
Modifying the Default Kubeconfig File
The default kubeconfig
file is ~/.kube/config
- and that's what
should contain your collection of clusters. Modifying that is pretty
easy - do that once for each cluster you have, and switching between
contexts is pretty straight-forward.
In the above example, you would add this content to the clusters
node:
1- name: "erik-test"
2 cluster:
3 server: "https://some-url:6443"
4 certificate-authority-data: "LS0tLS1CRUdJTiBDRVJUSU...."
Then add this to the contexts
node:
1- name: "erik-test-admin@erik-test"
2 context:
3 cluster: "erik-test"
4 user: "erik-test-admin"
Then add this to the users node:
1- name: "erik-test-admin"
2 user:
3 client-certificate-data: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS...."
4 client-key-data: "LS0tLS1CRUdJTiBSU0EgUFJJVkFU...."
Give a Friendly Name to your Context
As a bonus note, the name
of each context
is whatever is easy
for you to understand, so examples of name
values you might use
instead of erik-test-admin@erik-test
are:
dev-local-identity
prod-aws-identity
dev-identity@admin
Including some info about the application, location, envrironment,
and possibly user are possible suggestions - just use a name
value
that is meaningful to you.
Changing Contexts
Once you've got the changes made to your default kubeconfig
file,
you can use the following command to change the context:
1kubectl config use-context <name-of-context>
So in the example from the kubeconfig
content above, the following command would work:
1kubectl config use-context erik-test-admin@erik-test
The last argument is the name
of the context you are switching to,
and that's why choosing a value that's meaningful to you is important.
To get a list of the contexts you have available, use the following:
1kubectl config get-contexts
Include Cluster Name in Prompt
As a further aid in keeping track of your current context, you may want to include the current context (maybe even with the namespace!!) in your prompt if you are using something like Oh-My-Posh.
I've written a post covering how to do that.
In case you're curious, though, this is the current theme I'm using for Oh-My-Posh:
The json is here:
1{
2 "$schema": "https://raw.githubusercontent.com/JanDeDobbeleer/oh-my-posh/main/themes/schema.json",
3 "blocks": [
4 {
5 "alignment": "left",
6 "segments": [
7 {
8 "background": "#6272a4",
9 "foreground": "#ffffff",
10 "leading_diamond": "\ue0b6",
11 "trailing_diamond": "\ue0b0",
12 "style": "diamond",
13 "type": "os"
14 },
15 {
16 "background": "#bd93f9",
17 "foreground": "#ffffff",
18 "powerline_symbol": "\ue0b0",
19 "style": "powerline",
20 "properties": {
21 "style": "folder"
22 },
23 "type": "path"
24 },
25 {
26 "background": "#ffb86c",
27 "foreground": "#ffffff",
28 "powerline_symbol": "\ue0b0",
29 "properties": {
30 "branch_icon": "",
31 "fetch_stash_count": true,
32 "fetch_status": false,
33 "fetch_upstream_icon": true
34 },
35 "style": "powerline",
36 "template": " \u279c ({{ .UpstreamIcon }}{{ .HEAD }}{{ if gt .StashCount 0 }} \uf692 {{ .StashCount }}{{ end }}) ",
37 "type": "git"
38 },
39 {
40 "background": "#ff79c6",
41 "foreground": "#ffffff",
42 "powerline_symbol": "\ue0b0",
43 "style": "powerline",
44
45 "type": "kubectl",
46 "properties": {
47 "prefix": " \uF1D1 "
48 }
49 }
50 ],
51 "type": "prompt"
52 }
53 ],
54 "final_space": true,
55 "version": 2
56}