How Does K3S Manages Bootstrap Data

K3S is a fully compliant lightweight kubernetes distribution with a minimum memory footprint, its perfect for edge devices however its designed from day one to work on production, for more information about K3S you can refer to the official documenation to get started with installation and management of k3s.

In this post I am going to shed some light on how K3S manages bootstrap data and how is that important for K3S to setup HA clusters, but first we must discuss what is the bootstrap data and what is its purpose in K3S.

Bootstrap Data

The bootstrap data is a structure of data that contains the sensitve information for the k3s server to bootstrap, that includes:

etcd server ca (cert,key)
etcd peer ca (cert,key)
server ca (cert,key)
client ca (cert,key)
passwd file
requestheader ca (cert,key)
ipsec key for flannel backend
encryption config file

These CA certs has to be the same for all HA servers in the cluster so that all certificates generated by K3S will be signed by the same CA certificates, in addition to encryption config files that contains the keys for encryption secrets if its enabled by the user.

Token Management

In order for K3S to start it needs a token that will be used for various purposes, the token can be supplied to the k3s server by using --token flag, however if the user didn’t supply a token via the cli, a randomized token will be created for the server and saved in <data-dir>/server/token.

This token is really critical for different reasons, one of the reason is that the initial server will use this token to encrypt the bootstrap data mentioned in the previous section and save it in the datastore.

Inspecting Bootstrap Data

After starting K3S server you can inspect the bootstrap data saved in the datastore as follows:

1- Start K3S server with etcd backend

curl -sfL https://get.k3s.io | sh -s - "server --token test --cluster-init"

Note that using --cluster-init will allow the server to use etcd backend as its data store.

2- Using etcdctl to inspect the bootstrap

ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt get --prefix /bootstrap --keys-only 

/bootstrap/9f86d081884c

So the command will use etcdctl to list the keys in etcd datastore with prefix /bootstrap which is the prefix that is used to save the bootstrap data key, as for 9f86d081884c part, its the hash of the token passed to K3S server.

To take a little peak on the content of the bootstrap data you can use the same etcdctl without the --keys-only option:

ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt get --prefix /bootstrap
/bootstrap/9f86d081884c
6f0de9faf483a501:WC/bhDCHs1gAMm6DoLg1WIumyAb8SR6JLdZkawfD8qKw3j+........

So the following snippet of code can be used to decrypt this data, which happens internally in k3s of course:

func main() {
	cipher := "xxxxxxxxxxxx"
	data, _ := decrypt("test", []byte(cipher))
	fmt.Println(string(data))
}

func decrypt(passphrase string, ciphertext []byte) ([]byte, error) {
	parts := strings.SplitN(string(ciphertext), ":", 2)
	if len(parts) != 2 {
		return nil, fmt.Errorf("invalid cipher text, not : delimited")
	}

	clearKey := pbkdf2.Key([]byte(passphrase), []byte(parts[0]), 4096, 32, sha1.New)
	key, err := aes.NewCipher(clearKey)
	if err != nil {
		return nil, err
	}

	gcm, err := cipher.NewGCM(key)
	if err != nil {
		return nil, err
	}

	data, err := base64.StdEncoding.DecodeString(parts[1])
	if err != nil {
		return nil, err
	}

	return gcm.Open(nil, data[:gcm.NonceSize()], data[gcm.NonceSize():], nil)
}

After decrypting you should see something like that:

{
  "ClientCA": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdFkyeHAKWlc1MExXTmhRREUyTWpJNE16azVOalV3SGhjTk1qRXdOakEwTWpBMU1qUTFXaGNOTXpFd05qQXlNakExTWpRMQpXakFqTVNFd0h3WURWUVFEREJock0zTXRZMnhwWlc1MExXTmhRREUyTWpJNE16azVOalV3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFTSUUzWDg1bHZTMUNZN0dnUm5YRG1JY2tOQzZGdEJ2OVEwRm5qSVRiZzUKSmsvVGJSdTY2M25HVWJBVTRZbEkwbWJpcFlyMG9KQ0ErVlRsaTdlN2NWYnpvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVTlrS2Q3S21lYXNYT3VPK21hNCt0Cm9MR0xNSVV3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUlnSE5RaExqclUybmZ4bUhGYk4rN002Y1pEUmJlVXlUQXUKTERqS2VXd2ptcE1DSVFET04xQ0hQR0JqQldhdnB5TlN3cm1zQ2hUa2x6aXFnSEtVbjZUQUxBUnd4dz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K",
  "ClientCAKey": "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSVBYTmZQYWk3b1RrdTFSaitzTWJDaS9YdTI0YjM2Q1JuN2R1YXQveVBqNGZvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFaUJOMS9PWmIwdFFtT3hvRVoxdzVpSEpEUXVoYlFiL1VOQlo0eUUyNE9TWlAwMjBidXV0NQp4bEd3Rk9HSlNOSm00cVdLOUtDUWdQbFU1WXUzdTNGVzh3PT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=",
  "ETCDPeerCA": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkVENDQVJ1Z0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWlNU0F3SGdZRFZRUUREQmRsZEdOa0xYQmwKWlhJdFkyRkFNVFl5TWpnek9UazJOVEFlRncweU1UQTJNRFF5TURVeU5EVmFGdzB6TVRBMk1ESXlNRFV5TkRWYQpNQ0l4SURBZUJnTlZCQU1NRjJWMFkyUXRjR1ZsY2kxallVQXhOakl5T0RNNU9UWTFNRmt3RXdZSEtvWkl6ajBDCkFRWUlLb1pJemowREFRY0RRZ0FFTUU2d1VXazY5dkxiNnArd0lTYUZkN1hNRGRFdUxkQkJHWWFDYzQ5YTNMcFEKcE9vQURuSzFTcnJnVjhaTFVaSVpCbGI1K0JLem5VVXlEMHcxdlRISDJLTkNNRUF3RGdZRFZSMFBBUUgvQkFRRApBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkFaRURvUGJWcUcwOEZIR01OVklVQmVwCk8xbFVNQW9HQ0NxR1NNNDlCQU1DQTBnQU1FVUNJUURCT0RVT1JhazBWbVR5dE9UV2pRbDJwUHRMS0NIREp5SW0KbHJydTFzQ3p5UUlnQU9rdzF6RHlyOHlDdzhoWGNhbnNwOGdWbnI3ZURWVys2UzVPNmJ0NVN0dz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=",
  "ETCDPeerCAKey": "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUJ5WnZoYS9GdzExd1dJSlhEMU04NmwyZTdLNDB0V0RHc1Y1Q0xsV2RtZ3pvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTUU2d1VXazY5dkxiNnArd0lTYUZkN1hNRGRFdUxkQkJHWWFDYzQ5YTNMcFFwT29BRG5LMQpTcnJnVjhaTFVaSVpCbGI1K0JLem5VVXlEMHcxdlRISDJBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=",
  "ETCDServerCA": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlakNDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmxsZEdOa0xYTmwKY25abGNpMWpZVUF4TmpJeU9ETTVPVFkxTUI0WERUSXhNRFl3TkRJd05USTBOVm9YRFRNeE1EWXdNakl3TlRJMApOVm93SkRFaU1DQUdBMVVFQXd3WlpYUmpaQzF6WlhKMlpYSXRZMkZBTVRZeU1qZ3pPVGsyTlRCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJHdjRZeGxWOHNOSVpFcmxtNG8rOHVvYURDcmozQ0lBRDZGNmR2VE4KSjdsT1VERml1cklYMXZOMHZUSXhreEVCTGNqQlZQTWExN01CdHlRUTNCTzBUZUNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUkdFeVh3cTF6YnhLM2ZFL1lTCnN0aFJ0MU5FQVRBS0JnZ3Foa2pPUFFRREFnTkpBREJHQWlFQTZNeGpyOHpYOTl0TlRMbStQajZwT0xuTVpxZjIKTzFsbUJNcURXT3NxWlBZQ0lRQzh2MnAxa3pZR3JmcmJmOFpTUGFFTWtJTG94UkUrMWp4WDlnMTEyMVA5OEE9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==",
  "ETCDServerCAKey": "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUxJRUMzcnlqdm9pQ0RPcDRKQ3pKeCtaanpDbFZ0VnozVkJCZFllbjNUbW9vQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFYS9oakdWWHl3MGhrU3VXYmlqN3k2aG9NS3VQY0lnQVBvWHAyOU0wbnVVNVFNV0s2c2hmVwo4M1M5TWpHVEVRRXR5TUZVOHhyWHN3RzNKQkRjRTdSTjRBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=",
  "IPSECKey": "NzFiNGFlZDg2YzA2ODFmYjM0NjM1NjZhY2NkZmE1NGFmZDUxMTFlN2NkODhiNDg3YTA0OTIyMzA1YjgyZjFmOTg2NDE1OThjYzBhNGYwM2U0Y2VkYjZkNzIwYjY2YjUxCg==",
  "PasswdFile": "ZmUyNzQwM2JmYTZmYzhiYjk4Yzk4NzI2ODdkZjRkZmMsbm9kZSxub2RlLGszczphZ2VudApmZTI3NDAzYmZhNmZjOGJiOThjOTg3MjY4N2RmNGRmYyxzZXJ2ZXIsc2VydmVyLGszczpzZXJ2ZXIK",
  "RequestHeaderCA": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJpRENDQVMyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQXJNU2t3SndZRFZRUUREQ0JyTTNNdGNtVngKZFdWemRDMW9aV0ZrWlhJdFkyRkFNVFl5TWpnek9UazJOVEFlRncweU1UQTJNRFF5TURVeU5EVmFGdzB6TVRBMgpNREl5TURVeU5EVmFNQ3N4S1RBbkJnTlZCQU1NSUdzemN5MXlaWEYxWlhOMExXaGxZV1JsY2kxallVQXhOakl5Ck9ETTVPVFkxTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFZ0kyOU01emlPeFlqUWVZMEZIWHYKSG1ORVVRUTJXUHE4eHJKOUNZcnlCTUp6Qmc2MUJWNlRueWp3RWJtQllrZnhaaUdFTTZoUmsyRzVlMTZMZXp6QgpHcU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFCkZPdnorTC80dXNyUktkeE5KbXEyUnVYUFBUUTdNQW9HQ0NxR1NNNDlCQU1DQTBrQU1FWUNJUUNXYVVsK2FvWEwKUkZPVC9janBkN1NicTIwTlRvZFR0UmQ1ZEkzUFNkSHkxd0loQU5HYkJGMitpTXAvczY3ekZCb1RVSE51TmtEKwpXcG1Nbm9Oa05mOUxHeVI1Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K",
  "RequestHeaderCAKey": "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUsvb1ZqOUFRUEtZdkJOY0ttNzlJM0ZoVU5WL2FQeFVmc3BVSFZaNkZ6UG5vQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFZ0kyOU01emlPeFlqUWVZMEZIWHZIbU5FVVFRMldQcTh4cko5Q1lyeUJNSnpCZzYxQlY2VApueWp3RWJtQllrZnhaaUdFTTZoUmsyRzVlMTZMZXp6QkdnPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=",
  "ServerCA": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTWpJNE16azVOalV3SGhjTk1qRXdOakEwTWpBMU1qUTFXaGNOTXpFd05qQXlNakExTWpRMQpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTWpJNE16azVOalV3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFTMXpBbjVVdEFwUmR0ZGhremliaU1IU1dEbVlPVDdQZUxmU2IvdlhteDMKanFNREJ1Vkx3WDJYdWp6WEN6NDI3aDhIRUdkbWwvejM5OUJ4TFpramdNdVRvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVTdPWkJXeHlVN0lQNkJLV0ZRYTQxClFpNE9CaUl3Q2dZSUtvWkl6ajBFQXdJRFNRQXdSZ0loQUwrSGs5cklsNWtJSVNxUDlBREVBY3krRzErREFpZWYKUDd6cXpYUU9QUHRjQWlFQWtMZk5KakhxNzJzWjF6aDdXVmpicHY4RVJjYXJJMGFtb0RKTENIMENGRTQ9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K",
  "ServerCAKey": "LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUJPNEV3MDJPWUpVM3k3RUw5Tm5xMEZaZGJUSUM1dnJFczFlbWEvZXFpbk1vQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFdGN3SitWTFFLVVhiWFlaTTRtNGpCMGxnNW1Eayt6M2kzMG0vNzE1c2Q0NmpBd2JsUzhGOQpsN284MXdzK051NGZCeEJuWnBmODkvZlFjUzJaSTRETGt3PT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=",
  "ServiceKey": "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBeG1kb0tQc1pkMGlCOWZhNjF5cmRNMWNQVUpqS0FTcHl6LzExS0xsbjhkQlh2cFhyCjZQdGhxdG93UVBEU2diaTU1MXY1UVFHTkFPdWdEOTQ0UFNaRTNqWkw0VUdUcmkrTzg1UXpGdDc5Nko2WUgrOWgKOHlPSWlGSmhXcERqSEZXUjFtQXJadytuLzVoeWFJR0krZXIwamJZZHJBVWhDSmpsNE9mNFRuclpSb3hOd2U0KwpldGhjUUU1VkN1RGp6VXZ0L2NJUTRKckpaaERqcWtZb1dvdGZta3lzLy9zV1QrN25QWnREWUR4S0x0TWVnTi9zCm4yKy9UZUNyOERZN3BzWHpwbGNQMzc2TmdJQWljQTFhbGx1T2QyU1Z6N2J0U2oxUVkrdzc1VTRmRE9NeU8vVFQKUi96cXF5WUVEdHNJQnFKZVd6OFFNTjQwTGh0N1V5dlhocGFablFJREFRQUJBb0lCQUc2M2JlelFreGw1b094OQpUeTZiZ08wUmZENWh4UGg1azdCMXBGZWhmMXYyQThFYk1KYkhFVzJpYktNUXdLZ2JTY0xPRkg0dU8xMTBIOUk3CkUrOHIxK3FaS2liOXpVZTZ5bThyR0lkKzBQcHE1a0dMK0lFU0p6TExwZ0JBTWp5eGs3eXAxZzZoTWdMME00UVcKSlRZeDQxN2hiaFY4bHo2UTlGZFptN3RQMnNmK3IyQ2l3Z0dWK1dESHZtOWhIYjhDakVuWGx1clJFeDZQck1DcApnN0dBYzFXYVJPYXpRRnE3aS9RNzZuV0RkZmJLM3VBN05RempxMTdscVRUZ1hNTlBkeC9kN0RpTng5RGMwMnp0CnlVUHQzcEVkRGJjZHBhMDRob2NtN3FvQ1VkcGhRVTRNamJHVnVXTThhVGlTYkpyb1daVjdGNnRDSXZaUFA0REoKNkVUbG1vRUNnWUVBOUJNSmhsTkVHTXRCekpCTEZuSGJJSUlVMTA2M1B3VVZBWnBNcTlVaVl1VHJway9tN3hkWgpqNVpFVDNQMXVtNTEvR2RJcWZkT1AyaGNUQXNYYXNod0ltNnVUdS9iOExtSDdFYmZFaGhIT3JyQW5hQmozb2E2CnY0dDJ1MllqZjNOTVBpcXpSa0FIRGRIbS9KYktiR2dTRFdlT01HY2xBa0VxU2cvUSttMGJMRmtDZ1lFQTBCa2IKM0NqZUJZZnByZDFKYVZlRXhYSExMZGxtVjBnOW9YNDRrU0s5RjR1MFZkWDJDRVNEQ2pWYXRzTUwvWGRIY3lvOAphN3MvNEpiTmw3eUNrUjJrQU1QdC92ZDhzUUNkdmpMTjNZV3NZQkJWZDRhVFZ0Q0RzQUxsWjBwaFpEeHAwZjh1Cjd0aTZyY1pPVUZFbDRScmdCd3hha2t2MitYbXVORzFDcWlZUm51VUNnWUJLMWFOditndkJCbUVRVlpXUU5peVgKbFFySzBTbkNPczZzQlNGSTlqcHExQUdrcG84Sjg4RFVFNkN5TTcyK0FheFBUV09zS1ZWbWY1cjgvZEdhUGdhVApxQmZ0d0FGUmw3RWpNcittanFjWVRMWk9IaVZBejFKbTlGM2RLMzNxei9FcWhuRWRNdmgydGtyNlcwbUpYNGRaCmNNelo3bVljZ05wMGh0RHVWNUdqVVFLQmdRQzdRV3psZnN6UzAwdCt6WHFLMVZzL1JMZWRTVnlqTy9saGdiQXIKUWNSQUZOR3d6N2oyRlppSkxodTBDKzJWSnFsZncyamRWSUVWZXY0RUJYakVWcFRHcjRoUS82anZxZXRJUkhVbApXbTBUZ0g4MzhCeFRhWXltYk9TN1BwNUcwV296c2xvc1NWQXFvU0RGZTFBSHNPUlkwMUFKQjF4MWYzNDM1UDJ4CkdGS1ExUUtCZ1FDV1IvbGk3UDR2K0F0SGhncTVuY3BFbVRXMG1FMDFaT3MvTVRjYUY2dE1DZEpqYVZtWHo2THoKZmRqNU1FRHIzMG9SZHFRaktoZFI2Y2ZmaEsyRWpyUFZyUlhMWjdjT0dlVmlwK2V0WWNKMnZ6d1l2ejZjY0xHeQpkZGo2Mmh5d0t0SFcwa2RndDNjaldSQmhDQ0p3QjFtS0FoU1hWVWRKS0ZhdVh6WVQzUi9OMlE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo="

These data are base64 encoded and after decoding it will show the actual certificates and keys used by this k3s server. so this is how K3S actually saves bootstrap data in server nodes, in the next post I will show how k3s creates a HA cluster and how this bootstrap data is actually distributed and used.

August 8, 2021 ∙