Kubernetes Lab

If a label pattern is not specified, only labels matching ^[0-9a-z] are executed by default. By starting some labels with a non-alphanumeric character (such as _init) we can easily run all steps in two phases:

Install all packages and base configuration
Initialize databases and join nodes

Rather than installing a Container Networking Interface, we can configure the built-in bridge plugin and set a static route for each kubernetes host

route add 10.244.1.0/16 192.168.2.35
route add 10.244.2.0/16 192.168.2.36
route add 10.244.3.0/16 192.168.2.37

Prerequisites

Some of prerequisites are handled in Kickstart when the host is provisioned

tmux and tar included in %packages
firewall --disabled
selinux --disabled
swap disabled by removing zram-generator-defaults in %post
dns=default set in NetworkManager.conf

Routes and Configuration

# routes.pln
mykube1: kube/
→   suite.pln
→   init.pln

mykube{2..3}: kube/
→   suite.pln

Common

# suite.pln
execute_with=sudo
environment_file=mykube.env

install_kube_pkgs:
→   dnf install -y kubernetes kubernetes-kubeadm kubernetes-client

configure_cni:
→   # overwrite default network configuration before starting daemons
→   ${SD}/rinstall kube/bridge.conflist.$(hostname -s) /etc/cni/net.d/100-crio-bridge.conflist

add_kern_mods:
→   ${SD}/rinstall kube/k8s-mode.conf /etc/modules-load.d/k8s.conf
→   modprobe overlay
→   modprobe br_netfilter

add_sysctl:
→   ${SD}/rinstall kube/k8s-sysctl.conf /etc/sysctl.d/k8s.conf
→   sysctl --system

enable_srv:
→   systemctl enable --now crio
→   systemctl enable --now kubelet

_join:
→   [ -f /etc/kubernetes/kubelet.conf ] && exit
→   curl -s -o - http://$CONTROL_PLANE:8888/join.sh | sh

Source files:

# k8s-mode.conf
overlay
br_netfilter

# k8s-sysctl.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1

Control Plane

# init.pln
execute_with=sudo
environment_file=mykube.env

_init_create_cluster:
→   kubeadm init

execute_with=

_init_add_user_kubeconfig:
→   mkdir -p $HOME/.kube
→   sudo cp -f /etc/kubernetes/admin.conf $HOME/.kube/config
→   sudo chown $(id -u):$(id -g) $HOME/.kube/config

_init_join_script:
→   token=$(kubeadm token list | awk 'NR==2 { print $1 }')
→   cert_hash=$(openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der | openssl dgst -sha256 -hex | sed 's/^.* //')
→   mkdir -p ~/http; cd ~/http
→   echo "kubeadm join $CONTROL_PLANE:6443 --token $token --discovery-token-ca-cert-hash sha256:$cert_hash" > join.sh
→   tmux new-session -s httpd -d
→   tmux send-keys -t httpd:0 "python3 -m http.server 8888" C-m

Bootstrap

#!/usr/local/bin/bash

rset -o logs -p2 mykube{1..3}
rset -x '_init' mykube1
rset -x '_join' mykube2 mykube3

mkdir -p ~/.kube
scp mykube1:.kube/config ~/.kube/config

References

Creating a Kubernetes cluster on Fedora

Debugging DNS Resolution

Redash on Kubernetes