Kubernetes Cluster on Raspberry Pi B (part 2 – Base OS and software prep)

Continuing my documentation on building this Pi cluster. If you want to read the full thing, start at Part 1 – hardware.

This is a work in progress – stuff is broken, so don’t use this guide for now.

Now that we’ve got all our hardware assembled, it’s time to prep the base system.

Base OS

I was using a windows system, so I used balenaetcher to write the raspberry pi OS to disk.

I choose Raspian Buster Lite for the OS.

Use etcher to write the image to each SD card. After the image is written, eject and then put the card back in your system, and you should mount the /boot partition. Create an empty file called “ssh” on the root of the boot system to enable the ssh service at boot.

As these SD cards are written out, you can put each one into a pi and start configuring it. I’m seeing the arpwatch alerts going out when a new mac address connects to my network, so I know what ip address is being allocated as each pi comes online.

So, for each pi – ssh into it. I found there was a good order that I liked so I didn’t screw up my static DNS that I’d setup.

  1. Configure static ip (etc/dhcpcd.conf) – important note, you must not have more than two namedserver lines, or coredns will fail
  2. Reboot to get new ip (I should just be able to restart networking)
  3. use raspi-config to set hostname, timezone, enable ssh (not sure if that’s necessary), and expand the file system
  4. Reboot
  5. sudo apt-get install tmux lynx snmpd snmp
  6. ssh-copy-id from the system that you’ll be using to manage these nodes (I was using ubuntu under Win10) – side note, it’s pretty damn awesome that I can just install Ubuntu one-click via the Microsoft store onto Win10.

Begin installing Apps – * this part is still broken*

I was using tmux with a multiple server option to run these commands on all servers at once. This was another thing that I wanted to play around with – I knew other people did it, but hadn’t had a real reason to want to do it before.

Prereqs for tmux-multi-server:

  1. Have tmux installed on each destination server (raspberry pi)
  2. Have tmux installed on your source terminal (ubuntu under win10)
  3. Already used ssh-copy-id to remove auth between terminal and server
  4. ssh-multi script, I used this one from github

Once this is done, I can run:

./ssh-multi.sh -d "kube1 kube2 kube3 kube4"

and now, I have one terminal session with all four ssh sessions in it – and typing once echos across all servers at the same time.

Install docker on all systems at once:

curl -sSL get.docker.com | sh 
sudo usermod pi -aG docker
newgrp docker

** WRONG ** we must use cgroupfs instead of systemd?
To force the use of systemd instead of cgroupfs edit /lib/systemd/system/docker.service and edit the following line:
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
To be:
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd

Disable swap. Swap must be disabled to run docker. The instructions I followed were wrong on this – which meant whenever I did a reboot, suddenly all my nodes decided to stop running docker. I had to go hunt for the right syntax.

sudo systemctl disable dphys-swapfile.service

edit /boot/cmdline.txt and add this text to the end of the existing line:

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

reboot all servers.

Add repo key for google packages, then add distro to the list, update, and install kubeadm
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list && \
sudo apt-get update -q && \
sudo apt-get install -qy kubeadm

Update your repo files, and install kubeadm

sudo apt-get update
sudo apt-get install -qy kubeadm

Now tell iptables to allow bridging to make weave-net work

sudo sysctl net.bridge.bridge-nf-call-iptables=1

reboot everything just to make sure

At this point, I end the multi-ssh, and log in only to my first pi, which will be the master for the cluster.

Initialize the master node (only do this on your primary/master pi)

sudo kubeadm config images pull -v3
sudo kubeadm init --token-ttl=0 (this was for weave, which seemed to cause issues. Moved to flannel
sudo kubeadm init --token-ttl=0 --pod-network-cidr=

Now, the screen will provide you the join command for your worker nodes. Copy that text locally into your clipboard or notepad file or something. End your single ssh session to the master, and connect to just your worker nodes, then paste that text back in there.

kubeadm join --token [blah] \
--discovery-token-ca-cert-hash [blah]

Connect back to your master and run:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Check if the nodes are online:

kubectl get nodes
kube1 NotReady master 45m v1.15.3
kube2 NotReady 80s v1.15.3
kube3 NotReady 78s v1.15.3
kube4 NotReady 80s v1.15.3

Now, you’ll need to convert those nodes with no role into worker nodes. We do this with the label command: (I bet there’s a better way to do this)
kubectl label node kube2 node-role.kubernetes.io/worker=worker
kubectl label node kube3 node-role.kubernetes.io/worker=worker
kubectl label node kube4 node-role.kubernetes.io/worker=worker

Now you’ll see those node roles change from blank to worker
kubectl get nodes
kube1 NotReady master 49m v1.15.3
kube2 Ready worker 5m29s v1.15.3
kube3 Ready worker 5m27s v1.15.3
kube4 Ready worker 5m29s v1.15.3

install the weave net overlay.
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n') "

Install Flannel overlay:
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Now, get a list of all the pods running on your cluster: (it may take a minute for them to become ready)

kubectl get pods -A
kube-system coredns-5c98db65d4-lw9hf 1/1 Running 1 8m39s
kube-system coredns-5c98db65d4-xwnfs 1/1 Running 1 8m39s
kube-system etcd-kube1 1/1 Running 1 8m16s
kube-system kube-apiserver-kube1 1/1 Running 1 7m55s
kube-system kube-controller-manager-kube1 1/1 Running 1 8m20s
kube-system kube-flannel-ds-arm-7tb8k 1/1 Running 1 6m51s
kube-system kube-flannel-ds-arm-ln792 1/1 Running 1 8m7s
kube-system kube-flannel-ds-arm-m2jkg 1/1 Running 1 6m56s
kube-system kube-flannel-ds-arm-tgbxp 1/1 Running 1 7m3s
kube-system kube-proxy-79s5j 1/1 Running 1 7m3s
kube-system kube-proxy-dgqrj 1/1 Running 1 6m56s
kube-system kube-proxy-kqrzp 1/1 Running 1 8m39s
kube-system kube-proxy-vv278 1/1 Running 1 6m51s
kube-system kube-scheduler-kube1 1/1 Running 1 8m19s

Next step – remote admin.

This entry was posted in Technology. Bookmark the permalink.

Leave a Reply