Single-node K8s: KinD, Istio, & MetalLB
I have a cheap dedicated server with the German hosting company Hetzner. It's just for my own personal use, with lots of disk space to satisfy my data hoarding and enough ram to do the things I need to do. I have it mostly because my home internet is slow. Over time, there got to be a lot of services -- Plex, a Bitcoin node, an IPFS gateway, my e-mail, and recently in my attempts to de-google my life, I've added a search engine, a youtube replacement, a twitter replacement, and a pastebin replacement, among other things. (My rants about big data and the erosion of privacy are for another time!) Those services have been running with docker-compose but since I've been working with Kubernetes for my master's thesis project (a distributed stock exchange order matching engine running on K8s), I wanted to move them over to that. Aside from practicing my Kubernetes-fu, I wanted a setup that could easily be transferred from my single server to a more robust multi-node potentially cloud-hosted platform when I eventually upgrade.
The problem until recently has been that the single-server Kubernetes options were not great. Minikube worked fine but had the added overhead of a VM, and attaching host networking so I could expose my public IP was difficult. Other Docker-based options I tried worked well but had networking issues or other problems that made them not quite a full K8s experience. However, recently I tried Kubernetes in Docker (KinD), and found that none of the issues I had previously were present. It worked without hassle or extra setup, and it was easy to mount my host folders into the K8s nodes for services such as Plex and my backup jobs. And while logically there are many extra layers, those translate physically into nothing more than namespaces and other context wrappers -- which means they don't suffer the performance drawbacks of something like a VM. (I was initially concerned that the KinD nodes were VMs wrapped in containers a la Rancher VM, but it turns out they're just containerd wrapped in containers.) Obviously, there is still overhead, but it was little enough that the benefits finally outweighed the costs.
Although the configuration works well, there were a handful of small issues I ran into, so I thought I'd describe how I did it -- my single-node full Kubernetes instance with Kubernetes-in-Docker, MetalLB, and Istio. There's one hacky workaround to tunnel inbound traffic from the host into Kubernetes, but other than that, it's all mostly straightforward.
The first step is to have Docker and the KinD binary installed on your system. From there, you do a normal KinD installation using whatever configuration you want. For me, that's a basic installation except I've added some host directories to use for app storage rather than a third party PV provider. (I considered setting up a storage backend like Minio S3 to gain some experience with volumes and storage classes, but at some point practicality needs to win, and especially for things like Plex, the unnecessary overhead was more than a philosophical issue.)
Wanting to use the most recent version of Kubernetes, which at the time of this writing was 1.18.4, I manually specified the node image, and ran the setup:
$ kind create cluster --image docker.io/kindest/node:v1.18.4 \
--name kind --config /path/to/kind-config.yaml
The config file merely specified one control plane node and three workers, with my host storage mounted on every worker node. (This way, I could specify the node path in the application's pod spec and know that it would be there on whatever node the pod was created.)
$ cat kind-config.yaml
---
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
extraMounts:
- hostPath: /path/to/host/storage
containerPath: /mnt/host_storage/
- role: worker
extraMounts:
- hostPath: /path/to/host/storage
containerPath: /mnt/host_storage/
- role: worker
extraMounts:
- hostPath: /path/to/host/storage
containerPath: /mnt/host_storage/
---
At this point, once KinD has bootstrapped the cluster, you need to either get the raw kubeconfig data by running kind get kubeconfig, or let KinD set your kubectl context automatically by running kind export kubeconfig. Ideally, everything shows up as working correctly:
$ kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-66bff467f8-blj4s 1/1 Running 0 69s
kube-system pod/coredns-66bff467f8-jvtn5 1/1 Running 0 69s
kube-system pod/etcd-kind-control-plane 1/1 Running 0 78s
kube-system pod/kindnet-fgmqb 1/1 Running 0 52s
kube-system pod/kindnet-k8xpw 1/1 Running 0 69s
kube-system pod/kindnet-q47hs 1/1 Running 0 52s
kube-system pod/kindnet-rflnb 1/1 Running 2 52s
kube-system pod/kube-apiserver-kind-control-plane 1/1 Running 0 78s
kube-system pod/kube-controller-manager-kind-control-plane 1/1 Running 0 78s
kube-system pod/kube-proxy-4w24t 1/1 Running 0 52s
kube-system pod/kube-proxy-6k5zx 1/1 Running 0 69s
kube-system pod/kube-proxy-tz2vh 1/1 Running 0 52s
kube-system pod/kube-proxy-xgf7j 1/1 Running 0 52s
kube-system pod/kube-scheduler-kind-control-plane 1/1 Running 0 78s
local-path-storage pod/local-path-provisioner-67795f75bd-jx4dg 1/1 Running 0 69s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 86s
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 85s
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/kindnet 4 4 4 4 4 <none> 84s
kube-system daemonset.apps/kube-proxy 4 4 4 4 4 kubernetes.io/os=linux 85s
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/coredns 2/2 2 2 85s
local-path-storage deployment.apps/local-path-provisioner 1/1 1 1 83s
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/coredns-66bff467f8 2 2 2 69s
local-path-storage replicaset.apps/local-path-provisioner-67795f75bd 1 1 1 69s
and the host storage mounts are accessible everywhere an application might be deployed:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0c981780e342 kindest/node:v1.18.4 "/usr/local/bin/entr…" 2 minutes ago Up 2 minutes 127.0.0.1:41459->6443/tcp kind-control-plane
ecc5a3120914 kindest/node:v1.18.4 "/usr/local/bin/entr…" 2 minutes ago Up 2 minutes kind-worker
c126cf79ed65 kindest/node:v1.18.4 "/usr/local/bin/entr…" 2 minutes ago Up 2 minutes kind-worker3
5c05cf09e5d6 kindest/node:v1.18.4 "/usr/local/bin/entr…" 2 minutes ago Up 2 minutes kind-worker2
$ docker exec -it ecc5a3120914 ls -lAh /mnt/host_storage/
total 4.0K
-rw-r--r-- 1 1000 1000 11 Jul 8 04:17 file_on_host.txt
$ docker exec -it c126cf79ed65 ls -lAh /mnt/host_storage/
total 4.0K
-rw-r--r-- 1 1000 1000 11 Jul 8 04:17 file_on_host.txt
$ docker exec -it 5c05cf09e5d6 ls -lAh /mnt/host_storage/
total 4.0K
-rw-r--r-- 1 1000 1000 11 Jul 8 04:17 file_on_host.txt
I had in my head that Istio could also take over the function of the base overlay CNI network for Kubernetes, but I did not manage to get that working, so I let KinD install its default simple CNI network "kindnetd", and then I installed Istio on top of that. Even though I probably should have used the Kubernetes-native way of using a Helm chart, I'm lazy and just used the default profile that comes with Istio's istioctl program:
$ istioctl install
This will install the default Istio profile into the cluster. Proceed? (y/N) Y
Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See https://istio.io/docs/ops/best practices/security/#configure-third-party-service-account-tokens for details.
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ Addons installed
✔ Installation complete
And now we see Istio up and running, with its ingress gateway waiting patiently for an external IP that we will give it shortly.
$ kubectl get all -n istio-system
NAME READY STATUS RESTARTS AGE
pod/istio-ingressgateway-66c7db878f-gp9vz 1/1 Running 0 3m31s
pod/istiod-7cdc645bb4-2h75j 1/1 Running 0 4m15s
pod/prometheus-5c84c494dd-879h4 2/2 Running 0 3m31s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/istio-ingressgateway LoadBalancer 10.100.197.167 <pending> 15021:31353/TCP,80:32377/TCP,443:31939/TCP,15443:30195/TCP 3m31s
service/istiod ClusterIP 10.105.18.223 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP,53/UDP,853/TCP 4m15s
service/prometheus ClusterIP 10.107.187.215 <none> 9090/TCP 3m31s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/istio-ingressgateway 1/1 1 1 3m31s
deployment.apps/istiod 1/1 1 1 4m15s
deployment.apps/prometheus 1/1 1 1 3m31s
NAME DESIRED CURRENT READY AGE
replicaset.apps/istio-ingressgateway-66c7db878f 1 1 1 3m31s
replicaset.apps/istiod-7cdc645bb4 1 1 1 4m15s
replicaset.apps/prometheus-5c84c494dd 1 1 1 3m31s
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/istio-ingressgateway Deployment/istio-ingressgateway <unknown>/80% 1 5 1 3m31s
horizontalpodautoscaler.autoscaling/istiod Deployment/istiod <unknown>/80% 1 5 1 4m15s
At this point, everything works, but I still need to somehow route traffic from the host into the Kubernetes network. Since I wanted the manifests portable, I also wanted to be able to specify LoadBalancer as a service type, instead of just always using NodePorts for my configurations. After some research I ended up turning to MetalLB, the "bare metal load balancer" -- i.e. a set of configurations to setup Kubernetes with manual IP addresses.
The first issue I ran into with MetalLB was that it didn't create its namespace metallb-system and it would fail to install if that wasn't present. So I manually created the namespace by applying the following simple yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: metallb-system
labels:
app: metallb
istio-injection: disabled
---
Next was to create a basic config for it, describing which IP addresses it could give out as LoadBalancer addresses. I don't know if this was the right thing to do or not, but I ended up deciding to use the node IP address space for the KinD nodes. I was worried that if I told it to always use my host public IP that that might cause issues. I wasn't sure if the node IP address space would work or be visible from the nodes or services, but I tried it out and it worked... for the most part. The cluster nodes and the pods inside the cluster could access the IP, but the host machine could not. (Don't worry, we'll hack a fix for that shortly.)
The node cluster IP space can be found a number of ways. You can run
$ docker inspect network kind
or you can just query a node directly with something like
$ docker exec -it 0c981780e342 ip addr show dev eth0
My cluster's node IP space was 172.18.0.0/16 with nodes being assigned inside 172.18.0.0/24. I decided to use 172.18.5.0/24 and 172.18.6.0/24 figuring ~500 IPs would be plenty for whatever I was going to do. So the config map yaml which I applied next was the following:
---
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: custom-ip-space
protocol: layer2
addresses:
- 172.18.5.2-172.18.5.254
- 172.18.6.2-172.18.6.254
---
I didn't think I needed to leave room for a gateway and broadcast, since this wasn't actually a new network, but I also wasn't sure how MetalLB worked, so I figured better safe than sorry. (Editor's note: It turns out MetalLB works at a higher level than that, using ARP spoofing or alternatively BGP routing to direct traffic to the correct cluster node.)
Lastly, before MetalLB will work, we need to generate a random secret for it to use. The secret is nothing more than 128 random bytes, base64-encoded, with the line breaks intact. An example method to generate this secret string would be:
$ dd if=/dev/urandom bs=1 count=128 2>/dev/null | base64
or
$ openssl rand -base64 128
The secret has to have the name memberlist, be in the metallb-system namespace, and be a single key/value, with the key name secretkey and the value being the random bytes generated earlier.
An example command to create this new secret might be:
$ kubectl create secret generic -n metallb-system \
memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
Once this new secret is created, you can just install the latest version of MetalLB directly from the source. In my instance, the most recent release on github was 0.9.3, so I applied the following:
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
which created several new resources:
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
podsecuritypolicy.policy/controller created
podsecuritypolicy.policy/speaker created
serviceaccount/controller created
serviceaccount/speaker created
clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
role.rbac.authorization.k8s.io/config-watcher created
role.rbac.authorization.k8s.io/pod-lister created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
rolebinding.rbac.authorization.k8s.io/config-watcher created
rolebinding.rbac.authorization.k8s.io/pod-lister created
daemonset.apps/speaker created
deployment.apps/controller created
With that, the basic installation is complete. Istio is configured and any services that we assign LoadBalancers to will get an IP from the 172.18.5.0/24 or 172.18.6.0/24 range and be accessible from all pods and nodes. Services can also explicitly request/assign their own IP from within that range, which will come in handy shortly.
But first, let's run get all and make sure that everything is correct, including Istio's ingress gateway getting an IP address:
$ kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
istio-system pod/istio-ingressgateway-66c7db878f-gp9vz 1/1 Running 0 107m
istio-system pod/istiod-7cdc645bb4-2h75j 1/1 Running 0 108m
istio-system pod/prometheus-5c84c494dd-879h4 2/2 Running 0 107m
kube-system pod/coredns-66bff467f8-blj4s 1/1 Running 0 119m
kube-system pod/coredns-66bff467f8-jvtn5 1/1 Running 0 119m
kube-system pod/etcd-kind-control-plane 1/1 Running 0 119m
kube-system pod/kindnet-fgmqb 1/1 Running 0 119m
kube-system pod/kindnet-k8xpw 1/1 Running 0 119m
kube-system pod/kindnet-q47hs 1/1 Running 0 119m
kube-system pod/kindnet-rflnb 1/1 Running 2 119m
kube-system pod/kube-apiserver-kind-control-plane 1/1 Running 0 119m
kube-system pod/kube-controller-manager-kind-control-plane 1/1 Running 0 119m
kube-system pod/kube-proxy-4w24t 1/1 Running 0 119m
kube-system pod/kube-proxy-6k5zx 1/1 Running 0 119m
kube-system pod/kube-proxy-tz2vh 1/1 Running 0 119m
kube-system pod/kube-proxy-xgf7j 1/1 Running 0 119m
kube-system pod/kube-scheduler-kind-control-plane 1/1 Running 0 119m
local-path-storage pod/local-path-provisioner-67795f75bd-jx4dg 1/1 Running 0 119m
metallb-system pod/controller-57f648cb96-7q9cx 1/1 Running 0 72s
metallb-system pod/speaker-4pf7b 1/1 Running 0 72s
metallb-system pod/speaker-c22hp 1/1 Running 0 72s
metallb-system pod/speaker-c2f2l 1/1 Running 0 72s
metallb-system pod/speaker-crx56 1/1 Running 0 72s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 119m
istio-system service/istio-ingressgateway LoadBalancer 10.100.197.167 172.18.5.2 15021:31353/TCP,80:32377/TCP,443:31939/TCP,15443:30195/TCP 107m
istio-system service/istiod ClusterIP 10.105.18.223 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP,53/UDP,853/TCP 108m
istio-system service/prometheus ClusterIP 10.107.187.215 <none> 9090/TCP 107m
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 119m
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/kindnet 4 4 4 4 4 <none> 119m
kube-system daemonset.apps/kube-proxy 4 4 4 4 4 kubernetes.io/os=linux 119m
metallb-system daemonset.apps/speaker 4 4 4 4 4 beta.kubernetes.io/os=linux 72s
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
istio-system deployment.apps/istio-ingressgateway 1/1 1 1 107m
istio-system deployment.apps/istiod 1/1 1 1 108m
istio-system deployment.apps/prometheus 1/1 1 1 107m
kube-system deployment.apps/coredns 2/2 2 2 119m
local-path-storage deployment.apps/local-path-provisioner 1/1 1 1 119m
metallb-system deployment.apps/controller 1/1 1 1 72s
NAMESPACE NAME DESIRED CURRENT READY AGE
istio-system replicaset.apps/istio-ingressgateway-66c7db878f 1 1 1 107m
istio-system replicaset.apps/istiod-7cdc645bb4 1 1 1 108m
istio-system replicaset.apps/prometheus-5c84c494dd 1 1 1 107m
kube-system replicaset.apps/coredns-66bff467f8 2 2 2 119m
local-path-storage replicaset.apps/local-path-provisioner-67795f75bd 1 1 1 119m
metallb-system replicaset.apps/controller-57f648cb96 1 1 1 72s
NAMESPACE NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
istio-system horizontalpodautoscaler.autoscaling/istio-ingressgateway Deployment/istio-ingressgateway <unknown>/80% 1 5 1 107m
istio-system horizontalpodautoscaler.autoscaling/istiod Deployment/istiod <unknown>/80% 1 5 1 108m
Looks good. The only problem is that the host (and outside traffic) cannot connect inbound to that load balancer IP address. To accomplish this, we finally get to the hacky band-aid patch that I'm not thrilled about but which has so far worked extremely well: We run socat in host's Docker as reverse proxy, connecting a port on the host to the KinD node network space.
As an example, let us forward the host's TCP port 80 the Istio ingress gateway's load balancer on TCP port 80. To do this, on the host machine's Docker, we run a socat instance like so:
$ docker run -d --network kind -p 80:80 docker.io/alpine/socat:latest \
-dd tcp-listen:80,fork,reuseaddr tcp-connect:172.18.5.2:80
You can see at the end where the load balancer IP address goes, and you can also see with the "-p 80:80" how the container port is mapped from the host. This creates a straight line from a host port bind over to socat, over to the load balancer service inside the cluster. If we imagine UDP port 9000 was also listening on the Istio ingress gateway, we could do a UDP port forward from the host in a similar manner:
$ docker run -d --network kind -p "9000:9000/udp" \
docker.io/alpine/socat:latest -dd \
UDP4-RECVFROM:9000,fork UDP4-SENDTO:172.18.5.2:9000
(The -dd flag is just the verbosity of logging/debugging output and can be ommitted.)
For my setup, I have a single docker-compose file with each load-balanced service port getting its own socat container. An example might look like:
---
version: "3"
services:
socat_istio_ingress:
image: docker.io/alpine/socat:latest
restart: unless-stopped
container_name: socat_istio_ingress
networks:
- kind
ports:
- "0.0.0.0:80:80/tcp"
command: "-dd tcp-listen:80,fork,reuseaddr tcp-connect:172.18.5.2:80"
socat_istio_ingress_udp9000:
image: docker.io/alpine/socat:latest
restart: unless-stopped
container_name: socat_istio_ingress_udp9000
networks:
- kind
ports:
- "0.0.0.0:9000:9000/udp"
command: "-dd UDP4-RECVFROM:9000,fork UDP4-SENDTO:172.18.5.2:9000"
networks:
kind:
external:
name: kind
---
This feels too hacky to me, but it works phenomenally well, and I have had no issues with speed or bandwidth. It also allows me to cut off external traffic without touching the Kubernetes cluster. There's only two major downsides. The first one is that I no longer get source IP addresses, since as far as K8s is concerned, all traffic originates from the socat container IPs. That's not an issue for me (yet) on my single-node personal setup, but it would be problematic for organizations looking to block DDoS attacks, gather user geolocation metrics, or other things that utilize source IP addresses. The other (and lesser) downside is that I can't connect from the host to NodePort addresses, so I have to set those as load balancers, and map the host port as a localhost bind rather than a 0.0.0.0 bind. I know I could use iptables for the port forwarding, and that's probably the best solution for everything, but I want to keep things containerized and predictable without messing around with host networking. This setup is mostly so I can get some K8s experience, utilizing its full feature set, so I don't mind so hacking around the edges to make my setup work. (As an aside, kubectl port-forward also works reasonably well and you can set the timeout to inifinity, so I might just end up using that.)
Editor's note: After posting this article, I found out about a recent feature in KinD which is an additional configuration option extraPortMappings that lets you bind host ports to KinD container ports, similar to Docker's port forwarding option. This solves the issue and makes my socat hack unnecessary, and it is what I have switched to for forwarding incoming traffic.
Even with the few small issues, this setup does indeed provide a full feature-complete Kubernetes on my single bare-metal server, with the ability to assign load balancers to services. This allows my configuration to be more portable (even though I'll have to remove the manually specified LB IP addresses). And it allows me to work with and practice using Kubernetes in managing my own services. And of course, while there is certainly overhead to all of this, it's not too terrible. Plex still works fine, and I get all the added benefits of Istio, such as easy monitoring and tracing. (Although instead of Istio's ingress gateway, I use Traefik 2 as my ingress.)
There wasn't too much that was difficult about this, but it was still a kind of interesting setup, and I had to google a few times to find all of the information to glue all the bits and pieces together. So I thought it was still worth putting out there.
Plus, I needed content for my new homepage...