Why should we change the Loadbalancer and Ingress Controller for k3s ? If you want to use your network IPs and use multiple tcp/udp ports, the standard ServiceLB and the configured k3s Traefik Ingress Controller will not do the job.
In this post I will explain how you can install k3s with MetalLB and Ingress NGINX Controller .
Configure Network Interface
First of all, we should configure our local interface to support multiple local IPs.
For that, we will create following file /etc/network/interfaces.d/metallb. In this example, our interface is enp0s5 and we want to configure the range 10.10.0.151-10.10.0.162
1 auto enp0s5
2 allow-hotplug enp0s5
3 iface enp0s5 inet dhcp
4 up ip addr add 10.10.0.151/24 dev $IFACE label $IFACE:0
5 down ip addr del 10.10.0.151/24 dev $IFACE label $IFACE:0
6 up ip addr add 10.10.0.152/24 dev $IFACE label $IFACE:1
7 down ip addr del 10.10.0.152/24 dev $IFACE label $IFACE:1
8 up ip addr add 10.10.0.153/24 dev $IFACE label $IFACE:2
9 down ip addr del 10.10.0.153/24 dev $IFACE label $IFACE:2
10 up ip addr add 10.10.0.154/24 dev $IFACE label $IFACE:3
11 down ip addr del 10.10.0.154/24 dev $IFACE label $IFACE:3
12 up ip addr add 10.10.0.155/24 dev $IFACE label $IFACE:4
13 down ip addr del 10.10.0.155/24 dev $IFACE label $IFACE:4
14 up ip addr add 10.10.0.156/24 dev $IFACE label $IFACE:5
15 down ip addr del 10.10.0.156/24 dev $IFACE label $IFACE:5
16 up ip addr add 10.10.0.157/24 dev $IFACE label $IFACE:6
17 down ip addr del 10.10.0.157/24 dev $IFACE label $IFACE:6
18 up ip addr add 10.10.0.158/24 dev $IFACE label $IFACE:7
19 down ip addr del 10.10.0.158/24 dev $IFACE label $IFACE:7
20 up ip addr add 10.10.0.159/24 dev $IFACE label $IFACE:8
21 down ip addr del 10.10.0.159/24 dev $IFACE label $IFACE:8
22 up ip addr add 10.10.0.160/24 dev $IFACE label $IFACE:9
23 down ip addr del 10.10.0.160/24 dev $IFACE label $IFACE:9
24 up ip addr add 10.10.0.161/24 dev $IFACE label $IFACE:10
25 down ip addr del 10.10.0.161/24 dev $IFACE label $IFACE:10
26 up ip addr add 10.10.0.162/24 dev $IFACE label $IFACE:11
27 down ip addr del 10.10.0.162/24 dev $IFACE label $IFACE:11Install Helm
We will use Helm for the installation of MetalLB and Ingress NGINX Controller .
If you already have Helm , you can skip this. We are using Debian on our system, if you are using another Linux Distribution look at the Helm installation documentation .
1 curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
2 sudo apt-get install apt-transport-https --yes
3 echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
4 sudo apt-get update
5 sudo apt-get install helmInstall k3s
Now we can install k3s , where we need to make sure, that we disable the ServiceLB and Traefik .
At this point we will also configure the local storage path, in this case it would be /mnt/data/k3s/storage.
1 curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable traefik --disable servicelb --default-local-storage-path /mnt/data/k3s/storage" sh -To make sure, that kubectl is using the right configuration, we need to set the environment variable KUBECONFIG to /etc/rancher/k3s/k3s.yaml.
You can add it to your .profile or set it for the current session:
1 export KUBECONFIG=/etc/rancher/k3s/k3s.yamlInstall MetalLB
First of all, we need the configuration file (metallb.yaml) for MetalLB , were we configure the IP range which we want to use. In our case 10.10.0.151-10.10.0.162
1 configInline:
2 address-pools:
3 - name: default
4 protocol: layer2
5 addresses:
6 - 10.10.0.151-10.10.0.162Now we can install MetalLB using Helm and our configuration file.
1 helm upgrade --install metallb metallb \
2 --repo https://metallb.github.io/metallb \
3 -f metallb.yamlThats it, MetalLB is now installed and configured for our k3s installation.
Install Ingress NGINX Controller
I choosed the Ingress NGINX Controller , because for my use case at home, it provides all what I need, without needing complicated configuration. As an alternative, you could install your own configured Traefik Ingress Controller , which provides more options than the preconfigured from k3s .
We need a small configuration to make it the default ingress controller, put this into the file nginx.yaml.
1 controller:
2 ingressClassResource:
3 default: trueAnd now install Ingress NGINX Controller using Helm .
1helm upgrade --install ingress-nginx ingress-nginx \
2 --repo https://kubernetes.github.io/ingress-nginx \
3 --namespace ingress-nginx --create-namespace \
4 -f nginx.yamlOne more thing: Certmanager
To handle our SSL certificates automatically, we will use Certmanager , which can use Let’s Encrypt to generate and renew our certificates.
Certmanager provides a ready to use k8s configuration. You just need to make sure, that you are using the latest version (currently it’s version 1.10.1).
1 kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.yamlNow we need to configure Let’s Encrypt and the validation method. In our case, we are using Cloudflare DNS validation .
We will need a secret (cf-secret.yaml) were we will save our Cloudflare API token. (Unfortunately, I was not able to get it working with encoded secret)
1 apiVersion: v1
2 kind: Secret
3 metadata:
4 name: cloudflare-api-token
5 namespace: cert-manager
6 type: Opaque
7 stringData:
8 api-token: TOKEN_IN_PLAIN_TEXTNow we need to configure the ClusterIssuer and to avoid baning, we will configure staging and production environments. We can put both in one file called issuer.yaml.
1 apiVersion: cert-manager.io/v1
2 kind: ClusterIssuer
3 metadata:
4 name: letsencrypt-staging
5 spec:
6 acme:
7 # The ACME server URL
8 server: https://acme-staging-v02.api.letsencrypt.org/directory
9 # Email address used for ACME registration
10 email: [email protected]
11 # Name of a secret used to store the ACME account private key
12 privateKeySecretRef:
13 name: letsencrypt-staging
14 # Enable the HTTP-01 challenge provider
15 solvers:
16 - dns01:
17 cloudflare:
18 email: [email protected]
19 apiTokenSecretRef:
20 name: cloudflare-api-token
21 key: api-token
22 selector:
23 dnsZones:
24 - 'mydomain.com'
25
26 ---
27 apiVersion: cert-manager.io/v1
28 kind: ClusterIssuer
29 metadata:
30 name: letsencrypt-prod
31 spec:
32 acme:
33 # The ACME server URL
34 server: https://acme-v02.api.letsencrypt.org/directory
35 # Email address used for ACME registration
36 email: [email protected]
37 # Name of a secret used to store the ACME account private key
38 privateKeySecretRef:
39 name: letsencrypt-prod
40 # Enable the HTTP-01 challenge provider
41 solvers:
42 - dns01:
43 cloudflare:
44 email: [email protected]
45 apiTokenSecretRef:
46 name: cloudflare-api-token
47 key: api-token
48 selector:
49 dnsZones:
50 - 'mydomain.com'Make sure to use the correct admin email for Let’s Encrypt and the correct user email for Cloudflare .
Last step is to push our configuration to the cluster.
1 kubectl apply -f cf-secret.yaml
2 kubectl apply -f issuer.yamlExample usage
Now that we have everything installed and configured, we can use it in our applications.
Here is an example were install AdGuard Home and use TCP and UDP ports.
Certificate configuration
The file adguard-cert.yaml is to tell Certmanager to generate and renew the certificate for dns.mydomain.com.
1 apiVersion: cert-manager.io/v1
2 kind: Certificate
3 metadata:
4 name: adguard-cert
5 namespace: adguard
6 spec:
7 secretName: adguard-tls
8 issuerRef:
9 name: letsencrypt-prod
10 kind: ClusterIssuer
11 commonName: dns.mydomain.com
12 dnsNames:
13 - dns.mydomain.comSave it and just apply it on the cluster.
1kubectl apply -f adguard-cert.yamlHelm configuration
We are using the chart from k8s-at-home , and we need to make some custom configuration.
You can look which options are available for values.yaml configuration and which volumes are created at AdGuard Home values.yaml
We will use the IP 10.10.0.152 and the domain dns.mydomain.com.
1service:
2 main:
3 annotations:
4 metallb.universe.tf/allow-shared-ip: app.kubernetes.io/instance=k8s-at-home,app.kubernetes.io/name=adguard-home
5 type: LoadBalancer
6 loadBalancerIP: 10.10.0.152
7 externalTrafficPolicy: Local
8 dns-tcp:
9 annotations:
10 metallb.universe.tf/allow-shared-ip: app.kubernetes.io/instance=k8s-at-home,app.kubernetes.io/name=adguard-home
11 type: LoadBalancer
12 loadBalancerIP: 10.10.0.152
13 externalTrafficPolicy: Local
14 dns-udp:
15 annotations:
16 metallb.universe.tf/allow-shared-ip: app.kubernetes.io/instance=k8s-at-home,app.kubernetes.io/name=adguard-home
17 type: LoadBalancer
18 loadBalancerIP: 10.10.0.152
19 externalTrafficPolicy: Local
20ingress:
21 main:
22 enabled: true
23 hosts:
24 - host: dns.mydomain.com
25 paths:
26 - path: /
27 tls:
28 - secretName: "adguard-tls"Install everything using Helm .
1helm upgrade --install k8s-at-home adguard-home \
2--repo https://k8s-at-home.com/charts/ \
3--namespace adguard --create-namespace \
4-f values.yamlAfter waiting to get the pods running, you will be able to access https://dns.mydomain.com with a valid certificate and can use 10.10.0.152 or dns.mydomain.com as your DNS Server.