Create the cluster config file
We are now going to configure the cluster. For the sake of traceability, this configuration won’t be done via CLI flags, but via
a configuration file. The path of the cluster config file will later be referenced as the {{cluster.configFile}}, and should be inside /etc/kubernetes.
Following
flannel requirements, you need to use --pod-network-cidr with address 10.244.0.0./16. This CLI option is equivalent to networking.podSubnet in our {{cluster.configFile}} file (see
this issue).
The variable {{cluster.advertiseAddress}} must be set to the network address of your master node through the VPN. You can get it like so:
1
|
ip -4 a show tun0 | grep -Po 'inet \K[0-9.]*'
|
The variables {{audit.sourceLogDir}} & {{audit.sourceLogFile}} were set in
Setup the cluster's Audit Log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: {{cluster.advertiseAddress}}
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
clusterName: {{cluster.name}}
networking:
podSubnet: "10.244.0.0/16"
apiServer:
extraArgs:
audit-policy-file: /etc/kubernetes/audit-log-policy.yaml
audit-log-path: {{audit.sourceLogDir}}/{{audit.sourceLogFile}}
extraVolumes:
- name: audit-policy
hostPath: /etc/kubernetes/audit-log-policy.yaml
mountPath: /etc/kubernetes/audit-log-policy.yaml # See apiServer.extraArgs.audit-policy-file
readOnly: true
- name: audit-log
hostPath: {{audit.sourceLogDir}}
mountPath: {{audit.sourceLogDir}}
pathType: DirectoryOrCreate
readOnly: false
|
1
2
3
|
mv ./kubernetes/cluster-config.yaml {{cluster.configFile}}
chown root:root {{cluster.configFile}}
chmod 600 {{cluster.configFile}}
|
Finally, init the cluster
Pay attention to the feedbacks of the kubeadm command. It will show warnings about misconfigurations.
1
2
3
4
|
# Init the cluster with our cluster config file
kubeadm init --config {{cluster.configFile}}
# Setup kubectl
mkdir -p $HOME/.kube && cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && chown $(id -u):$(id -g) $HOME/.kube/config
|
Now, the kubelet has been configured. Well, mainly. Because, as mentioned
here, it assumes that it should work through the default gateway (our public network), but that’s not what we want. So, we need to explicitly declare our node’s IP.
1
2
3
4
|
sed -i.bak "s/KUBELET_EXTRA_ARGS=/KUBELET_EXTRA_ARGS=--node-ip=$(ip -4 a show tun0 | grep -Po 'inet \K[0-9.]*') /" /etc/sysconfig/kubelet
systemctl restart kubelet.service
# Verify that the `--node-ip` flag is appended to the `/usr/bin/kubelet` process
systemctl status kubelet.service
|
To communicate with each other, pods need a network layer. We’ll use flannel for this. Following its
installation instruction, you need to deploy
this file. But there’s a problem: as mentioned in the
configuration documentation, flannel use the default route (our public network) by default, and we still want to use the VPN fio this. So, I’ve just added a single line in the
kube-flannel file to specify our VPN interface (line 188, - --iface=tun0).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
|
# From https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unused in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.13.1-rc1
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.13.1-rc1
command:
- /opt/bin/flanneld
args:
- --iface=tun0
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
|
1
2
3
4
|
# If you want to run pods on the master (not recommended), run the following command:
kubectl taint nodes $(hostname) node-role.kubernetes.io/master-
# To undo, run the following
kubectl taint nodes $(hostname) node-role.kubernetes.io/master:NoSchedule
|
Join workers
At the end of the kubeadm init... command, a join command was issued if everything went OK. Execute this command on every workers you want in your cluster. The command is something like below:
1
2
|
kubeadm join xxx.xxx.xxx.xxx:yyy --token foo.barqux123456 \
--discovery-token-ca-cert-hash sha256:fed2136f5e41d654f6e6411d4f5e646512fd5
|
If lost, you can create a new one by executing following command on the control pane with:
1
|
kubeadm token create --print-join-command
|
1
2
3
4
|
sed -i.bak "s/KUBELET_EXTRA_ARGS=/KUBELET_EXTRA_ARGS=--node-ip=$(ip -4 a show tun0 | grep -Po 'inet \K[0-9.]*') /" /etc/sysconfig/kubelet
systemctl restart kubelet.service
# Verify that the --node-ip flag is appended to the /usr/bin/kubelet process
systemctl status kubelet.service
|
You can check nodes by running following command from the control pane
1
2
3
|
kubectl get nodes
# Or watch
kubectl get nodes -w
|
After some time, you should see the new node joining the cluster !
You may repeat this part of the process during the life of your cluster to add new nodes.
Create a metallb configmap, from the kubernetes/metallb-configmap.yaml template.
See the docs for full reference on this config file & how to adapt it to your network configuration..
The {{cluster.networkAddress}} corresponds to the network part of your {{cluster.advertiseAddress}}.
1
2
3
4
5
6
7
|
# Deploy metallb
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
# On first install only
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
# Create the configmap
kubectl apply -f ./kubernetes/metallb-configmap.yaml
|
To check if everything works so far, start a test nginx instance:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
|
kubectl create namespace nginx-test
kubectl --namespace nginx-test run nginx --image nginx
# This may take some time to fetch the container
kubectl --namespace nginx-test expose pod nginx --port 80 --type LoadBalancer
nginx_ip="$(kubectl --namespace nginx-test get svc nginx --output json | jq --raw-output '.status.loadBalancer.ingress[].ip')"
if [[ ! -z "$nginx_ip" ]]; then
echo -e "$(tput setaf 2)Has public IP $nginx_ip. Testing connection. If nothing appears bellow, you might have a firewall configuration issue.$(tput sgr0)"
if ! timeout 5 curl http://$nginx_ip ; then
echo -e "$(tput setaf 1)nginx unreachable. You might have a firewall configuration issue.$(tput sgr0)"
fi
else
echo "No public IP"
fi
unset nginx_ip
|
This should return Has public IP with an IP that should be reachable from the host & the HTML of the default nginx page. If not, then you might have additional configuration to do.
Cleanup the namespace afterwards
1
|
kubectl delete namespace nginx-test
|
Hey, we’ve done important things here ! Maybe it’s time to commit…
1
2
3
4
|
git add .
git commit -m "Kickstart the cluster
Following guide @ https://gerkindev.github.io/devblog/walkthroughs/kubernetes/02-cluster/"
|
Troubleshoot
Kubelet is not running
I had to reinstall kubelet to clear previous runs configurations.
1
|
dnf reinstall -y kubelet kubeadm kubectl --disableexcludes=kubernetes
|
Nginx external ip is always pending
Check that iptables is patched correctly.
1
2
3
4
5
6
|
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
update-alternatives --set iptables /usr/sbin/iptables-legacy
|
Check firewall, SELinux & swap
1
2
|
getenforce
cat /proc/swaps
|
Make sure your nodes are ready and that the networking plugin is correctly installed.
Cluster never starts
Move or remove the existing kubeadm config file (if any) in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Check firewall, getenforce & swap status.
Network interfaces are not deleted after reseting kubeadm
1
2
3
|
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
ip link delete cni0
ip link delete flannel.1
|
Usefull commands memo
- Force reinit cluster:
1
2
|
( sudo kubeadm reset -f && sudo rm -rf /etc/cni/net.d || 1 ) && \
sudo kubeadm init --config cluster-config.yaml
|
