Because we are installing our cluster bare metal on servers exposed on the Internet, we’ll need a way to secure all of our network traffic around the critical parts of kubernetes. To do so, we’ll use OpenVPN to create a virtual secured network where all of our nodes will work. Moreover, this network will also contains MetalLB services when
configuring our bare metal load balancer.
You may need to edit your /etc/hosts files to associate vpn.{{cluster.baseHostName}} to your future OpenVPN server on each of the devices that will join the cluster (if vpn.{{cluster.baseHostName}} is not a real DNS name).
1
|
echo '{{vpn.publicServerIp}} vpn.{{cluster.baseHostName}}' >> /etc/hosts
|
See the
docs of kylemanna/openvpn (our OpenVPN server).
OpenVPN server initial setup
On the OpenVPN server, create a volume for OpenVPN so that it can store files, and generate configuration.
1
2
3
4
5
6
7
8
|
# Create the volume
docker volume create --name {{vpn.volumeName}}
# Init OpenVPN configuration & certificates
docker run -v {{vpn.volumeName}}:/etc/openvpn --rm kylemanna/openvpn:2.4 ovpn_genconfig -Nd -u udp://vpn.{{cluster.baseHostName}}:1194
# Generate the EasyRSA PKI certificate authority. This will prompt a password, that you should keep safe. It will be used to generate new client certificates & configs
docker run -v {{vpn.volumeName}}:/etc/openvpn --rm -it kylemanna/openvpn:2.4 ovpn_initpki
# Start the server
docker run -v {{vpn.volumeName}}:/etc/openvpn -it -p 1194:1194/udp --cap-add=NET_ADMIN kylemanna/openvpn:2.4
|
Note on the -Nd flags of the line 4: see
this RTFM page for split tunnel (partial traffic tunnel)
Once the last command is executed, your OpenVPN server should start. If it started properly, just kill it. We will set it up as a systemd service for our host.
Make a systemd service for OpenVPN through docker
Install the systemd/kubernetes-vpn.service template into /usr/lib/systemd/system, then enable this service. It will run our OpenVPN server container.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
[Unit]
Description=OpenVPN server through Docker
After=syslog.target network-online.target docker.service
Wants=network-online.target
Requires=docker.service
Documentation=man:openvpn(8)
Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO
Documentation=https://github.com/kylemanna/docker-openvpn
[Service]
ExecStop=-/usr/bin/docker stop %n
ExecStopPost=-/usr/bin/docker rm %n
ExecStartPre=/usr/bin/docker pull kylemanna/openvpn:2.3
ExecStart=/usr/bin/docker run --name %n -v {{vpn.volumeName}}:/etc/openvpn --rm -p 1194:{{vpn.port}}/udp --cap-add=NET_ADMIN kylemanna/openvpn:2.3
TimeoutStartSec=30
TimeoutStopSec=15
Restart=always
RestartSec=10s
Type=simple
[Install]
WantedBy=multi-user.target
|
1
2
3
4
5
|
mv ./systemd/kubernetes-vpn.service /usr/lib/systemd/system
# Reload available services to take into account our new `kubernetes-vpn`
systemctl daemon-reload
# Start & auto start it
systemctl enable --now kubernetes-vpn.service
|
You can check our docker container with docker container inspect kubernetes-vpn.service & get our OpenVPN logs with journalctl -u kubernetes-vpn.service.
Now, get the value of the variable {{vpn.serverIp}} with this command:
1
2
3
4
5
|
# Show interface informations
docker exec -it kubernetes-vpn.service ip -4 addr show tun0
# Or, fancy buggy variant to show only interface IP
docker exec -it kubernetes-vpn.service ip -4 addr show tun0 `# Get the "tun0" interface infos` \
| grep -Po 'inet \K[0-9.]*'
|
See
Static IP Addresses documentation for docker-openvpn
Setup clients
This section is meant to be repeated for each of your cluster’s nodes. For every node, replace the {{node.ip}} & {{node.name}} variables.
Important: {{node.ip}} is the desired IP of your machine in your VPN. It must be on the same network than {{vpn.serverIp}} (usually, 192.168.255.XXX)
Generate credentials
For each of our clients, we’ll need to generate credentials so that they can connect to the vpn server. Those clients may use static IPs. The master(s) must have a static IP since it must be reachable via a constant address for kubectl.
On your OpenVPN‘server host:
1
2
3
4
5
6
|
# Generate a client
docker run -v {{vpn.volumeName}}:/etc/openvpn --rm -it kylemanna/openvpn:2.4 easyrsa build-client-full {{node.name}} nopass
# Set its static IP
echo "ifconfig-push {{node.ip}} {{vpn.serverIp}}" | docker run -v {{vpn.volumeName}}:/etc/openvpn -i --rm kylemanna/openvpn:2.4 tee /etc/openvpn/ccd/{{node.name}}
# Get its config to your host
docker run -v {{vpn.volumeName}}:/etc/openvpn --rm kylemanna/openvpn:2.4 ovpn_getclient {{node.name}} > {{node.name}}.ovpn
|
Move this {{node.name}}.ovpn file to the {{node.name}} node by a safe mean. Those files are super critical, so be very careful to not put it anywhere usafe.
Next operations have to be run on clients.
Install OpenVPN client
Install required dependencies.
1
2
3
|
dnf install epel-release git
dnf update
dnf install openvpn
|
Install certificates
Install the certificate previously copied with:
1
2
3
4
|
# Install the OpenVPN configuration
install -o root -m 400 {{node.name}}.ovpn /etc/openvpn/client/{{node.name}}.conf
# Enable the OpenVPN client
systemctl enable --now openvpn-client@{{node.name}}
|
Repeat those steps for each of our nodes, then make sure that you can reach each of your nodes from each other and you can still access the Internet.
1
2
3
4
5
|
# Check internet connection by pinging Google
ping -c 4 8.8.8.8
# Check in-VPN connection
ping -c 4 {{vpn.serverIp}}
# Eventually, check connection to other nodes
|
If you’re having troubles pinging 8.8.8.8 or another’s node IP, please refer to
this troubleshooting section
You should be good to go ! 🔥
Troubleshoot
No internet connection on nodes, or no connection between nodes
Check in case-by-case.
I had to add a route push in my server configuration to make it work. See
How To Guide: Set Up & Configure OpenVPN Client/server VPN | OpenVPN
1
2
|
docker exec -it kubernetes-vpn.service bash -c "echo 'push \"route 192.168.255.0 255.255.255.0\"' >> /etc/openvpn/openvpn.conf"
systemctl restart kubernetes-vpn.service
|
In another setup, I had to comment out push "block-outside-dns" from the server config file (see
this comment).
Usefull commands demo
Flush all routes
1
2
|
sudo iptables -t filter -F
sudo iptables -t filter -X
|
Remove a client
I bet there’s a better way to do this, but I noted this for myself.
1
2
3
4
5
|
# Remove your node from this file
vim /var/lib/containers/storage/volumes/{{vpn.volumeName}}/_data/pki/index.txt
docker exec -it kubernetes-vpn.service rm /etc/openvpn/pki/issued/kube-master.crt
docker exec -it kubernetes-vpn.service rm /etc/openvpn/pki/private/kube-master.key
docker exec -it kubernetes-vpn.service rm /etc/openvpn/pki/reqs/kube-master.req
|
Regenerate client configs & copy them
1
2
3
4
5
6
7
8
9
10
11
12
|
OVPN_DATA=ovpn-data-cluster
clients=kube-master-1 kube-worker-1
docker run -v {{vpn.volumeName}}:/etc/openvpn --rm kylemanna/openvpn:2.4 ovpn_genconfig -u udp://vpn.bar.com:1194
docker run -v {{vpn.volumeName}}:/etc/openvpn --rm -it kylemanna/openvpn:2.4 ovpn_initpki
sudo systemctl restart kubernetes-vpn
for client in $clients; do
docker run -v {{vpn.volumeName}}:/etc/openvpn --rm -it kylemanna/openvpn:2.4 easyrsa build-client-full $client nopass
docker run -v {{vpn.volumeName}}:/etc/openvpn --rm kylemanna/openvpn:2.4 ovpn_getclient $client > $client.ovpn
done
sudo install -o root -m 400 $(hostname).ovpn /etc/openvpn/client/$(hostname).conf
sudo systemctl restart openvpn-client@$(hostname)
scp kube-worker-1.ovpn gerkin@192.168.1.26:~
|
