Proxmox cluster | Reverse proxy with noVNC and SPICE support

I have a 3 node proxmox cluster in production and I was trying to find a way to centralize the webgui management.

Currently the only way to access proxmox cluster web interface is by connecting to each cluster node individually, e.g https://pve1:8006 , https://pve2:8006 etc from your web browser.

The disadvantage of this is that you have either to bookmark every single node on your web browser, or type the url manually each time.

Obviously this can become pretty annoying, especially as you are adding more nodes into the cluster.

Below I will show how I managed to access any of my PVE cluster nodes web interface by using a single dns/host name (e.g https://pve in my case).

Note that you don’t even need to type the default proxmox port (8006) after the hostname since Nginx will listen to default https port (443) and forward the request to the backend proxmox cluster nodes on port 8006.

My first target was the web management console and secondly it was making noVNC and SPICE work too. The last seemed to be more tricky.

We will use Nginx to handle Proxmox web and noVNC console traffic (port 8006) and HAProxy to handle SPICE traffic (port 3128).

Note The configuration below has been tested with the following software versions:

• Debian GNU/Linux 8.6 (jessie)
• nginx version: nginx/1.6.2
• HA-Proxy version 1.5.8
• proxmox-ve: 4.3-66 (running kernel: 4.4.19-1-pve)

What you will need

1. A basic Linux vm. My preference for this tutorial was Debian Jessie.

2. Nginx + HAProxy for doing the magic.

3. OpenSSL packages to generate the self signed certificates.

4. Obviously a working proxmox cluster.

5. Since this will be a critical vm, It would be a good idea to configure it as a HA virtual machine into your proxmox cluster.

The steps

– Download Debian Jessie net-install.

– Assign a static IP address and create the appropriate DNS record on your DNS server (if available, otherwise use just hostnames).
In my case, I created an A record named ‘pve‘ which is pointing to 10.1.1.10 . That means that when you manage to complete this guide your will be able to access all proxmox nodes by using https://pve (or https://pve.domain.local) on your browser! You will not even need to type the default port which is 8006.

– Update package repositories by entering ‘apt-get update’

– Install Nginx and HAProxy:

apt-get install nginx && apt-get install haproxy

Nginx and OpenSSL setup

– Assuming that you are logged in as root, create backup copy of the default config file.

cp /etc/nginx/sites-enabled/default /root

– Remove /etc/nginx/sites-enabled/default:

rm /etc/nginx/sites-enabled/default

– Download OpenSSL packages:

apt-get install openssl

– Generate a private key (select a temp password when prompted):

openssl genrsa -des3 -out server.key 1024

– Generate a csr file (select the same temp password if prompted):

openssl req -new server.key -out server.csr

– Remove the password from the key:

openssl rsa -in server.key -out server_new.key

– Remove old private key and rename the new one:

rm server.key && mv server_new.key server.key

– Make sure only root has access to private key:

chown root server.key && chmod 600 server.key

– Generate a certificate:

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

– Create a directory called ssl in /etc/nginx folder and copy server.key and server.crt files:

mkdir /etc/nginx/ssl && cp server.key /etc/nginx/ssl && cp server.crt /etc/nginx/ssl

– Create an empty file:

vi /etc/nginx/sites-enabled/proxmox-gui

– Paste the code below and save the file. Make sure that you change the ip addresses to match your proxmox nodes ip addresses:

Edit (11-11-2017)

upstream proxmox {ip_hash;    #added ip hash algorithm for session persistencyserver 10.1.1.2:8006;
server 10.1.1.3:8006;
server 10.1.1.4:8006;
}
server {
listen 80 default_server;
rewrite ^(.*) https://$host$1 permanent;
}
server {
listen 443;
server_name _;
ssl on;
ssl_certificate /etc/nginx/ssl/server.crt;
ssl_certificate_key /etc/nginx/ssl/server.key;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $http_host;
location / {
proxy_pass https://proxmox;
}
}

– Create a symlink for /etc/nginx/sites-enabled/proxmox-gui in /etc/nginx/sites-available:

ln -s /etc/nginx/sites-enabled/proxmox-gui /etc/nginx/sites-available

– Verify that the symlink has been created and it’s working:

ls -ltr /etc/nginx/sites-available && cat /etc/nginx/sites-available/proxmox-gui (You should see the above contents after this)

– That’s it! You can now start Nginx service:

systemctl start nginx.service && systemctl status nginx.service (Verify that it is active (running).

HAProxy Setup

– Create a backup copy of the default config file.

cp /etc/haproxy/haproxy.cfg /root

– Create an empty /etc/haproxy/haproxy.cfg file (or remove it’s contents):

vi /etc/haproxy/haproxy.cfg

– Paste the following code and save the file. Again make sure that you change the ip addresses to match your proxmox hosts. Also note that the hostnames must also match your pve hostnames, e.g pve1, pve2, pve3

global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
user haproxy
group haproxy
daemon
stats socket /var/run/haproxy/haproxy.sock mode 0644 uid 0 gid 107
defaults
log global
mode tcp
option tcplog
option dontlognull
retries 3
option redispatch
maxconn 2000
timeout connect 5000
timeout client 50000
timeout server 50000
listen proxmox_spice *:3128
mode tcp
option tcpka
balance roundrobin
server pve1 10.1.1.2:3128 weight 1
server pve2 10.1.1.3:3128 weight 1
server pve3 10.1.1.4:3128 weight 1

– Note that the above configuration has been tested on HA-Proxy version 1.5.8.
If the Nginx service fails to start please troubleshoot by running:

haproxy -f /etc/haproxy/haproxy.cfg ...and check for errors.

– Start HAProxy service:

systemctl start haproxy.service && systemctl status haproxy.service (Must show active and running)

Testing our setup…

Open a web browser and enter https://pve . You should be able to access PVE webgui. (remember in my case I have assigned ‘pve’ as hostname to the Debian VM and I have also created a similar entry on my DNS server. That means that your client machine must be able to resolve the above address properly otherwise it will fail to load proxmox webgui).

You can now also test noVNC console and SPICE. Please note that you may need to refresh noVNC window in order to see the vm screen.

UPDATE: You can seamesly add SSH to the proxied ports if you wish to ssh in any of pve host.

Just add the lines below to your /etc/haproxy/haproxy.cfg file. Note that I’m using port 222 instead of 22 in order to prevent conflicting ports with the actual Debian vm which already listens on port tcp 22.


listen proxmox_ssh *:222
mode tcp
option tcpka
balance roundrobin
server pve1 10.1.1.2:22 weight 1
server pve2 10.1.1.3:22 weight 1
server pve3 10.1.1.4:22 weight 1

Now if you try to connect from your machine as root@pve at port 222 (ssh root@pve -p 222), the first time you will be asked to save the ECDSA key of the host to your .ssh/known_hosts file and then you will login to the first proxmox node e.g pve1.
If you attempt to connect for a second time your request will be rejected since HAProxy will forward your request to the second proxmox node e.g pve2 which happens to have a different fingerprint from the first. This is good of course for security reasons but in this case we will need to disable the check for the proxied host, otherwise we will not be able to connect to it.

– On your client machine, modify /etc/ssh/ssh_config file (not sshd_config !).

– Remove the following entry:

Host *

– Add the following at the end of the file:

Host pve
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null
ServerAliveInterval 5

This will prevent the security ECDSA key checks ONLY for host pve and enable them from ALL other hostnames. So in short it’s quite restrictive setting.ServerAliveInterval is used in order to keep the ssh session alive during periods of inactivity.I’ve noticed that without setting that parameter to ssh client, it will drop the session quite often.