Cert Camel

1. What this is, and why not the Runtime API

HAProxy has two ways to change a certificate without restarting:

2. Check your version

haproxy -v

You need 2.2 or newer. 2.1 can replace a certificate HAProxy already knows about but cannot add a new one; below 2.1 there is no runtime certificate update at all.

3. Install the Data Plane API

HAProxy Enterprise (HAPEE)

It ships with the product. Install the package and skip to step 4.

sudo apt-get install hapee-<version>-lb-dataplaneapi   # Debian/Ubuntu
sudo yum install hapee-<version>-lb-dataplaneapi       # RHEL/Rocky

Community HAProxy

It is a separate download from haproxytech/dataplaneapi. Pick the release matching your HAProxy minor version where possible.

There are .deb, .rpm and .apk packages as well as a tarball. Prefer the package — it handles the paths and gives you something to uninstall:

# Debian/Ubuntu - note "amd64" for packages
curl -fsSLO https://github.com/haproxytech/dataplaneapi/releases/download/v3.3.5/dataplaneapi_3.3.5_linux_amd64.deb
sudo apt install ./dataplaneapi_3.3.5_linux_amd64.deb
dataplaneapi --version

If you would rather drop in a binary, the tarball uses x86_64 where the packages use amd64 — an easy five minutes to lose:

curl -fsSLO https://github.com/haproxytech/dataplaneapi/releases/download/v3.3.5/dataplaneapi_3.3.5_linux_x86_64.tar.gz
tar xzf dataplaneapi_3.3.5_linux_x86_64.tar.gz
sudo install -m 0755 build/dataplaneapi /usr/local/bin/dataplaneapi

4. Give HAProxy an admin-level stats socket

The Data Plane API drives HAProxy through this socket. Without level admin it can read but not change anything.

global
    stats socket /var/run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s

Note the mode: 660, not 666. Anyone who can write to this socket can reconfigure your load balancer, so it should be owned by a group only HAProxy and the Data Plane API belong to.

Two things to check before adding the line: distro packages often ship a stats socket line already (Debian and Ubuntu's uses /run/haproxy/admin.sock) — if one is there, add level admin to it rather than declaring a second socket. And the directory has to exist, or the reload fails to bind it; the packaged systemd unit creates /run/haproxy for you, so reuse that path if in doubt.

5. Create the API user — with a hashed password

The Data Plane API reads its credentials from a userlist in the HAProxy configuration. Generate a hash first:

mkpasswd -m sha-512          # from the "whois" package on Debian/Ubuntu
# or, with no extra packages:
openssl passwd -6

Then add the userlist. userlist is a top-level section — it sits alongside global, defaults, frontend and backend, not inside any of them. Anywhere at that level works; near the top keeps it findable.

# /etc/haproxy/haproxy.cfg

global
    stats socket /var/run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    # ... rest of your global section ...

# <- top level, a sibling of global. NOT indented inside it.
userlist dataplaneapi
    user certcamel password $6$mZ8k...<the hash you generated above>

defaults
    mode http
    # ... etc ...

frontend fe_https
    bind *:443 ssl crt /etc/haproxy/ssl/example.com.pem
    default_backend be_app

The name after userlist — dataplaneapi here — is arbitrary, but it must match the userlist: value in the YAML in step 6. A mismatch there is the single most common cause of a 401.

Give the API its own user. Do not reuse the stats page login: those credentials tend to be shared around a team and pasted into chat, and this one can rewrite your configuration.

6. Configure the Data Plane API

Create /etc/haproxy/dataplaneapi.yaml (HAPEE uses /etc/hapee-extras/dataplaneapi.yml):

dataplaneapi:
  # Bind to one address, not everything. See step 8.
  host: 127.0.0.1
  port: 5555

  transaction:
    # Not under /tmp: transaction directories hold copies of your
    # configuration, and /tmp is world-writable.
    transaction_dir: /var/lib/dataplaneapi/transactions

  resources:
    # Certificates uploaded through the API land HERE and nowhere else.
    ssl_certs_dir: /etc/haproxy/ssl
    maps_dir: /etc/haproxy/maps

haproxy:
  config_file: /etc/haproxy/haproxy.cfg
  haproxy_bin: /usr/sbin/haproxy

  reload:
    reload_cmd: systemctl reload haproxy
    restart_cmd: systemctl restart haproxy
    reload_delay: 5

  # Must match the userlist name from step 5. This is the only auth
  # mechanism you want - do NOT also add a "user:" list under
  # dataplaneapi:, which is a second, separate mechanism.
  userlist:
    userlist: dataplaneapi

Make the directories, and keep the certificate one tight — it will hold private keys:

sudo mkdir -p /etc/haproxy/ssl /var/lib/dataplaneapi/transactions
sudo chown -R haproxy:haproxy /etc/haproxy/ssl /var/lib/dataplaneapi
sudo chmod 700 /etc/haproxy/ssl

7. Run it as a service

HAPEE ships a unit already. For the community build, create /etc/systemd/system/dataplaneapi.service:

[Unit]
Description=HAProxy Data Plane API
After=network.target haproxy.service
Requires=haproxy.service

[Service]
# Not root. It needs the config file, the certificate directory and the socket.
User=haproxy
Group=haproxy
ExecStart=/usr/local/bin/dataplaneapi -f /etc/haproxy/dataplaneapi.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now dataplaneapi
sudo systemctl status dataplaneapi

8. Lock it down

Bind to one address

The default listens on every interface. On a load balancer, "every interface" includes the public one. Bind it to the management address, or to 127.0.0.1 with an SSH tunnel if Cert Camel runs elsewhere.

Make a certificate for the API

The API needs its own certificate, and Cert Camel cannot issue this one — it deploys certificates through the API, so the API has to be reachable before the tool can do anything. Chicken and egg. Make it by hand, once; it is an internal certificate that no browser ever sees.

sudo mkdir -p /etc/haproxy/ssl

# One command, valid 10 years. Replace the IP with the address Cert Camel
# will actually connect to - that is what has to match the SAN.
sudo openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
  -keyout /etc/haproxy/ssl/dataplaneapi.key \
  -out    /etc/haproxy/ssl/dataplaneapi.crt \
  -subj   "/CN=lb1" \
  -addext "subjectAltName = IP:10.0.0.11, DNS:lb1"

# The key must not be world-readable. Match the user the API runs as.
sudo chown haproxy:haproxy /etc/haproxy/ssl/dataplaneapi.*
sudo chmod 600 /etc/haproxy/ssl/dataplaneapi.key
sudo chmod 644 /etc/haproxy/ssl/dataplaneapi.crt

Each node gets its own, with its own address in the SAN. Do not copy one between nodes — a certificate naming 10.0.0.11 presented by 10.0.0.12 is exactly the mismatch this is meant to catch.

Because it is self-signed, Cert Camel will not trust it by default. Tick allow self-signed / private certificates on the load balancer group — see section 9. That accepts this connection being unverified; it does not weaken anything else.

Put TLS on it

Basic authentication over plain HTTP sends the password, base64-encoded, on every single request. Base64 is not encryption. Terminate TLS on the API:

dataplaneapi:
  # Both of these are required. Without "scheme", no TLS listener starts
  # at all - see the warning below.
  scheme:
    - https
  tls:
    tls_host: 10.0.0.11
    tls_port: 5555
    tls_certificate: /etc/haproxy/ssl/dataplaneapi.crt
    tls_key: /etc/haproxy/ssl/dataplaneapi.key

Alternatively bind to localhost and reach it through an SSH tunnel, which needs no certificate at all.

If the API uses a private or self-signed certificate

Almost nobody puts a publicly trusted certificate on a management API, so this is the normal case rather than the exception. There are two ways to handle it, and the first is better:

# 1. a CA, once - keep ca.key somewhere safe, it signs everything below
openssl req -x509 -newkey rsa:4096 -nodes -days 3650 \
  -keyout ca.key -out ca.crt -subj "/CN=Internal LB CA"

# 2. a key and request for THIS node
openssl req -newkey rsa:2048 -nodes \
  -keyout dpa.key -out dpa.csr -subj "/CN=lb1.internal"

# 3. sign it, with the SANs you will actually connect to - see below
openssl x509 -req -in dpa.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -days 825 -out dpa.crt \
  -extfile <(printf "subjectAltName=DNS:lb1.internal,IP:10.0.0.11")

# 4. point dataplaneapi.yml at dpa.crt / dpa.key, then restart the API

1. Install the API's CA on the machine running Cert Camel. Copy the CA certificate across and import it into the Windows Trusted Root Certification Authorities store — certutil -addstore -f Root ca.crt as administrator, or per-user via certutil -user -addstore Root ca.crt. Cert Camel then verifies the API normally and the tick-box stays off.
2. Tick Skip TLS verification on the target. Fine for a lab, and the only option when you cannot install the CA. It affects only the connection to the API — certificate verification never trusts anything regardless, because it reads what a node serves rather than trusting it.

The certificate also needs the address you connect to in its SANs. A certificate issued only for lb1.internal will be rejected when Cert Camel is pointed at 10.0.0.11, correctly and confusingly.

$c = New-Object Security.Cryptography.X509Certificates.X509Certificate2 'ca.crt'
$ch = New-Object Security.Cryptography.X509Certificates.X509Chain
$ch.ChainPolicy.ExtraStore.Add($c); $ch.Build($leaf); $ch.ChainStatus

Restrict who can reach it

Start from what is actually standing there. The API password is a single factor, and it is the whole of the authentication: anyone who can reach port 5555 and knows it can rewrite your configuration. TLS does not change that arithmetic. A server certificate protects the client from connecting to an impostor; it does nothing to restrict who may connect to the real one, because an attacker simply does not verify it — the same thing Cert Camel's own allow self-signed tick does. The CA certificate is handed out to clients rather than kept from them. It is not a second credential.

So the question worth answering is who can reach the port at all, and the answer should be the machine running Cert Camel, and nothing else — not your workstation, not the rest of the office, not whatever else shares that subnet. Nothing else has any business here.

Restricting it leaves your sites alone: 5555 is the management API, and whatever HAProxy is proxying on 80 and 443 keeps answering everyone exactly as it did before.

level=info msg="10.0.0.50 - - [16/Aug/2026:19:56:38 +0000] \"GET /v3/services/haproxy/configuration/version HTTP/1.1\" 401 37 \"-\" \"curl/8.13.0\""

One more, about Cert Camel rather than about the API: it retries on a schedule with a stored password. Rotate the credential on the load balancer without updating it here, and unattended renewal produces failed logins against every node until somebody notices — every six hours, not once a day, so a ban threshold arrives sooner than you might expect. Anything that bans on repeated failures will duly ban Cert Camel, and keep it banned after you have fixed the password — so exempt its address from the ban rather than discovering this at four in the morning.

And if these are a keepalived pair, whatever you put in front of them must not drop protocol 112: the nodes stop hearing each other and both become master. That failure and how to recognise it are in section 12.

The rest

• A dedicated user for the API, not the stats login.
• A long random password. Nobody types it — Cert Camel stores it DPAPI-encrypted — so there is no reason for it to be memorable.
• Not root. Run as the HAProxy user with just enough rights to reload.
• Rotate it like any other credential, and immediately if the config file containing the hash leaks.
• Watch the logs. Every configuration change goes through this API, so its log is an audit trail worth keeping.

9. Prove it works

From the machine that will run Cert Camel, not from the load balancer itself — the point is to test the path that will actually be used:

# should return a number, the current config version
curl -su certcamel:PASSWORD http://10.0.0.11:5555/v3/services/haproxy/configuration/version

# should return a JSON array, possibly empty
curl -su certcamel:PASSWORD http://10.0.0.11:5555/v3/services/haproxy/storage/ssl_certificates

If you already enabled TLS in step 8, use https:// with --cacert ca.crt (or -k for a quick check) — the http:// form above will be refused, which is exactly what step 8 configured, not a fault to debug.

If /v3 gives a 404, try /v2 — older builds serve that. Cert Camel probes both, so either is fine.

10. Prepare your certificate paths

HAProxy identifies a certificate by its file path. That single fact decides whether hitless replacement is possible at all. A path containing a date or a version cannot be updated in place, because next time it is a different path — which means editing the config, which means a reload.

Before                                     After
/certs/2026.example.com/fullchain.pem  ->  /etc/haproxy/ssl/example.com.pem
/certs/2027.example.com/fullchain.pem      (path never changes; contents replaced)

Two things to do:

1. Move the certificates into ssl_certs_dir — the API only manages files there. This is usually more work than the config lines.
2. Give each certificate a stable filename with no date in it, and point the bind line at it.

frontend fe_https
    bind *:443 ssl crt /etc/haproxy/ssl/example.com.pem alpn h2,http/1.1

Or, better, a crt-list — which additionally lets you add a new certificate at runtime, not just replace an existing one. Cert Camel makes direct use of this: give it the crt-list path (in the deployment group's crt-list path field, exactly as it appears on the bind line) and any certificate it pushes that the list does not reference yet is appended and hot-loaded — a brand-new certificate starts serving with no config edit and no reload window to schedule. The list must live inside ssl_certs_dir, since that is the only directory the API manages:

frontend fe_https
    bind *:443 ssl crt-list /etc/haproxy/ssl/crt-list.txt alpn h2,http/1.1

# /etc/haproxy/ssl/crt-list.txt
/etc/haproxy/ssl/example.com.pem
/etc/haproxy/ssl/other.com.pem

A PEM for HAProxy is leaf, then intermediates, then the private key, in one file. That is exactly what Cert Camel's <name>-full.pem already is.

11. Point Cert Camel at it

Settings → Load balancers → Add a load balancer group. One group per set of nodes sharing credentials. Nodes go one per line:

lb1 https://10.0.0.11:5555
lb2 https://10.0.0.12:5555

The URL is the base only — scheme, host and port, with no /v3 or /v2 on the end. Cert Camel adds the version itself, after asking the API which one it speaks. Nearly every example URL in HAProxy's documentation carries a /v3, so this is an easy one to paste in by accident; Cert Camel strips a trailing version segment rather than building /v3/v3/… and reporting a 404 that reads like a missing endpoint.

Fill in the username and password from step 5. The two filename-related fields depend on which bind form you chose in step 10:

• Using a crt-list (the recommended form): put its path in crt-list path, exactly as it appears on the bind line, and leave Certificate filename on HAProxy at the default {certId}.pem. Each certificate gets its own stable file, and Cert Camel appends the reference to the list itself the first time it pushes one the list does not know.
• Using a plain crt line: leave crt-list path empty and set Certificate filename on HAProxy to the exact name on the bind line — see the warning below, because this is the one that bites silently.

Then press Test — it reports per node, with the API version and the certificates each one can see. Then save, assign the group to a certificate on its row in the Certificates table, and press Deploy.

12. Pairs and clusters

Every node needs the Data Plane API installed and configured. They can share a username and password — that is what a group is — but each is pushed to and verified individually.

Which address, exactly

Each node's own permanent IP — the address that belongs to that box and never moves — on the port HAProxy serves the site on. Not the VIP, which floats. In most builds that permanent address is also the management address, but the thing that matters is not that it is used for management: it is that HAProxy is listening for the site on it, on that particular node. By default Cert Camel uses the host from the API URL, which is normally the same machine and needs no configuration at all.

13. When it breaks

The Data Plane API's own log is the first place to look; it records every request including the ones it rejected.

14. Complete example configurations

Everything above as three finished files, for a pair of load balancers. Adapted from a working two-node lab rather than written from memory — the structure, the health check and the reload behaviour have all actually been run.

The example is lb1 and lb2 sharing a virtual address that moves between them:

lb1   10.0.0.11   priority 110   normally MASTER
lb2   10.0.0.12   priority 100   normally BACKUP
VIP   10.0.0.10                  wherever the master is

haproxy.cfg

Identical on both nodes except the node line. Nothing here is Cert Camel-specific except the admin socket and the crt-list, both marked.

global
    log /dev/log local0
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon

    # REQUIRED BY THE DATA PLANE API. It drives HAProxy through this socket,
    # and "level admin" is what lets it load certificates at runtime.
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s

    # OPTIONAL, and think before you set it - see the note under this config.
    # It names the node in the stats page and the logs. Cert Camel does NOT
    # need it: it labels each node from its own settings and tells them apart
    # by URL.
    node lb1
    description edge load balancer, site A

    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
    tune.ssl.default-dh-param 2048

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5s
    timeout client  30s
    timeout server  30s
    retries 3

frontend fe_http
    bind *:80
    http-request redirect scheme https unless { ssl_fc }

frontend fe_https
    # CERT CAMEL WRITES THE crt-list, NOT THIS LINE. The bind stays as you
    # wrote it; deployment adds and updates entries in the file it points at,
    # so new certificates appear without the configuration changing.
    bind *:443 ssl crt-list /etc/haproxy/certs/crt-list.txt alpn h2,http/1.1

    http-response set-header Strict-Transport-Security "max-age=31536000"
    default_backend be_app

backend be_app
    balance roundrobin
    option httpchk GET /healthz
    http-check expect status 200
    server app1 10.0.1.21:8080 check
    server app2 10.0.1.22:8080 check

# Bound to the node's own address, not the VIP - you want to reach a SPECIFIC
# node when working out which one is misbehaving.
listen stats
    bind 10.0.0.11:9000
    stats enable
    stats uri /
    stats refresh 10s
    stats auth admin:CHANGE-THIS

userlist dataplaneapi
    user certcamel insecure-password CHANGE-THIS
    # Better - a hashed password, see section 5:
    # user certcamel password $5$rounds=...

And the crt-list it points at. Create the file even if it is empty — HAProxy will not start with a crt-list that does not exist, and Cert Camel appends to it rather than creating it:

# /etc/haproxy/certs/crt-list.txt
# One line per certificate. Cert Camel maintains these.
/etc/haproxy/certs/example.com.pem
/etc/haproxy/certs/wildcard.example.com.pem

keepalived.conf — the VRRP pair

This is what moves the virtual address between the two nodes, and it is separate from HAProxy entirely: HAProxy binds *:443 on both machines and has no idea a virtual address exists. keepalived decides which node currently holds it.

lb1, the higher priority:

global_defs {
    router_id lb1
    enable_script_security
    script_user root
}

# The piece people leave out, and the one that matters most. Without it the VIP
# happily stays on a node whose HAProxy has died - the machine is still up, so
# VRRP sees nothing wrong, and every request goes to a dead listener.
vrrp_script chk_haproxy {
    script "/usr/bin/killall -0 haproxy"
    interval 2
    timeout 2
    fall 2      # 2 failures before it counts as down
    rise 2      # 2 successes before it counts as up again
    weight -30  # drop priority by 30 - enough to lose to the backup
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51     # MUST match on both nodes, and be unique on the LAN
    priority 110             # higher wins
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass CHANGE-ME  # 8 characters maximum, and sent in clear
    }

    virtual_ipaddress {
        10.0.0.10/24 dev eth0
    }

    track_script {
        chk_haproxy
    }
}

lb2 is the same file with three changes — everything else, especially virtual_router_id, auth_pass and virtual_ipaddress, stays identical:

    router_id lb2
    state BACKUP
    priority 100

    unicast_src_ip 10.0.0.11
    unicast_peer {
        10.0.0.12
    }

dataplaneapi.yml

One per node, each managing only its own HAProxy. Full detail is in section 6; this is the whole file for lb1:

config_version: 2
name: lb1

dataplaneapi:
  host: 0.0.0.0
  port: 5555

  # TLS on the API itself. It carries private keys and a password, so this is
  # not optional. Section 8 has the one openssl command that creates these two
  # files - and note Cert Camel cannot issue this certificate, because it
  # deploys THROUGH the API, so the API has to be up before the tool works.
  # Self-signed is fine; tick "allow self-signed" in Cert Camel.
  tls:
    tls_host: 0.0.0.0
    tls_port: 5555
    tls_certificate: /etc/haproxy/ssl/dataplaneapi.crt
    tls_key: /etc/haproxy/ssl/dataplaneapi.key
  scheme:
    - https

  # Authentication comes from the HAProxy userlist above - section 5.
  userlist:
    userlist: dataplaneapi

  transaction:
    transaction_dir: /var/lib/dataplaneapi/transactions
    backups_number: 10
    backups_dir: /var/lib/dataplaneapi/backups

  resources:
    # WHERE CERT CAMEL WRITES CERTIFICATES. This must be the same directory
    # your crt-list entries point at, or deployment succeeds and HAProxy keeps
    # serving the old certificate - one of the more confusing failures, because
    # nothing reports an error.
    ssl_certs_dir: /etc/haproxy/certs
    maps_dir: /var/lib/dataplaneapi/maps
    general_storage_dir: /var/lib/dataplaneapi/general
    spoe_dir: /var/lib/dataplaneapi/spoe

haproxy:
  config_file: /etc/haproxy/haproxy.cfg
  haproxy_bin: /usr/sbin/haproxy
  reload:
    reload_delay: 2
    reload_cmd: systemctl reload haproxy
    restart_cmd: systemctl restart haproxy
    reload_strategy: custom

log_targets:
  - log_to: file
    log_file: /var/log/dataplaneapi.log
    log_level: info
    log_types:
      - access
      - app

Pointing Cert Camel at the pair

One group, two nodes, each reached by its own address. A node line takes up to three fields:

name   Data Plane API URL        verify address (optional)

lb1    https://10.0.0.11:5555    10.0.0.11:443
lb2    https://10.0.0.12:5555    10.0.0.12:443

The first two are how Cert Camel deploys. The third is how it checks the result: it opens a TLS connection there, asks for each name in turn, and compares the serial of what comes back. Different jobs, so different addresses — and this one has to reach a specific node, which is what catches a standby that missed an update.

Checking it before you trust it

# configuration is valid
haproxy -c -f /etc/haproxy/haproxy.cfg

# who holds the VIP right now - run on both, exactly one should show it
ip addr show eth0 | grep 10.0.0.10

# watch a failover: stop HAProxy on the master and the VIP should move
systemctl stop haproxy && journalctl -fu keepalived

# what a node is actually serving, asked of the node itself
echo | openssl s_client -connect 10.0.0.11:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -subject -dates

That last one is worth keeping. It is what Cert Camel does after every deployment, against every node individually — and running it by hand is how you confirm the tool is telling you the truth.