Get root on the server

The server in your project is yours, and so is root on it. You rarely need it: the portal’s operations run through pbx-agent and need no shell. This page is for the times you do, such as reading logs the portal cannot show, or running the cluster without PaaSbox, and for closing the way again afterwards.

In progress: not bookable yet

In progress In the lab run of 2026-10-10 a privileged pod, started with an admin kubeconfig, reached the server’s file system. Adding a key that way and logging in over SSH has not been run on a PaaSbox cluster yet. Tell me when it opens.

How the server is set up

• The node image runs an SSH server that accepts root with a key only. There are no passwords on the image.
• The portal adds no SSH key when it creates the server, and the cluster’s firewall keeps port 22 closed.
• Root’s keys are read from /etc/ssh/authorized_keys.d/root. That file survives reboots and upgrades; /root does not.

Hetzner’s web console is not a way in. It needs a root password, and the image sets none.

What you need

• Your SSH public key.
• The public address you work from.
• For the first way: an admin kubeconfig (Get a kubeconfig) and kubectl.
• Access to the cluster’s project in the Hetzner console.

Add your key through the cluster

Use this while the cluster runs.

1. Start a pod on the node that sees the server’s file system. The node is named after the cluster’s DNS label, such as upcheck-prod-cp-1:

   kubectl debug node/upcheck-prod-cp-1 -it --profile=sysadmin --image=busybox

   --profile=sysadmin runs the pod privileged, the way the lab’s pod on the node ran.

2. Add your key. Inside the pod, the server’s file system is at /host. Put your own public key between the quotes:

   mkdir -p /host/etc/ssh/authorized_keys.d
   echo 'ssh-ed25519 AAAA... you@laptop' >> /host/etc/ssh/authorized_keys.d/root
   exit

3. Delete the debug pod. kubectl get pods shows it; its name starts with node-debugger-.

   kubectl delete pod <its name>

4. Open port 22 for your address with a firewall of your own. In the Hetzner console, under Firewalls, create a firewall with one inbound rule, TCP port 22 from your address (such as 203.0.113.7/32), and apply it to the server. Do not add the rule to the cluster’s own firewall: the portal writes that one’s rules again whenever the API ranges are saved (Change who can reach the API).

5. Log in.

   ssh root@<the server's IPv4>

Use Hetzner’s rescue system

Use this when the cluster does not run, or before pbx-agent ever enrolled. The rescue system is a Linux that Hetzner boots from the network instead of the server’s disk, with the disk attached. The server reboots for it, so the cluster and your apps are down while you work in it.

1. Open port 22 for your address with a firewall of your own, as in step 4 above. The cluster’s firewall applies to the rescue system too.

2. In the Hetzner console, open the server’s Rescue tab, choose your SSH key and enable the rescue system; it takes effect with the next start, so power-cycle the server.

3. Log in with ssh root@<the server's IPv4>. The server’s disk is attached but not mounted; lsblk lists it.

4. When you are done, reboot. Hetzner uses the rescue system for one start only, so the server boots the node image again, and k3s and pbx-agent start on their own.

Close it again

• Close port 22. Remove your firewall from the server in the Hetzner console, or delete it. A delete of the cluster does not remove it: it carries no label of the cluster’s.
• Remove your key. The image only ever adds keys. Delete your line from /etc/ssh/authorized_keys.d/root, over SSH before you close the port, or through a debug pod as above.

What to keep in mind on the server

• The portal never connects to the server, and nothing you do there goes through it.
• What you change by hand on the server is yours to answer for (How it works).
• Do not edit the image’s own files in /etc. /etc is an overlay kept on the persistent /var: a file changed there shadows every later image’s version of it, for good. Put a k3s setting of your own into a drop-in of your own, such as /etc/rancher/k3s/config.yaml.d/60-mine.yaml; the 50-pbx-*.yaml drop-ins belong to pbx-agent, which writes them again.

Run without PaaSboxWhat you need on the server to keep the cluster running on your own.

TroubleshootingWhat a state means, and what to do when the cluster does not become ready.