Servers
Harblet starts on one server. Add more servers to run more services, and Harblet places services on all of them. When a server fails, its services start on another one.
The first server
The server you installed Harblet on is the first server. It has a special role:
- It runs Harblet, Harblet's database, the image registry and the build service.
- It receives all traffic for your domains and routes it to the services on every server. Point all DNS records at it.
- It can't be removed.
See your servers
Servers in the sidebar lists every server with its status (Ready or Not ready), IP address, operating system, k3s version, and CPU and memory use. The page refreshes every 5 seconds. Admins also see a card per server on the Overview page.
Add a server
Prepare the server
Set up a fresh server with Ubuntu 24.04 or 26.04, like the first one. It must be able to:
- reach Harblet's address, to download the join script,
- reach the other servers directly on TCP 6443 and 10250 and UDP 51820.
Servers in one private network, or servers with public IP addresses, work. Servers behind NAT at different providers don't, yet.
Get the join command
On Servers, click Add server. Harblet shows a command:
curl -fsSL https://harblet.example.com/api/join-scripts/<token> | sudo sh
The command works for 24 hours. Every click creates a new one.
Run it on the new server
Run the command on the new server as root. It installs the k3s agent at the same version as the first server. The server shows up on the page within seconds, and Harblet starts placing services on it.
When a server fails
- Services without data start on another server. In tests, a failed server was marked Not ready after about 50 seconds, and its services ran on another server about 30 seconds later.
- Databases stay on the server that holds their data. They wait for it to come back.
- When the server comes back, it rejoins by itself.
Services keep running when Harblet itself is down. Only deploys, logs and backups need Harblet.
Remove a server
Delete its databases
Harblet refuses to remove a server that stores the data of a database. The message names them. Harblet can't move a database to another server, so download a backup first if you need the data, then delete them.
Remove it
On Servers, click the trash button of the server and confirm. Harblet moves the server's services to the other servers and removes it. Join commands that are still valid stop working.
Uninstall it
Run the command Harblet shows on the removed server:
sudo /usr/local/bin/k3s-agent-uninstall.sh
Until you run it, the removed server keeps trying to reconnect and its boot doesn't finish, though SSH still works.
Removing a server that's powered off works too. You can add the same machine again later.
k3s upgrades
Each Harblet release comes with the k3s version it was tested with. After you update Harblet, it upgrades k3s on every server by itself, the first server first and then one server at a time. Services keep running.
While a server upgrades, its k3s version shows Upgrading to the new version. If an upgrade gets stuck, an alert The k3s upgrade is stuck above the list shows Kubernetes' message. See Troubleshooting.