FreeBSD and Hetzner Cloud Private Networking
In my ever-developing curiosity I’ve recently tried to mix Hetzner private network, Cloud, Dedicated servers and FreeBSD all together. In June 2022, Hetzner announced Flexible Networking for its Cloud offerings which means that a newly created Cloud Server will no longer automatically include public IP addresses (IPv4, IPv6). The total price for cloud servers with public IP addresses will be the same as before, but by removing the Primary IPv4 we have the ability to reduce costs. Servers without public network still need access to package repositories and similar resources, thus let us explore adding FreeBSD in the mix.
Overview
The network topology we want to achieve:
Connectivity works by using the existing private network mechanisms for both systems:
- For cloud servers, we will create a “Network” and add cloud servers to it.
- For dedicated root servers, we will create a “vSwitch” and attach our dedicated root servers to it.
Networks provide private environments for servers to communicate directly. Every server receives a private IP address that is not on the internet. Instead, dedicated network interfaces are used to provide private layer 3 (OSI model) links between Hetzner Cloud servers. The vSwitch feature is a tool for our dedicated root servers that lets us connect servers in multiple locations to each other using virtual layer 2 networks. On a Hetzner account, on the Robot administration interface, we can create and configure vSwitches using the “vSwitches” button in the server overview:
Using hcloud the command-line interface for Hetzner Cloud
hcloud is a command-line interface for interacting with Hetzner Cloud.
Installation
You can download pre-built binaries for Linux, FreeBSD, macOS, and Windows on the releases page.
$ wget -q https://github.com/hetznercloud/cli/releases/download/v1.30.3/hcloud-linux-amd64.tar.gz
$ tar -zxvf hcloud-linux-amd64.tar.gz
LICENSE
README.md
hcloud
Move the binary somewhere in your $PATH:
$ mv hcloud .local/bin/
Configuration
To configure hcloud we need an API token. Create an API token, by signing in into the Hetzner Cloud Console choose a Project, then go to Security -> API Tokens, and generate a new token.
Make sure to copy the token because it won’t be shown to you again. A token is bound to a Project, to interact with the API of another Project you have to create a new token inside the Project
$ hcloud context create mytest
Token:
Context mytest created and activated
List our context:
$ hcloud context list
ACTIVE NAME
* mytest
You can also use the following environment variables to configure hcloud:
HCLOUD_TOKEN: Instead of creating a context, you can set the token via theHCLOUD_TOKENenvironment variable.HCLOUD_CONTEXT: When usinghcloudin scripts, we can configure a per-directory context by settingHCLOUD_CONTEXT=my-context
Adding SSH keys
To create a SSH key for our gateway server:
$ hcloud ssh-key create --name andreibuzoianu --public-key-from-file .ssh/id_ed25519
$ hcloud ssh-key list
ID NAME FINGERPRINT
8520345 andreibuzoianu 9e:1d:08:cf:41:e6:e5:00:60:4b:a3:20:ab:b1:79:ec
Hetzner Cloud Network
$ hcloud network create --name mytest --ip-range 172.16.0.0/23
Network 2146252 created
$ hcloud network add-subnet mytest --type cloud --network-zone eu-central --ip-range 172.16.0.0/24
700ms [==================================] 100.00%
Subnet added to network 2146252
$ hcloud network add-subnet mytest --type vswitch --network-zone eu-central --ip-range 172.16.1.0/24 --vswitch-id 44385
600ms [==================================] 100.00%
Subnet added to network 2146252
Add a default gateway (IP of the FreeBSD server) for our network:
$ hcloud network add-route mytest --destination 0.0.0.0/0 --gateway 172.16.0.10
1.8s [===================================] 100.00%
Route added to network 2146252
Same can be achived using the Hetzner Cloud Console:
In Hetzner Cloud Console, we can now see the route:
To list our newly created network:
$ hcloud network list
ID NAME IP RANGE SERVERS
2146252 mytest 172.16.0.0/23 0 server
Or in Hetzner Cloud Console:
To get all the information related to our network:
$ hcloud network describe mytest
ID: 2146252
Name: mytest
Created: Fri Nov 25 22:27:43 EEST 2022 (27 minutes ago)
IP Range: 172.16.0.0/23
Subnets:
- Type: cloud
Network Zone: eu-central
IP Range: 172.16.0.0/24
Gateway: 172.16.0.1
- Type: vswitch
Network Zone: eu-central
IP Range: 172.16.1.0/24
Gateway: 172.16.0.1
vSwitch ID: 44385
Routes:
- Destination: 0.0.0.0/0
Gateway: 172.16.0.10
Protection:
Delete: no
Labels:
No labels
Creating and installing the FreeBSD gateway
To create a new server we can use hcloud:
$ hcloud server create --name gateway --image debian-9 --type cx11 --ssh-key andreibuzoianu
6s [====================================================================] 100%
Server 325211 created
Hetzner recommends using their Image functionality to install virtual servers, but they also provide some stock ISOs so we can install more exotic operating systems by ourself. When creating the server we did use debian-9 image since we are required to provide an image. Next we’ll attach an FreeBSD ISO to the server.
Currently listed ISOs for FreeBSD:
$ hcloud iso list | grep FreeBSD
ID NAME DESCRIPTION TYPE
7300 FreeBSD-13.0-RELEASE-amd64-bootonly.iso FreeBSD 13.0 (amd64/netinstall) public
14376 FreeBSD-13.1-RELEASE-amd64-bootonly.iso FreeBSD 13.1 (amd64/netinstall) public
To attach an FreeBSD ISO to our Server use:
$ hcloud server attach-iso gateway FreeBSD-13.1-RELEASE-amd64-bootonly.iso
Select Power from the right menu and then Power-Cycle to restart the Server and boot from the FreeBSD ISO:
Select Console from top right:
Then install FreeBSD using the text-based installation program named bsdinstall. Use DHCP for IPv4 configuration which will automatically configure the network interface correctly for Hetzner.
FreeBSD gateway configuration
In FreeBSD, the rc.conf file contains system configuration information, such as descriptive information about the local host name, configuration details for any potential network interfaces and which services should be started up at system initial boot time.
Excerpt from rc.conf to show the network configuration:
ifconfig_vtnet0="DHCP"
ifconfig_vtnet1="inet 172.16.0.10 netmask 255.255.255.255 broadcast 172.16.0.255 descr LAN"
static_routes="cloudgateway cloudnet dedicated"
route_cloudgateway="-host 172.16.0.1 -interface vtnet1"
route_cloudnet="-net 172.16.0.0/24 172.16.0.1"
route_dedicated="-net 172.16.1.0/24 172.16.0.1"
Ensure IP forwarding is enabled:
$ sysctl -a|grep ip.forwarding
net.inet.ip.forwarding: 1
To make it persistent:
$ cat /etc/rc.conf.d/routing
gateway_enable="YES"
Configuration of Hetzner Cloud Servers and Dedicated Servers
When creating cloud servers disable the public network, in Hetzner Cloud Console, for that particular server:
w1 routing table:
[w1 ~]$ ip r l
default via 172.16.0.1 dev enp7s0 proto static metric 100
172.16.0.1 dev enp7s0 proto static scope link metric 100
w1 network interface:
[w1 ~]$ ip a l enp7s0
2: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel state UP group default qlen 1000
link/ether 86:00:00:26:b4:07 brd ff:ff:ff:ff:ff:ff
inet 172.16.0.3/32 scope global noprefixroute enp7s0
valid_lft forever preferred_lft forever
The dedicated server is also a FreeBSD server. Excerpt from rc.conf:
hostname="dedicated"
vlans_igb0="cloud"
create_args_cloud="vlan 4000"
ifconfig_cloud="inet 172.16.1.2 netmask 255.255.255.0 mtu 1400"
static_routes="cloud"
route_cloud="-net 172.16.0.0/24 172.16.1.1"
The virtual interface as configured:
dedicated ~$ ifconfig cloud
cloud: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1400
options=4600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether a8:a1:59:82:3c:d4
inet 172.16.1.2 netmask 0xffffff00 broadcast 172.16.1.255
groups: vlan
vlan: 4000 vlanproto: 802.1q vlanpcp: 0 parent interface: igb0
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
The dedicated server routing table related to Hetzner Networks:
dedicated ~$ netstat -rn|grep 172.16
172.16.0.0/24 172.16.1.1 UGS cloud
172.16.1.0/24 link#5 U cloud
172.16.1.2 link#5 UHS lo0
Testing our setup
gateway ~ $ ping -c 1 172.16.0.1
PING 172.16.0.1 (172.16.0.1): 56 data bytes
64 bytes from 172.16.0.1: icmp_seq=0 ttl=255 time=0.482 ms
--- 172.16.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.482/0.482/0.482/0.000 ms
gateway ~ $ ping -c 1 172.16.0.3
PING 172.16.0.3 (172.16.0.3): 56 data bytes
64 bytes from 172.16.0.3: icmp_seq=0 ttl=63 time=1.116 ms
--- 172.16.0.3 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.116/1.116/1.116/0.000 ms
gateway ~ $ ping -c 1 172.16.1.2
PING 172.16.1.2 (172.16.1.2): 56 data bytes
64 bytes from 172.16.1.2: icmp_seq=0 ttl=63 time=1.638 ms
--- 172.16.1.2 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.638/1.638/1.638/0.000 ms
gateway ~ $ mtr -r -c 1 172.16.1.2
Start: 2022-11-25T22:46:29+0300
HOST: gateway Loss% Snt Last Avg Best Wrst StDev
1.|-- 172.16.0.1 0.0% 1 5.7 5.7 5.7 5.7 0.0
2.|-- 169.254.255.255 0.0% 1 1.2 1.2 1.2 1.2 0.0
3.|-- 172.16.1.2 0.0% 1 1.1 1.1 1.1 1.1 0.0
gateway ~ $ mtr -r -c 1 172.16.0.3
Start: 2022-11-25T22:47:06+0300
HOST: gateway Loss% Snt Last Avg Best Wrst StDev
1.|-- 172.16.0.1 0.0% 1 5.0 5.0 5.0 5.0 0.0
2.|-- 172.16.0.3 0.0% 1 1.4 1.4 1.4 1.4 0.0
dedicated ~$ ping -c 1 172.16.0.10
PING 172.16.0.10 (172.16.0.10): 56 data bytes
64 bytes from 172.16.0.10: icmp_seq=0 ttl=62 time=1.680 ms
--- 172.16.0.10 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.680/1.680/1.680/0.000 ms
dedicated ~$ mtr -n -r -c 1 172.16.0.10
Start: 2022-11-26T09:22:55+0000
HOST: nasogor Loss% Snt Last Avg Best Wrst StDev
1.|-- 172.16.1.1 0.0% 1 1.2 1.2 1.2 1.2 0.0
2.|-- 172.16.0.10 0.0% 1 1.2 1.2 1.2 1.2 0.0
dedicated ~$ mtr -n -r -c 1 172.16.0.3
Start: 2022-11-26T09:25:35+0000
HOST: nasogor Loss% Snt Last Avg Best Wrst StDev
1.|-- 172.16.1.1 0.0% 1 0.6 0.6 0.6 0.6 0.0
2.|-- 172.16.0.3 0.0% 1 1.0 1.0 1.0 1.0 0.0
[w1 ~]$ mtr -n -r -c 1 172.16.1.2
Start: 2022-11-25T19:50:24+0000
HOST: w1 Loss% Snt Last Avg Best Wrst StDev
1.|-- 172.16.0.1 0.0% 1 3.4 3.4 3.4 3.4 0.0
2.|-- 169.254.255.255 0.0% 1 13.3 13.3 13.3 13.3 0.0
3.|-- 172.16.1.2 0.0% 1 1.1 1.1 1.1 1.1 0.0
[w1 ~]$ mtr -n -r -c 1 1.1.1.1
Start: 2022-11-26T06:07:44+0000
HOST: w1 Loss% Snt Last Avg Best Wrst StDev
1.|-- 172.16.0.1 0.0% 1 3.4 3.4 3.4 3.4 0.0
2.|-- 172.16.0.10 0.0% 1 1.2 1.2 1.2 1.2 0.0
3.|-- 172.31.1.1 0.0% 1 5.2 5.2 5.2 5.2 0.0
4.|-- 159.69.96.44 0.0% 1 1.6 1.6 1.6 1.6 0.0
5.|-- ??? 100.0 1 0.0 0.0 0.0 0.0 0.0
6.|-- 213.239.225.45 0.0% 1 37.9 37.9 37.9 37.9 0.0
7.|-- 213.239.239.137 0.0% 1 16.2 16.2 16.2 16.2 0.0
8.|-- 213.239.229.73 0.0% 1 6.7 6.7 6.7 6.7 0.0
9.|-- 213.239.224.178 0.0% 1 6.8 6.8 6.8 6.8 0.0
10.|-- 162.158.84.254 0.0% 1 7.3 7.3 7.3 7.3 0.0
11.|-- 172.70.240.3 0.0% 1 37.4 37.4 37.4 37.4 0.0
12.|-- 1.1.1.1 0.0% 1 7.0 7.0 7.0 7.0 0.0
Final thoughts
I’ve had a lot of fun leveraging Hetzner Networks and FreeBSD on virtual instances. Networks allows servers to communicate through a private network and setup complex network topologies, while FreeBSD added a nice touch to it.