One public IPv4 for the whole cluster

Public IPv4 addresses are now a line item worth thinking about. Paying for one per node, so that machines which only ever make outbound connections each get their own, is money spent on nothing. So I run a single one: the control plane has a routable public address, and the workers have none.

That arrangement is common enough. What is less commonly written down is the specific way it fails when you build it on a cloud private network, and why the fix is not where you would look for it.

The shape

In my setup, application ingress arrives at the node with the public IPv4 address, then routes to workloads across the cluster. Direct access to workers is controlled separately through their network configuration and firewall rules.

Outbound is where the work is. Workers still need to reach the internet: pulling images, calling third party APIs, fetching packages during a build. With no public address of their own, that traffic has to leave through the one node that has one. So the master becomes a NAT gateway for the rest of the cluster.

Check public IPv4 and IPv6 separately. A worker without a public IPv4 address may still have public IPv6 connectivity. Removing IPv4 alone does not make the machine unreachable from the internet; routing, firewall rules, and exposed services determine that.

The part that does not work

The obvious configuration is to give each worker a default route pointing at the master’s private address. It reads correctly. It is what you would write on a whiteboard.

# looks right, does not work
ip route add default via 10.0.0.2 dev eth0 onlink

It fails silently. Packets go nowhere and you get no useful error, because nothing is technically wrong with the configuration. The problem is an assumption underneath it.

A cloud private network that presents itself as 10.0.0.0/24 is not necessarily a flat layer 2 segment. On the provider I use, each node’s interface holds a /32. Every node has exactly one address and no notion of a neighbouring range. All traffic between hosts, even hosts that appear to sit in the same subnet, is routed through the provider’s gateway.

The consequence is that nodes are not directly reachable at the link layer. Their ARP tables hold the gateway’s MAC address and nothing else. onlink tells the kernel to send the frame directly to the next hop without needing a route to it, which is precisely the thing that cannot happen here. The frame cannot be delivered to the master, because there is no path to the master that does not go through the gateway first.

You can confirm this quickly on any node:

ip route show
# 10.0.0.0/24 via 10.0.0.1 dev eth0

ip neigh show dev eth0
# only the gateway appears, never the other nodes

If your route table says the whole subnet is reachable via a gateway rather than directly on the device, you are in this situation, and any configuration that assumes host to host adjacency will fail.

Where the fix actually lives

The fix spans both the workers and the provider network. Each worker needs a default route to the private network gateway; the provider network then needs a route forwarding internet-bound traffic to the NAT host.

On Hetzner, add a network route for 0.0.0.0/0 with the NAT host’s private address as the destination gateway. On each worker, configure a default route via the provider’s private gateway, such as 10.0.0.1. The two routes serve different purposes, and both are needed. Make the worker route persistent using the host’s network configuration.

The master then needs to be willing to forward and to masquerade, which is ordinary Linux:

net.ipv4.ip_forward=1

iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE

Two things to get right here. Make the sysctl persistent, or the cluster loses egress on the next reboot of the master. And make sure your masquerade rule survives whatever else is writing to iptables on that host, which on a Kubernetes node is a great deal.

What this costs you

Worth being clear that this is a real tradeoff, not a free saving.

• The master becomes a single point of failure for all outbound traffic. If it goes down, workers keep running but stop being able to pull images or reach any external service.
• All egress shares one source address, so any rate limiting or reputation applied by an external service now applies to your whole cluster at once.
• The master carries traffic it would not otherwise carry, which matters if it is also your smallest node.

For a cluster where a brief loss of outbound connectivity is an inconvenience rather than an outage, that is a reasonable trade. For anything where it is not, you want a second gateway and a failover story, and at that point the cost saving has mostly evaporated.

The general lesson I took from it is narrower than the setup: when a network behaves in a way your configuration says is impossible, check whether the subnet you were handed is actually a subnet.

For the provider-specific routing model and examples, see Hetzner’s network architecture documentation.