One public IPv4 for the whole cluster

Public IPv4 addresses are now a line item worth thinking about. Paying for one per node, so that machines which only ever make outbound connections each get their own, is money spent on nothing. So I run a single one: the control plane has a routable public address, and the workers have none.

That arrangement is common enough. What is less commonly written down is the specific way it fails when you build it on a cloud private network, and why the fix is not where you would look for it.

The shape

Inbound is straightforward. Everything arrives at the one node that has an address, and the ingress controller routes from there. Workers receive no direct inbound traffic at all, which is a security property worth having on its own.

Outbound is where the work is. Workers still need to reach the internet: pulling images, calling third party APIs, fetching packages during a build. With no public address of their own, that traffic has to leave through the one node that has one. So the master becomes a NAT gateway for the rest of the cluster.

One detail that catches people out before they even get to routing: a node without a public address is not necessarily addressless. On some providers it will be handed an address from the carrier grade NAT range, 100.64.0.0/10. It looks like an address. It is not routable, it changes across reboots, and it is useless for both ingress and egress. Do not build anything on it.

The part that does not work

The obvious configuration is to give each worker a default route pointing at the master’s private address. It reads correctly. It is what you would write on a whiteboard.

# looks right, does not work
ip route add default via 10.0.0.2 dev eth0 onlink

It fails silently. Packets go nowhere and you get no useful error, because nothing is technically wrong with the configuration. The problem is an assumption underneath it.

A cloud private network that presents itself as 10.0.0.0/24 is not necessarily a flat layer 2 segment. On the provider I use, each node’s interface holds a /32. Every node has exactly one address and no notion of a neighbouring range. All traffic between hosts, even hosts that appear to sit in the same subnet, is routed through the provider’s gateway.

The consequence is that nodes are not directly reachable at the link layer. Their ARP tables hold the gateway’s MAC address and nothing else. onlink tells the kernel to send the frame directly to the next hop without needing a route to it, which is precisely the thing that cannot happen here. The frame cannot be delivered to the master, because there is no path to the master that does not go through the gateway first.

You can confirm this quickly on any node:

ip route show
# 10.0.0.0/24 via 10.0.0.1 dev eth0

ip neigh show dev eth0
# only the gateway appears, never the other nodes

If your route table says the whole subnet is reachable via a gateway rather than directly on the device, you are in this situation, and any configuration that assumes host to host adjacency will fail.

Where the fix actually lives

Not on the nodes. The routing decision has to be made by the thing that is already routing everything, which is the provider’s network.

Cloud private networks generally support adding routes as a property of the network itself. You add a route saying 0.0.0.0/0 goes to the master’s private address, and the provider’s gateway starts forwarding accordingly. The workers need no default route of their own for this to work. They send to the gateway, as they already did for everything else, and the gateway hands it onward.

The master then needs to be willing to forward and to masquerade, which is ordinary Linux:

net.ipv4.ip_forward=1

iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o eth0 -j MASQUERADE

Two things to get right here. Make the sysctl persistent, or the cluster loses egress on the next reboot of the master. And make sure your masquerade rule survives whatever else is writing to iptables on that host, which on a Kubernetes node is a great deal.

What this costs you

Worth being clear that this is a real tradeoff, not a free saving.

• The master becomes a single point of failure for all outbound traffic. If it goes down, workers keep running but stop being able to pull images or reach any external service.
• All egress shares one source address, so any rate limiting or reputation applied by an external service now applies to your whole cluster at once.
• The master carries traffic it would not otherwise carry, which matters if it is also your smallest node.

For a cluster where a brief loss of outbound connectivity is an inconvenience rather than an outage, that is a reasonable trade. For anything where it is not, you want a second gateway and a failover story, and at that point the cost saving has mostly evaporated.

The general lesson I took from it is narrower than the setup: when a network behaves in a way your configuration says is impossible, check whether the subnet you were handed is actually a subnet.