Sharing an IP between NAT clients and a server

2018-03-10 08:19

Sharing an IP between NAT clients and a server

Sharing an IP address between NAT clients and a server

The situation is thus, you have an upstream connection (say, to the Internet) for which you have a single IP address. Multiple hosts need access to this network, using usual protocols like TCP or UDP. This is accomplished with a masquerading device, translating a different subnet's (usually a private RFC1918 address space) addresses to the single upstream address and rewriting port numbers to provide return paths to the individual hosts.

So far so good. Now if want to expose services that do not require establishment from this private network first, the normal solution is to configure the router to fixedly map the port numbers needed to the appropriate local address. Or in some cases to map an entire protocol that the NAT functionality cannot support, like IPsec, or even SCTP, to a single address in the private network.

This works well enough, most of the time. But some protocols, espeically those designed to work on large-scale networks, or at least networks which do not have the complex and essentially broken addressing structure that NAT imposes are not as easy to simply rewrite the transport and have continue to function. One solution here is to have the NAT device become ever more complex, understanding these protocols and doing higher-layer translations, acting as an ALG for the protocol.

The other is what I am attempting here. I seek to create a NAT router that does not itself use the upstream IP address for communications directly, but via its private address; reservingthe up stream address for the designated server host, if no NAT mapping, or other configuration is required. This allows the router to continue to firewall the entire network, but places a single 'designated' server in a sort of DMZ, allowing the local network to access it, and it to the local network (firewall rules permitting), as well as directly assigning the upstream IP adddress to its interface, so that protocols relying on accurate addressing information can run unaffected.

| Uplink Device |
  ----------     ---------------------
  | Router | <-> | Designated Server |
  ----------     ---------------------
| Client Devices |

The premise of this setup is that the designated server need not have any special configuration, it appears to be sitting directly connected to the upstream network, and is configured as such. The only complication or unusual piece of configuration is to ensure that the emphemeral ports used by the designated server to do not overlap the ports used in NAT rewriting by the router. Similarly clients do not have any unusual configuration, they behave exactly as any other host behind a masquerading NAT device.

All of the complexity of routing and address translation is confined to the router device, in the diagram above.

      |---------------- wan0------------|
      |                                 |
      |           ROUTER                |
   [lan-ip]                         [wan-gw-ip]
     lan0                              dmz0
      ^                                 ^
      |                                 |
      V                                 V
|--------------|                |-----[wan-ip]-----|
|  CLIENT      |                |     server0      |
|  HOSTS       |                |                  |
|--------------|                |   DESIGNATED     |
                                |   SERVER         |

So in this configuration, the designated server is configured as if it were directly connected to the upstream network. The router needs specific forwarding and routing rules so that all traffic destined to the upstream network from the designated server is directly forwarded, while traffic from the lan side is masqueraded. Return path traffic follows port numbers for when those are recognized by the router, in order to operate the NAT portion of its function, but for other port numbers, or any protocols for which port numbers are not available, the traffic is directly forwarded to the designated server.

Traffic for the designated server from the lan is addressed to the [wan-ip], for which the router needs to have a route, as well, the router's [lan-ip] is configured as an onlink route for the designated server, and the gateway for the lan addresses.

Simple, on paper. But how to configure the router to do this?

After reading the documentation on iproute2 and nftables, I figured I may as well give it a go, setting up a pile of virtual machines to test the behavior and get the configuration working. (Or otherwise; at this point I am not convinced the networking stack will let me do this.)


For a test, I need to define the addresses, so here goes. WAN Subnet The directly connected upstream subnet WAN IP The assigned from upstream IP address WAN GW The assigned from upstream gateway address WAN Host An arbitrary host 'beyond' the WAN gateway LAN Subnet The local (private) LAN subnet LAN Gatway The local address of the router on the LAN subnet LAN Client Host An arbitrary host on the LAN subnet which should be a "client" (subject to NAT to reach the WAN)

With that defined, I've set about creating a set of virtual machines connected by virtual bridges using libvirt/kvm to test how to configure this. I'm using a Debian "Buster" install DVD to build machines without access to the normal network/Internet so that the testing can be isolated sensibly.

The networks and domains were defined using these XML files.

ip-sharing/test-lan.xml (Source)

    <bridge name="test-lan" stp="off"/>
    <domain name="lan.invalid"/>

ip-sharing/test-wan.xml (Source)

    <bridge name="test-wan" stp="off"/>

ip-sharing/test-dmz.xml (Source)

    <bridge name="test-dmz" stp="off"/>

ip-sharing/router.xml (Source)

<domain type='kvm' id='15'>
  <memory unit='KiB'>262144</memory>
  <currentMemory unit='KiB'>262144</currentMemory>
  <vcpu placement='static'>1</vcpu>
    <type arch='x86_64' machine='pc-1.1'>hvm</type>
    <boot dev='hd'/>
    <boot dev='cdrom'/>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>kvm64</model>
    <feature policy='require' name='hypervisor'/>
  <clock offset='utc'/>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/home/enimihil/vm/test-router.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/enimihil/vm/debian-testing-amd64-DVD-1.iso'/>
      <target dev='hdc' bus='ide'/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    <filesystem type='mount' accessmode='passthrough'>
      <driver type='path'/>
      <source dir='/home/enimihil/vm/'/>
      <target dir='/shared'/>
      <alias name='fs0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    <interface type='network'>
      <mac address='52:54:00:4e:3f:5c'/>
      <source network='test-lan' bridge='test-lan'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    <interface type='network'>
      <mac address='52:54:00:61:08:1f'/>
      <source network='test-wan' bridge='test-wan'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    <interface type='network'>
      <mac address='52:54:00:ad:8a:69'/>
      <source network='test-dmz' bridge='test-dmz'/>
      <target dev='vnet2'/>
      <model type='virtio'/>
      <alias name='net2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    <serial type='pty'>
      <source path='/dev/pts/13'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      <alias name='serial0'/>
    <console type='pty' tty='/dev/pts/13'>
      <source path='/dev/pts/13'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    <graphics type='spice' port='5900' autoport='yes' listen=''>
      <listen type='address' address=''/>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'>

Since the nftables package is not available on the installation DVD(s), I had to manually download a version from the mirrors and shared it over 9pfs with the guest machines, along with the libnftnl7 dependency.

So, with my various machines created and connected to their virtual networks, set about to manually configure the client machine, server, and wan-gw as fairly standard hosts.

Set up the client machine with a private address and default route.

client# ip link set ens9 name client0
client# ip link set up client0
client# ip addr add dev client0
client# ip route add default via

client# sysctl -w net.ipv4.conf.all.log_martians=1

Set up the WAN gateway with the WAN subnet (eventually add routed network beyond for it.)

wan-gw# ip link set ens9 name wan0
wan-gw# ip link set up client0
wan-gw# ip addr add dev wan0

client# sysctl -w net.ipv4.conf.all.log_martians=1

Set up the server as if it were directly connected to the WAN subnet.

server# ip link set ens9 name server0
server# ip link set up server0
server# ip addr add dev server0
server# ip route add default via

server# sysctl -w net.ipv4.conf.all.log_martians=1

And now comes the complex part, how to set up the router to make all of this talk the way I want it to.

router# ip link set ens3 name lan0
router# ip link set up lan0
router# ip addr add dev lan0

router# ip link set ens8 name wan0
router# ip link set up wan0
router# ip addr add dev wan0
router# ip route add default via

router# ip link set ens9 name dmz0
router# ip link set up dmz0
router# ip addr add dev dmz0

router# sysctl -w net.ipv4.ip_forward=1
router# sysctl -w net.ipv4.conf.all.log_martians=1

And now we see, looking at the routing table on the router, our first challenge.

router# ip route
default via dev lan0 dev lan0 proto kernel scope link src dev wan0 proto kernel scope link src dev dmz0 proto kernel scope link src

We now have the same WAN subnet on two different interfaces. The reality is that there's only a single on the dmz0 interface, and the rest will be reachable via wan0.

A ping from the client does not appear on the server's network interface, as the router absorbs the traffic and responds.

Removing the address on the router allows the client to ping the server, via the router, however the wan-gw cannot reach the server as the router does not answer arp requests and the wan-gw needs the server address to be on link.

Similarly, the router has the WAN GW address configured as a local address so that the server can route through it, and this causes the server to be able to ping the WAN gateway address, though it does not actually reach the wan-gw host.

Let's try policy routing.

# Add a mapping to 1=dmz in /etc/iproute2/rt_tables

router# ip addr del dev dmz0
router# ip addr del dev wan0

router# ip rule add from iif dmz0 table dmz
router# ip rule add to iif wan0 table dmz
router# ip route add table dmz dev wan0
router# ip route add table dmz dev dmz0

Now our router doesn't know how to route to the subnet from itself. But, now when we try to ping the wan-gw from the server, we see that the server is attempting to resolve the link address of the wan-gw (and failing to do so.) Likewise, the wan-gw attempts to resolve via ARP the link address of the server.

This is a job for the proxy ARP feature, so lets turn that on for the relevant interfaces.

router# sysctl -w net.ipv4.conf.dmz0.proxy_arp=1
router# sysctl -w net.ipv4.conf.wan0.proxy_arp=1

Ah, success, we can now ping between the WAN GW and the server.

However, we can't ping or access the WAN GW from the router, nor the serverHowever, we can't ping or access the WAN GW from the router, nor the server. We are also unable to ping the server IP from the client, nor the wan-gw from the client.

router# ip route add dev dmz0

Adding the WAN IP to the main routing table gets us the ability for the client to talk to the server.

router# ip route add dev wan0

At this point, the policy routing rules and the dmz table aren't used.

Our default route still doesn't work, though. Adding a route to the main table allows the ping to traverse to the wan-gw from the client, but we need to configure NAT, as the client address is not routable from the wan-gw.

This seems to not work, for anything I've tried, without configuring addresses on the router's interfaces for wan0 and dmz0. So back to that.

router# ip addr add dev wan0
router# ip addr add dev dmz0
router# ip route del dev wan0
router# ip route del dev dmz0
router# ip route del table local local dev wan0
router# ip route del table local local dev dmz0

Here we delete all the automagic routes and local routes for the addresses on the router's interfaces that aren't its own. Proxy ARP is still active and the previous direct /32 routes for the specific IP addresses we need to reach are still there.

Now a ping from the server to the wan IP goes to the wan gateway and vice versa, due to the proxy arp and removed local routes.

Now for the NAT.