We have a couple of 802.11q VLAN interfaces on our Check Point Firewalls for a number of small public DMZ networks, providing full layer 2 isolation for usually only one or two related servers in those networks. Using a /29 subnet for each VLAN means minus broadcast and network address, we can assign 6 (public) IP addresses in those networks. We also need an IP for the virtual ClusterXL interface to be used as a gateway by the servers. Up until recently, we also used 2 more IPs of each subnet for the members VLAN interfaces, so they can communicate via Check Points Cluster Control Protocol or CCP. This left us with merely 3 public IPv4 addresses per /29 subnet for actual servers inside those VLANs.
Wasting IPs like this on interfaces transparent for your actual services is not really practical in times of scarce IPv4 address space though. So what we did was getting rid of using IP addresses of the public subnet for the VLAN interfaces of each cluster member during our upgrade to R75.40 and Gaia on new systems. This configuration is fully supported (actually since long ago) as per ClusterXL Admin Guide:
Only one routable IP address is required in a ClusterXL cluster, for the virtual cluster interface that faces the Internet. All cluster member physical IP addresses can be non-routable.Configuring different subnets for the cluster IP addresses and the member addresses is useful in order to:
– Enable a multi-machine cluster to replace a single-machine gateway in a pre-configured network, without the need to allocate new addresses to the cluster members.
– Allow organizations to use only one routable address for the ClusterXL Gateway Cluster. This saves routable addresses.
ClusterXL virtual IPs and your members physical (or VLAN) interfaces do not need to be on the same subnet. So you can simply use whichever addresses you like for each of the cluster interfaces (apart from internal/management and external/VPN-routable interfaces obviously). And of course this applies to physical untagged interfaces unlike our case too.
I settled for using tiny Class B private space /30 subnets for each VLAN, enough for just our 2 cluster members like this. The topology would then look like this.
Beware of the spoofing and routing
Now here’s just 2 catches with this configuration. First off, anti-spoofing will apply to the members local interface network and not the ClusterXL virtual one, so you can’t use the comfortable “Network defined by the interface IP and Net Mask” setting unless you want all your traffic dropped/detected as spoofed. Instead just define a specific subnet object representing the ClusterXL interface subnet.
The second thing which shortly caused some headache for me was that SPLAT/Gaia wouldn’t know where it needs to route the public subnet. Now that the physical interfaces to those subnets had different IPs, the OS naturally lacked the proper routing information and would forward traffic through the default route.
To solve this, I added static interface-based routes for each public subnet like this. To my confusion however, they didn’t help and seemed to have no effect. Checking the firewall nodes routing table via SSH confirmed that there was no corresponding entry present.
Instead I had to issue the following in expert mode on the nodes to activate my routes:
# route add -net 18.104.22.168/29 eth8.356
The routing table would now look like this:
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
172.31.255.16 0.0.0.0 255.255.255.252 U 0 0 0 eth8.356
22.214.171.124 0.0.0.0 255.255.255.248 U 0 0 0 eth8.356
On the new Gaia CLI it looks like this:
> show route
Codes: C – Connected, S – Static, R – RIP, B – BGP,
O – OSPF IntraArea (IA – InterArea, E – External, N – NSSA)
A – Aggregate, K – Kernel Remnant, H – Hidden, P – Suppressed
C 126.96.36.199/29 is directly connected, eth8.356
C 172.31.255.16/30 is directly connected, eth8.356
The route command in expert mode alone doesn’t survive a reboot, you still need to set all routes in the Gaia/SPLAT CLI/Webinterface on all members. I confirmed that the routes are being applied properly after a reboot. Also, static routes using Gateway IPs do not need a reboot either, so this seems like a bug specific to using interface-based routes.