Running Check Point ClusterXL member interfaces on different subnets than the virtual interfaces

We have a couple of 802.11q VLAN interfaces on our Check Point Firewalls for a number of small public DMZ networks, providing full layer 2 isolation for usually only one or two related servers in those networks. Using a /29 subnet for each VLAN means minus broadcast and network address, we can assign 6 (public) IP addresses in those networks. We also need an IP for the virtual ClusterXL interface to be used as a gateway by the servers. Up until recently, we also used 2 more IPs of each subnet for the members VLAN interfaces, so they can communicate via Check Points Cluster Control Protocol or CCP. This left us with merely 3 public IPv4 addresses per /29 subnet for actual servers inside those VLANs.
Wasting IPs like this on interfaces transparent for your actual services is not really practical in times of scarce IPv4 address space though. So what we did was getting rid of using IP addresses of the public subnet for the VLAN interfaces of each cluster member during our upgrade to R75.40 and Gaia on new systems.  This configuration is fully supported (actually since long ago) as per ClusterXL Admin Guide:

Configuring Cluster Addresses on Different Subnets
Only one routable IP address is required in a ClusterXL cluster, for the virtual cluster interface that faces the Internet. All cluster member physical IP addresses can be non-routable.Configuring different subnets for the cluster IP addresses and the member addresses is useful in order to:
– Enable a multi-machine cluster to replace a single-machine gateway in a pre-configured network, without the need to allocate new addresses to the cluster members.
– Allow organizations to use only one routable address for the ClusterXL Gateway Cluster. This saves routable addresses.

ClusterXL virtual IPs and your members physical (or VLAN) interfaces do not need to be on the same subnet. So you can simply use whichever addresses you like for each of the cluster interfaces (apart from internal/management and external/VPN-routable interfaces obviously). And of course this applies to physical untagged interfaces unlike our case too.
I settled for using  tiny Class B private space /30 subnets for each VLAN, enough for just our 2 cluster members like this. The topology would then look like this.

Beware of the spoofing and routing

Now here’s just 2 catches with this configuration. First off, anti-spoofing will apply to the members local interface network and not the ClusterXL virtual one, so you can’t use the comfortable “Network defined by the interface IP and Net Mask” setting unless you want all your traffic dropped/detected as spoofed. Instead just define a specific subnet object representing the ClusterXL interface subnet.

The second thing which shortly caused some headache for me was that SPLAT/Gaia wouldn’t know where it needs to route the public subnet. Now that the physical interfaces to those subnets had different IPs, the OS naturally lacked the proper routing information and would forward traffic through the default route.
To solve this, I added static interface-based routes for each public subnet like this. To my confusion however, they didn’t help and seemed to have no effect.  Checking the firewall nodes routing table via SSH confirmed that there was no corresponding entry present.

Instead I had to issue the following in expert mode on the nodes to activate my routes:
# route add -net eth8.356
The routing table would now look like this:
# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface U     0      0        0 eth8.356 U     0      0        0 eth8.356

On the new Gaia CLI it looks like this:
> show route

Codes: C – Connected, S – Static, R – RIP, B – BGP,
       O – OSPF IntraArea (IA – InterArea, E – External, N – NSSA)
       A – Aggregate, K – Kernel Remnant, H – Hidden, P – Suppressed
C     is directly connected, eth8.356
C    is directly connected, eth8.356
The route command in expert mode alone doesn’t survive a reboot, you still need to set all routes in the Gaia/SPLAT CLI/Webinterface on all members. I confirmed that the routes are being applied properly after a reboot. Also, static routes using Gateway IPs do not need a reboot either, so this seems like a bug specific to using interface-based routes.

2 thoughts on “Running Check Point ClusterXL member interfaces on different subnets than the virtual interfaces

  1. Hi,
    I had some issues with the default route going missing after reboot.
    You can still see it via “show route” in normal mode but “route -n” in expert mode does not list it.
    I have opened a ticket with CheckPoint and it seems like this is a know issue for Gaia. I got a hotfix and after installing reboot was no longer an issue and all worked fine.
    I did not get an sk number so am posting the file name below:

  2. confirming reply from Marcin. we however got ilsiebel01_3276_8992_0_os_cprd_cfg-0.1-cp986138001.i386.rpm
    This is for versions below R75.45 I believe. It’s now at least part of the resolved issues.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s