Thank you very much for your explaination. That clarifies a lot. Strangely, the passage at the end of this section of the LARTC article somehow escaped my attention. (But even if I had heeded that passage, I could probably not have grasped its meaning as fully as I can in the lights of your kind comments.)

Thank you for your comment and the link. You bring up some interesting points. In fact, we started our original research with the same article you mentioned. However, I believe the difference is highlighted in this quote from the LARTC article:

“It will work for all processes running on the router itself, and for the local network, if it is masqueraded. If it is not, then you either have IP space from both providers [...] you will want to add rules selecting which provider to route out from based on the IP address of the machine in the local network.”

In our setup, we have several servers behind the router/fw on the local LAN. Each of these servers have corresponding “external” IP addresses provided by the respective two ISPs. Depending on which ISP, the router has bound to its own interfaces these external IP addresses and DNATs/SNATs them accordingly to/from our servers so that to the outside world, it appears that our servers all have external IP addresses with ports open on the services desired. This is useful so that if we wanted to host two identical services (say Web servers) using two different external IPs, we could.

Here’s an illustration:

ISP 1:
123.234.1.10 –maps-to–> router’s eth1
123.234.1.11 –router-DNATs-to–> server 1, port 1494
123.234.1.12 –router-DNATs-to–> server 2, port 25,80
123.234.1.13 –router-DNATs-to–> server 3, port 80

ISP 2:
216.111.1.10 –maps-to–> router’s eth2
216.111.1.11 –router-DNATs-to–> server 1, port 1494
216.111.1.12 –router-DNATs-to–> server 2, port 25

Router has local LAN address 172.16.1.1
Server 1 has local LAN address 172.16.1.131
Server 2 has local LAN address 172.16.1.130
Server 3 has local LAN address 172.16.1.140

Now because most of these servers have two external IPs (i.e. users can access the service via either provider) and because they map to only ONE internal IP, there is no way for the router to tell which interface to send SNATed packets back out. The only thing it can do is consult the default routing table which would have a default gateway, or at best (as the article suggests) consult a rule in the routing policy database that says essentially “if the packet originates FROM 172.16.1.x, send it out according to routing table {aaaa}”, where aaaa is the routing table for one of the providers.

This latter part actually works (sometimes) but only insomuch as receiving packets from one provider and sending a reply packet out via another provider. However, this is less than desirable, as the whole point of this is to handle the situation when one of the providers might be down. Also, many ISPs will not let you send a packet with a source address other than one of their own.

Thus, in our situation, it is not possible to do what we want to do without connection marking.

Hope that helps to clarify things.

BTW, if you haven’t already, you may also wish to read the addendum article I also posted which clarifies a few additional details.

Jonathan.