Best practice for dual-homing site with two ISPs?

Question

I am subscribed to two ISPs, a fast expensive one and a cheap but slower one. They use different technologies, cable and ADSL, so there isn't much of a single point of failure, and all my comms equipment is powered from a UPS.

ISPs in the UK go down in a fairly random pattern. Over many years I've yet to encounter a moment when both my ISPs have gone down simultaneously, so clearly the two-ISP strategy is helpful if you want uninterrupted net access here.

The problem however is how to organize your site's networking to take advantage of this improved availability. Many ISPs don't allow you to run your own AS and routing protocols, so you're mostly stuck with splitting your static routing across your two egress pipes by destination. This is less than brilliant, and requires manual intervention when one ISP drops off the face of the planet. With the help of numerous scripts I handle ISP outages with reasonable success and not much effort, and it's become business as usual. It's not great though. It feels like some technology is missing.

1. Is there a better way, generally?
2. Is there a better way for IPv6 only (I have dual stack on one ISP, and could tunnel on the other)? That would be a clear boon for IPv6.

Answer 1

What kind of equipment do you have connecting to the providers? If it is a Cisco device you could use IP SLA to ping a destination like 8.8.8.8 over the primary ISP. As soon as you don't get a reply failover to the other static route. Example config:

! ISP1
ip route 0.0.0.0 0.0.0.0 x.x.x.x track 1
! ISP2
ip route 0.0.0.0 0.0.0.0 y.y.y.y 250
ip sla 1
icmp-echo 8.8.8.8 source-interface <ISP1_interface>
frequency 3
timeout 1000
ip sla schedule 1 life forever start-time now

You may have to put a static route for 8.8.8.8 over ISP1 to make it always exit out that path. Obviously if you really use 8.8.8.8 choose another IP because otherwise you won't have reachability to it if ISP1 goes down.

!x.x.x.x is next-hop to ISP1
ip route 8.8.8.8 255.255.255.255 x.x.x.x

Answer 2

You might want to investigate if there is a LISP provider available. LISP is a protocol that can make a site independent from the upstream ISPs it connects to. You get one or more IP addresses from the LISP ISP and they route them to wherever you are connected. It is a tunnelling technique, but with a lot of cool features. You can control both inbound and outbound load balancing over the links, you can do IPv6 multihoming without having to resort to hacks like NPT66 (prefix translation). You could even move to the other side of the planet without changing IP addresses ;-)

I use LISP myself and my office network has a block of /26 IPv4 and a /48 block of IPv6 addresses that are independent of the upstreams (a UPC cable connection with one dynamic IPv4 address and a Solcon DSL connection with both a static IPv4 address and a static block of IPv6 addresses). A Cisco 1841 runs LISP in the office and uses whatever link is available to connect to the rest of the Internet. As long as one link works my office is connected using its own addresses.

Full disclosure: I run my own LISP based ISP in The Netherlands, so I am biased. LISP is still a cool protocol though :-)

Answer 3

In IPv6 multi-homing with provider aggregated addresses, each host in the network will get one address prefix from each of the providers. The host stack/application's source address selection (RFC6724) and choice of SA/DA pair (RFC6555) determines which exit is used.

I.e the host/application's choice of source address picks which exit link is used. Various implementations do this in various ways, and none do it very well at the moment.

The network uses source address dependent routing to forward the traffic on to the correct exit. (Otherwise BCP38 (ingress filtering) would have dropped a packet sent with ISP B's source address to ISP A). See https://datatracker.ietf.org/doc/html/draft-troan-homenet-sadr-01 We have an implementation in OpenWRT. But it can also be implemented reasonably well on any router supporting policy based routing.

An application should be smart enough to change connection (pick another SA/DA pair) when the current connection fails. It isn't. In the mean time our recommendation is to set the lifetime of the failing link's address prefix to 0, meaning that new connections will not use that address.

Answer 4

Q1. You can install a router/firewall that supports multihoming. In terms of free software, pfSense fits the bill. http://www.pfsense.org/. The pfSense docs refer to this as Multi-WAN. http://doc.pfsense.org/index.php/Multi-WAN_2.0

Helpfully, pfSense has automatic failover and load balancing.

What I have found is that a small number of web apps do not work when you are multihomed. The web apps are usually financial sites, like banks. For some reason, web app programmers sometimes think it is OK to test security on the basis of IP address. For users who need this access, you can reserve an IP address for their computer, and create a "LAN rule" in pfSense to always use a certain gateway and not the other for that IP address.

I find that this works quite well for the combination of cable modem and ADSL modem.

Q2. There is no reason why multihoming wouldn't work in IPv6 as well as IPv4. That said, pfSense does not have a fully-supported release that properly handles IPv6. The current version of pfSense is 2.0x. Once 2.1 is released, pfSense will have good IPv6 support and will include multihoming.

Answer 5

You could setup a remote server/VPS in a reliable data center and then you could setup VPN tunnels from your router to the remote server over each ISP. Now your router at home could route packets over these tunnels based on a bandwidth ratio, and then the remote server can route the traffic between the rest of the internet.

The disadvantage is that you'll incur additional cpu, storage and bandwidth costs for the remote server.

The advantage is that you can use the entire bandwidth provided by both the providers while still having a fail over option for reliability.

Answer 6

I've used OpenWRT with Multiwan (http://wiki.openwrt.org/doc/uci/multiwan) for some time at home to multihome between my DSL and Cable ISP. It worked quite nicely. I wouldn't advice it as a corporate solution, but it's ok for SOHO and home setups.