Unifi Security Gateway Dual WAN Policy Routing

Background

I live in the boonies of Michigan, and my only option for unmetered and unlimited internet is SpeedConnect. The company is a smaller WISP that, for the most part, has been pretty solid. Over the past six years, though, they have performed a single network upgrade that doubled my speed to a blazing 6mbps down and 300kbps up.

I began to look for other options to offset the lack of speed during these weird COVID-19 times, where my home internet became much more critical.

I found a “no contract” GSM hotspot that I tether through an OpenWRT router to provide an interface to my USG Pro 4. It is, at best, a hack, but it is one that has increased my speed five fold. Not bad.

Dilema

The “no contract” GSM rides the Sprint; I mean T-Mobile, network, and is supposedly fully unlimited. The Band 41 coverage with the external antennas has been reliable, but there are reports of heavy throttling with tons of usage, which I have because I am a geek.

I wanted to work around this by focusing on heavy consumers to use the bulk SpeedConnect connection while pushing the majority of daily traffic to the GSM link.

This means that things like Steam and Sony PSN downloads go through the SpeedConnect at night when we are sleeping.

Technically speaking, this should be easy peasy with EdgeOS and my Unifi USG device. It was not. What would have taken me 15 minutes on a Cisco, Juniper, or Pfsense device took forever.

Lessons Learned

  1. You cannot use the interface as the next-hop within the LOAD_BALANCE rule. You have to use the IP address.
  2. It’s important that the static tables come before you attempt to populate.
  3. Certain traffic is NOT traversing the table regardless. No clue why, but if you have any traffic that is a mix of TCP/UDP and other…good luck. Sometimes it works, mostly it doesn’t.
  4. dnsmasq and ipset work, but are difficult to maintain as the DNS changes with updates to the various LB in play at these corporations.
  5. Check your config manually before trying to generate the config.gateway.json file.

The Config

Be sure you SSH into your USG and enter configure mode!

### Modify the monitoring address
### The default ping.ubnt.com forced my WAN monitoring to flap every few minutes
set load-balance group wan_failover interface eth2 route-test type ping target 8.8.8.8
set load-balance group wan_failover interface eth2 route-test interval 2
set load-balance group wan_failover interface eth2 route-test initial-delay 5
set load-balance group wan_failover interface eth2 route-test count failure 8
set load-balance group wan_failover interface eth2 route-test count success 2
set load-balance group wan_failover interface eth3 route-test type ping target 8.8.8.8
set load-balance group wan_failover interface eth3 route-test interval 2
set load-balance group wan_failover interface eth3 route-test initial-delay 5
set load-balance group wan_failover interface eth3 route-test count success 2
set load-balance group wan_failover interface eth3 route-test count failure 8
commit

### Protocol tables
# SpeedConnect
# I create both tables as I have different uses when an interface goes down
set protocols static table 5 route 0.0.0.0/0 next-hop 192.168.0.254
set protocols static table 6 interface-route 0.0.0.0/0 next-hop-interface eth2
# GSM
set protocols static table 7 route 0.0.0.0/0 next-hop 192.168.8.1
set protocols static table 8 interface-route 0.0.0.0/0 next-hop-interface eth3
commit

### Port 1
set firewall modify LOAD_BALANCE rule 2500 action modify
set firewall modify LOAD_BALANCE rule 2500 modify table 5
set firewall modify LOAD_BALANCE rule 2500 source address xxx.xxx.xxx.xxx/xx
set firewall modify LOAD_BALANCE rule 2500 destination port xxx
set firewall modify LOAD_BALANCE rule 2500 protocol tcp_udp  
commitcon

### Port 2
set firewall modify LOAD_BALANCE rule 2501 action modify
set firewall modify LOAD_BALANCE rule 2501 modify table 5
set firewall modify LOAD_BALANCE rule 2501 source address xxx.xxx.xxx.xxx/xx
set firewall modify LOAD_BALANCE rule 2501 destination port xxx
set firewall modify LOAD_BALANCE rule 2501 protocol tcp_udp
commit

### Port 3
set firewall modify LOAD_BALANCE rule 2502 action modify
set firewall modify LOAD_BALANCE rule 2502 modify table 5
set firewall modify LOAD_BALANCE rule 2502 source address xxx.xxx.xxx.xxx/xx
set firewall modify LOAD_BALANCE rule 2502 destination port xxx
set firewall modify LOAD_BALANCE rule 2502 protocol tcp_udp
commit

### Setup dnsmasq groups
# SpeedConnect
set firewall group address-group RouteThroughSC
set firewall group address-group RouteThroughSC description "Route through SpeedConnect"
# GSM
set firewall group address-group RouteThroughGSM
set firewall group address-group RouteThroughGSM description "Route through GSM"
commit

### IPSET to route specific items through each WAN
### I use benchmarking tools and ip checking websites to get around how crappy unifi WAN traffic monitoring is
set service dns forwarding options ipset=/ipchicken.com/netflix.com/nflxext.com/nflximg.com/nflxso.net/nflxvideo.net/dvd.netflix.com/hulu.com/hulustream.com/youtube.com/tv.youtube.com/spotify.com/i.scdn.co/scdn.co/audio-ak-spotify-com.akamaized.net/heads4-ak-spotify-com.akamaized.net/spotifycdn.net/speedtest.net/RouteThroughSC
set service dns forwarding options ipset=/whatismyip.com/whatismyipaddress.com/fast.com/RouteThroughGSM
commit

# SpeedConnect
set firewall modify RouteThroughSC rule 2503 action modify
set firewall modify RouteThroughSC rule 2503 modify table 5
set firewall modify RouteThroughSC rule 2503 description "SpeedConnect Routes"
set firewall modify RouteThroughSC rule 2503 destination group address-group RouteThroughSC
set firewall modify RouteThroughSC rule 2503 protocol all

# GSM
set firewall modify RouteThroughGSM rule 2504 action modify
set firewall modify RouteThroughGSM rule 2504 description "GSM Routes"
set firewall modify RouteThroughGSM rule 2504 destination group address-group RouteThroughGSM
set firewall modify RouteThroughGSM rule 2504 modify table 7
set firewall modify RouteThroughGSM rule 2504 protocol all

### Save it locally
commit; save; exit

After you complete this configuration…TEST IT! Assuming it works then you can save it off and upload it to your cloudkey/controller!

mca-ctrl -t dump-cfg > config.txt

Images of the Setup

Hack to get around lack of USB Modem support on Unifi

SpeedConnect Tree Modem

Updated: